Practical MLOps for better models

PyData Global 2022

Isabel Zimmerman, Posit PBC

hi im isabel

started my career as a software engineer/data scientist, working on operationalizing models, mostly in kubernetes environments. if you know anything about k8s, they can be a little frustrating. so, giving my brain a break, i started training my little dog named toast to do tricks.

when first taught my dog to sit, stood in front of him, had him sit. give him a treat. and of course, when i first took him out for a walk, i wanted to practice, so as he was walking right next to me, i looked down from my shoulder and told him to sit. and he did NOT sit. he acted like he had never

now, as a data scientist, i realized i had not trained my dog very well. he only knew sit when i was standing in front of him. but all the information i had, i had used correctly. he was sitting really well inside my home!

toast taught me a hard lesson. even if youre training for the right outcome, sitting, being out of my cozy local home environment brought a new set of challenges, and he behaved differently. i realized i needed new tools to expand my mindset.

if you develop models…

you can operationalize them

but, this is not a dog training conference–so, i’ll tell you this.

the “real world” value of models often times comes from integrating

my advice for you, is to bring your models outside!!

you can learn to operationalize models using a set of practices called MLOps—

MLOps is…

a set of practices to deploy and maintain machine learning models in production reliably and efficiently

and these practices can be HARD.

these practices are to …

with Kubernetes-based + models. after a while where i STRUGGLED with the tools at hand, i felt that, as a data scientist, i needed to productionalize models, but i didn’t want to have to be a cloud architect as well a data scientist.

and while i don’t think we should be oblivious to everything in the dev ops world, i do think that there is space for tools to help data scientists more effectively communicate with their IT or devops teams, while still feeling ergonomic for data scientists.

• so i changed career paths to build new tools to make mlops feel like an extension of a data scientist’s current workflow. and this tool, is named vetiver.

i do work for posit, which you might have heard of from hadley’s keynote yesterday– this company was formerly known as RStudio, so it may come as no surprise that vetiver is a python package, but also an R package as well.

when you start learning about data science, you see an image that looks something like this: you learn about tools such as –

and so when you write data science code in Python using the packages and BEST practices you’ve learned, it goes something like this:

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

set seed for reproducibility

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

split into training and test sets

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)
oe = preprocessing.OrdinalEncoder().fit(X_train)
rf = ensemble.RandomForestRegressor().fit(oe.transform(X_train), y_train)
rf_pipe = pipeline.Pipeline([('ordinal_encoder',oe), ('random_forest', rf)])

choose the right feature engineering for the job, and train your model

put these together in a pipeline

then i got to my first job and realized there’s more to think about than splitting test/training/etc you need to figure out how to share this model with teammates in a way that DOESNT include emailing your model to other people, you need to integrate it into some larger application (am i going to write this IN MY SHINY APP?), you need to ensure that it still performs well a week from now, or a month from now

you start to realize that these practices are not always enough!

what are some pieces used in mlops?

• orchestration
• experiment tracking
• model versioning
• model serving
• model monitoring

Each open source project falls into one (or sometimes multiple) different categories to fulfill these different tasks: all-in-one, data pipeline, infrastructure, modeling and training, monitoring, and serving

versioning

people think of versioning, its usually in the context of git! but we version lots of different things, and mostly badly

versioning

model

model_final

model_final_final

model_final_final_actually

model_final_final_actually (1)

versioning your model is the foundation for success in machine learning deployments…

we can already see here this is not going to scale for ONE MODEL lacks context

it would be nice if my models: - lived in a central location - were discoverable by my team - loaded right into memory

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # can also be s3, azure, gcs, connect
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object
vetiver_pin_write(model_board, v)

Meta(title='ads: a pinned Pipeline object',
    description="Scikit-learn <class 'sklearn.pipeline.Pipeline'> model", 
    created='20221102T094151Z', 
    pin_hash='4db397b49e7bff0b', 
    file='ads.joblib', 
    file_size=1087, 
    type='joblib', 
    api_version=1, 
    version=VersionRaw(version='65155'), 
    name='ads', 
    user={'required_pkgs': ['vetiver', 'scikit-learn']})

know what your input data should look like

• save a piece of your data to better debug when things go wrong

trying to figure out how to solve a puzzle is a lot harder when you don’t know what the finished product looks like

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp(
    allow_pickle_read = True)

v = VetiverModel(rf, "ads", ptype_data = X_train)
vetiver_pin_write(model_board, rf)

utilizing model cards

not only good models, but good models

• summary
• documentation
• fairness

from a team at google

kinda like writing down a recipe

this feels a bit different, since it is not as maybe TECHNICALLY involved

utilizing model cards

vetiver_pin_write(model_board, v)

utilizing model cards

Model Cards provide a framework for transparent, responsible reporting. 
 Use the vetiver `.qmd` Quarto template as a place to start, 
 with vetiver.model_card()

utilizing model cards

vetiver.vetiver_pin_write(model_board, v)
vetiver.model_card()

utilizing model cards

utilizing model cards

utilizing model cards

my model doesn’t have any ethical challenges, it’s predicting youtube likes

instead of deleting any such section because you have incomplete or imprecise information, you note your own process and considerations.

actually, my dad has a quote that he always tells me–“if you haven’t written it down, you haven’t thought it out”

deploy your model

creating model as a REST API endpoint

useful bc model can be used in memory just like you loaded it! without having to load it

also useful since API endpoints are testable,

deploy your model

deploy your model

my_api = VetiverAPI(v)
my_api.run()

deploy your model

vetiver.deploy_rsconnect(
    connect_server = connect_server, 
    board = model_board, 
    pin_name = "ads", 
    version = "59869")

vetiver.write_app(board=board, pin_name="ads")
vetiver.write_docker(app_file="app.py")

monitoring

model is deployed a data scientist’s work is not done!

now, monitoring means somthing unique in MLOps– not necessarily looking at CPU usage, runtime, etc,

looking at statistical properties of input data or predictions

monitoring

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    [mean_absolute_error, r2_score], 
    "like_count", 
    "y_pred"
    )

vetiver.pin_metrics(
    model_board, 
    metrics, 
    "metrics_pin_name", 
    overwrite = True
    )
    
vetiver.plot_metrics(metrics)

we won’t look super deep at these functions right, but

monitoring

it is SO IMPORTANT TO TRACK your model’s performance metrics start decaying.

software engineering–when things went wrong, ERROR

models fail silently! and they can still run with no error, even if your accuracy is zero percent–

if you are not monitoring your model in some way, you are oblivious to decay.

Why should I be excited about vetiver?

Composability

• Internally, with VetiverAPI and VetiverModel
• Externally, leveraging the tools vetiver is built on

Ergonomics

• feels good to use
• works with the tools you like

few simple tools that you are able to compose together to make complex objects

since vetiverapi is built on fastapi, can build out to be quite complex also has methods to add other POST endpoints

also is composable with other tools to build out a custom framework that works well for your team

---

one thing vetiver has worked really hard on is to lower the barrier to entry on deploying models, making this feel like a natural extension of your current data science workflow

you are still able to use the tools you want

also with leveraging pins, it makes it easy to move data between R and Python at places where this is possible

vetiver.rstudio.com

in a composable and ergonomic way