Demystifying MLOps

Isabel Zimmerman, RStudio PBC

rstudio::conf(2022)

im here to demystfy mlops because machine learning operations are HARD i spent a lot of time with deploying models,,,

more importantly I LOVE BAKING

machine learning models are like chocolate chips…

while model development occurs in notebooks, the value of models often comes when they are integrated into a larger system

what is MLOps?

set of practices to deploy and maintain machine learning models in production reliably and efficiently

guidelines to help your model live outside a notebook

what are some MLOps practices?

• write down our recipe
• baking the dough
• making changes when things don’t go as planned

in the data science world… a few practices that come to mind are… - version - deploy - monitor

now, let’s pause here for a moment– MLOps is not a disjoint piece where data scientists can pass off models to IT, or whoever handles deployment. MLOps is part of the data science lifecycle….

vetiver is a package in r and a package in python. this means that you can use install.packages or pip install or both to get started!

version

how do we track and manage change?

when i bake, sometimes i try to make a new recipe. i’ll test out gluten free flour, an egg replacement, and i’ll mark down how i changed it, in case i want to re-use the recipe later.

in the ds world…

version

model

model_final

model_final_ACTUALLY

model_final_ACTUALLY_1

lacks context not scalable

version

versioning is useful to track changes across time, but it should also be for different implementations

we saw how versioning might be useful across time. might need to update a model days, or months from now, so it helps to have structure and context around how you version.

this is also important if you have different implementations– for example, staging and production.

really, anytime you have multiple models saved, some sort of versioning will help organize your models and make it easier to deploy later on.

version::vetiver

in r

library(vetiver)
library(pins)

model_board <- board_temp()

in python

import vetiver
import pins

model_board = board_temp(
    allow_pickle_read = True)

vetiver has a secret best friend pins, which is also available as a package in r and python.

board holds, organizes, and created metadata for almost any objects, but it has some special features when you use a vetiver model

version::vetiver

in r

library(vetiver)
library(pins)

model_board <- board_temp()

v <- vetiver_model(model, "name")

in python

import vetiver
import pins

model_board = board_temp(
    allow_pickle_read = True)

v = VetiverModel(model, "name", 
    ptype_data = mtcars)

version::vetiver

in r

library(vetiver)
library(pins)

model_board <- board_temp()

v <- vetiver_model(model, "name")

model_board %>% 
    vetiver_pin_write(v)

in python

import vetiver
import pins

model_board = board_temp(
    allow_pickle_read = True)

v = VetiverModel(model, "name", 
    ptype_data = mtcars)
vetiver_pin_write(model_board, v)

then, you write your pin to your board and pins will automatically version it for you. you can also read a specific version of the pin at a later date, and your model will be loaded right into memory!

we can see that pins helps with scale, but what about context?

version::vetiver

model_board %>% pin_meta("name")

model_board.pin_meta("name")

created: 20220719T142221Z
description: Scikit-learn  model
file: name.joblib
file_size: 1087
pin_hash: 4db397b49e7bff0b
title: 'name: a pinned LinearRegression object'
type: joblib

user:
  ptype: '{"cyl": 6.0, "disp": 160.0, "hp": 110.0, "drat": 3.9, "wt": 2.62, "qsec":
    16.46, "vs": 0.0, "am": 1.0, "gear": 4.0, "carb": 4.0}'
  required_pkgs:
  - vetiver
  - scikit-learn

with infrastructure to version our model, it is time to

deploy

when building vetiver, we had to choose WHERE to deploy these models. there’s a few options, so let’s go on a quick journey to see how we made our decision.

deploy

• in XML (with PMML)
  • flexible in integration
  • not flexible in modeling

one option was to deploy models in XML. in the baking world, this is like baking a cookie over an open flame– while fires are, very portable, they’re not very customizable in temperature

XML is best known for displaying documents on the internet, but using PMML to deploy your model gives you only limited options in models. so, this was not the right place for most data scientists.

deploy

• in XML (with PMML)

• in databases (with SQL stored procedures)

  • flexible in modeling
  • not flexible in integration

our next option was to deploy models in databases, using sql stored procedures

baking cookies in a waffle iron– works reaaalllyyyy well for some recipes, but others you just kind of end up with partially burned/partially goopy mess

this is a great option for people who have model lifecycles based around a database that you can easily interact with. but this is a bit too niche to be a best-case scenario, so we had to continue our quest find the perfect method of deploying models.

deploy

• in XML (with PMML)

• in databases (with SQL stored procedures)

• in an API (with RESTful APIs)

  • highly flexible in modeling
  • highly flexible in integration

… best friends with your OVEN, known for its predictable temperatures and easy to use interface that makes baking a great activity for all skill levels.

rest apis are interfaces that can connect applications in a standardized way.

Any model you can write, you can deploy to a rest API. You can interact in the browser to debug. You can deploy on your computer, an on-prem server, inside a docker container, or in pretty much all cloud providers. I don’t have time to go into all the different deployment options, but I know my colleague James Blair is hosting a birds of a feather session for AWS Sagemaker that includes vetiver tomorrow morning, and there’s another talk called “Yes, You can Use Python With RStudio Team” by my colleagues Xu Fei and Gagan tomorrow afternoon.

And one more reason to be excited about rest APIs–they are quite accessible for different skill levels, with visual documentation automatically created in the API endpoint, so teammates who maybe aren’t as modeling obsessed as present company are still able to discover and interact with your model, without even having to download R or Python.

vetiver

in r

library(plumber)

pr() %>%
  vetiver_api(v)

in python

api = VetiverAPI(v)
api.run()

prepopulated with

ping endpoint predict endpoint interact with the model see more information about your API

you can make predictions, including batch predictions, from your endpoint from INSIDE YOUR NOTEBOOK–feels like it is inside your environment, even though it ISNT

monitor

now that the cookies are in the API oven, we have to keep an eye on them so they don’t burn!

monitor

• monitor for data drift

does your data from when you trained your model 2 months ago look the same as today?

monitor

• monitor for data drift
• monitor for model drift

model drift when your model’s performance metrics start decaying. this is SO IMPORTANT TO TRACK. models fail silently! and they can still run with no error, even if your accuracy is zero percent–

if you are not monitoring your model in some way, you are oblivious to decay.

• i listened to waaaayyyy more jonas brothers in 2012 than i do now, and my spotify recommendation model has had to adapt to keep a customer!

monitor

• monitor for data drift
• monitor for model drift
• know what to do when things go wrong

if performance is declining, - retraining model? - new model type altogether?

if you are using model versioning, it becomes easier to roll back to the latest version

vetiver

in r

metrics <-
    augment(v, new_data = new) %>%
    vetiver_compute_metrics(
        date_col, 
        "week", 
        mpg, 
        .pred
        )

in python

metrics = vetiver.compute_metrics(
    new_data, 
    "date_col", 
    timedelta(weeks = 1), 
    metric_set, 
    "mpg", 
    "preds"
    )

vetiver monitoring looks at a model over time, since, even if time isn’t a feature used for predictions, it does affect your model.

you will start with a dataframe with information you’ve gathered since you deployed your model. vetiver needs column names for

• name of the column with the date the prediction was made

• name of column for the actual value

• name of column for predicted values

• time frame you’re wanting to aggregate (1 week, 2 weeks)

• set of metrics you wish to compute (R Side has a default, python you’ll have to specify what metrics(RMSE, R^2, etc))

vetiver

in r

# compute, then ... 

model_board %>%
    vetiver_pin_metrics(
        metrics, 
        "metrics_pin_name", 
        overwrite = TRUE
        )

in python

# compute, then ... 

vetiver.pin_metrics(
    model_board, 
    metrics, 
    "metrics_pin_name", 
    overwrite = True
    )

once computed, you can pin the dataframe of metrics to keep a log of your model performance! give it a fancy name, and you can also specify if you would like to overwrite any overlapping dates from the last time you computed/pinned metrics.

vetiver

in r

# compute and pin, then ...

vetiver_plot_metrics(metrics)

in python

# compute and pin, then ...

vetiver.plot_metrics(metrics)

finally, you can plot these metrics to get an out-of-the-box

putting it all together

best practices:

• version your model
• deploy your model
• monitor your model

but also!

• responsible reporting
• data validation
• more!

versioning, deploying, and monitoring are the three main verbs i want you all to walk away with when thinking about vetiver. but,

model cards: - partially automated rmarkdown template - relevant factors for your model: environmental, technical, or ethical

input data prototype: - validate data types at prediction time - data at the actual API endpoint does not look the same as when you trained it. (dates)

…many more best practices to keep in mind…

must be end of our time, our cookies are taking a bow…

challenge you think of what your own data science workflow…where can you add in MLOps best practices to make it better?

check out vetiver in PYTHON OR R, and come visit in the open source or pro products lounge

---

what’s next for vetiver? taking a nap! building out workflows for cloud, supporting more model types

what models are supported? tidymodels workflow caret mlr3 XGBoost ranger lm() and glm() torch scikit learn

explainability?

how does monitoring work? partially deployment related. in connect, you can run a cron job! most cloud vendors will allow you to run scripts on some sort of cadence

what if SQL? tidypredict :)