Holistic MLOps for better science

PyData NYC 2022

Isabel Zimmerman, Posit PBC

hello i am isabel! i have a dog named toast who i teach to do lots of silly tricks! i graduated from florida polytechnic with a degree in data science, and i decided to go back to school and am a graduate student at florida poly

to destress i do things like play video games… i wouldnt say im a GAMER, i mostly play things like mario kart with friends (if you’re not familiar with mario kart, its a game where you get to drive go karts around on different tracks, one of the most famous is called rainbow road)

MLOps is…

a set of practices to deploy and maintain machine learning models in production reliably and efficiently

and these practices can be HARD.

• i said i started with Kubernetes-based + models. after a while where i STRUGGLED with the tools at hand, i felt that, as a data scientist, the tools were not built for me. they were built for a software engineer that had pretty deep knowledge of infrastructure–and i needed to productionalize models, but i didn’t want to have to be a cloud architect as well a data scientist.

hands up, if you have deployed a model keep it up if it was a DELIGHTFUL experience

• so i changed career paths to build new tools designed specifically for data scientists to deploy models

• vetiver, which is an mlops framework to help data scientists in R and Python

put my student hat back on– when you start learning about data science, you see an image that looks something like this: you learn about tools such as –

and so when you write data science code in Python using the packages and BEST practices you’ve learned, it goes something like this:

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

df.head()

set seed for reproducibility

	like_count	funny	show_product_quickly	celebrity	danger	animals
0	1233.0	False	False	False	False	False
1	485.0	True	True	True	True	False
2	129.0	True	False	False	True	True
3	2.0	False	True	False	False	False
4	20.0	True	True	False	True	True

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

split into training and test sets

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)
oe = preprocessing.OrdinalEncoder().fit(X_train)
rf = ensemble.RandomForestRegressor().fit(oe.transform(X_train), y_train)
rf_pipe = pipeline.Pipeline([('ordinal_encoder',oe), ('random_forest', rf)])

choose the right feature engineering for the job, and train your model

put these together in a pipeline

if you develop models…

then i got to my first job and realized there’s more to think about than splitting test/training/etc you need to figure out how to share this model with teammates in a way that DOESNT include emailing your model to other people, you need to integrate it into some larger application (am i going to write this IN MY SHINY APP?), you need to ensure that it still performs well a week from now, or a month from now

you start to realize that these practices are not always enough!

you probably should operationalize them

the business values of models generally

enough data science, lets think about mario kart again.

jk, but really, its helpful to learn mlops this way.

• 50cc: slower
• 100cc:

versioning

people think of versioning, its usually in the context of git! but we version lots of different things, and mostly badly

versioning

model

model_final

model_final_final

model_final_final_actually

model_final_final_actually (1)

versioning your model is the foundation for success in machine learning deployments…

we can already see here this is not going to scale for ONE MODEL lacks context

versioning

• living in a central location
• discoverable by team
• load right into memory

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # can also be s3, azure, gcs, connect
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object
vetiver_pin_write(model_board, v)

Meta(title='ads: a pinned Pipeline object',
    description="Scikit-learn <class 'sklearn.pipeline.Pipeline'> model", 
    created='20221102T094151Z', 
    pin_hash='4db397b49e7bff0b', 
    file='ads.joblib', 
    file_size=1087, 
    type='joblib', 
    api_version=1, 
    version=VersionRaw(version='65155'), 
    name='ads', 
    user={'required_pkgs': ['vetiver', 'scikit-learn']})

• Python
• R

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp(
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")
vetiver_pin_write(model_board, v)

library(vetiver)
library(pins)

model_board <- board_temp()

v <- vetiver_model(rf, "ads")
model_board %>% 
  vetiver_pin_write(v)

writing data to the pin

know what your input data should look like

• save a piece of your data to better debug when things go wrong

trying to figure out how to solve a puzzle is a lot harder when you don’t know what the finished product looks like

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp(
    allow_pickle_read = True)

v = VetiverModel(rf, "ads", ptype_data = X_train)
vetiver_pin_write(model_board, rf)

utilizing model cards

not only good models, but good models

• summary
• documentation
• fairness

from a team at google

kinda like writing down a recipe

this feels a bit different, since it is not as maybe TECHNICALLY involved

utilizing model cards

vetiver_pin_write(model_board, v)

utilizing model cards

Model Cards provide a framework for transparent, responsible reporting. 
 Use the vetiver `.qmd` Quarto template as a place to start, 
 with vetiver.model_card()

utilizing model cards

vetiver.vetiver_pin_write(model_board, v)

vetiver.model_card()

utilizing model cards

utilizing model cards

utilizing model cards

my model doesn’t have any ethical challenges, it’s predicting youtube likes

• Also, consider the possibility of gathering feedback from those impacted by the machine learning system, especially those with marginalized identities.

• However, we strongly advise that instead of deleting any such section because you have incomplete or imprecise information, you note your own process and considerations.

The process of documenting the extent and limits of a machine learning system is part of transparent, responsible reporting.

ultimately the extent of its value depends on you

actually, my dad has a quote that he always tells me–“if you haven’t written it down, you haven’t thought it out”

deploy your model

creating model as a REST API endpoint

useful bc model can be used in memory just like you loaded it! without having to load it

also useful since API endpoints are testable,

deploy your model

deploy your model

my_api = VetiverAPI(v)

my_api.run()

deploy your model

vetiver.deploy_rsconnect(
    connect_server = connect_server, 
    board = model_board, 
    pin_name = "ads", 
    version = "59869")

deploy your model

vetiver.write_app(board=board, pin_name="ads", version = "59869")
vetiver.write_docker(app_file="app.py")

app.py file is going to read your pin, and use a vetiver model and then something called a vetiver api, which you can think of as a sort of model aware fastapi

with this docker container in hand, you can bring your model to numerous cloud services!

AWS ECR + ECS Azure container registry and app service

model doesn’t live in deployment

your model is deployed, and you are happily using it!

have to think about some sophistication in the architecture.

might think you can put model in docker container, but i’m going to try to convince you this might not be the best choice

there are a few things that are useful to keep in mind

model doesn’t live in deployment

when you start up your docker container, will load model into memory from pin in your s3 bucket, etc

model doesn’t live in deployment

your model is deployed, and you are happily using it! but, maybe you

model lives inside container, this process is immediately painful before you even update, container is larger. to update, you save model in container. if you are using versioned models inside your container, now you have two models in there (not going to be easily accessible to others to pin_read, unless you are ALSO storing them somewhere else).

model doesn’t live in deployment

ad-hoc analysis of models over time. If the models exist inside containers, those containers have to be spun up and deployed to do analysis with them over multiple versions of models.

Monitoring usually involves more dependencies but doesn’t need to be as performant.

Are you going to spin up another container from your “real” production image to use for monitoring purposes only? Probably don’t want to make HTTP calls to a real production container for your monitoring.

For monitoring, it probably doesn’t really help to get new predictions over HTTP anyway (can predict in the same computational environment as where the validation data is).

model lives outside container. you pin your model to the board that lives on, say, s3. you change the version number inside your dockerfile, redeploy.

monitoring

model is deployed a data scientist’s work is not done!

now, monitoring means somthing unique in MLOps– not necessarily looking at CPU usage, runtime, etc,

looking at statistical properties of input data or predictions

monitoring

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    [mean_absolute_error, r2_score], 
    "like_count", 
    "y_pred"
    )

vetiver.pin_metrics(
    model_board, 
    metrics, 
    "metrics_pin_name", 
    overwrite = True
    )

vetiver.plot_metrics(metrics)

we won’t look super deep at these functions right, but

monitoring

it is SO IMPORTANT TO TRACK your model’s performance metrics start decaying.

software engineering–when things went wrong, ERROR

models fail silently! and they can still run with no error, even if your accuracy is zero percent–

if you are not monitoring your model in some way, you are oblivious to decay.

what’s the difference between MLOps tools?

mlops is a happening space…if you google what the landscape looks like…

composability

• VetiverModel
• VetiverAPI

because it’s pretty

few simple tools that you are able to compose together to make complex objects

since vetiverapi is built on fastapi, can build out to be quite complex also has methods to add other POST endpoints

also is composable with other tools to build out a custom framework that works well for your team

ergonomics

• feels good to use
• works with the tools you like

one thing vetiver has worked really hard on is to lower the barrier to entry on deploying models, making this feel like a natural extension of your current data science workflow

you are still able to use the tools you want

also with leveraging pins, it makes it easy to move data between R and Python at places where this is possible

vetiver.rstudio.com or visit us at the Posit booth!

in a composable and ergonomic way