Extending the data science workflow: {vetiver} and {pins}

R-Ladies Rome

Isabel Zimmerman || Posit, PBC

while you listen

• if you would like, there is a very small code-along at the end of this talk!
• to prepare, open up RStudio, and have vetiver, pins, tidymodels, and ranger packages

vetiver pins tidymodels ranger plumber

this talk is going to show python code, but don’t worry! the concepts i’ll show you are the exact same

who am I?

@isabelizimm

@isabel-zimmerman

work at posit as software engineer for two years–i did the whole full time school and work thing and graduated last year!

graduated w data science degree, yippee!!

when i started learning about data science, you see an image that looks something like this: you learn about tools such as –

and so when you write data science code in Python using the packages and BEST practices you’ve learned, it goes something like this:

best practices in data science lifecycle

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

set seed for reproducibility

best practices in data science lifecycle

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

split into training and test sets

best practices in data science lifecycle

import pandas as pd
import numpy as np

np.random.RandomState(500)
raw = pd.read_csv('https://bit.ly/3sWty5A')
df = raw[["like_count", "funny", "show_product_quickly", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)
oe = preprocessing.OrdinalEncoder().fit(X_train)
rf = ensemble.RandomForestRegressor().fit(oe.transform(X_train), y_train)
rf_pipe = pipeline.Pipeline([('ordinal_encoder',oe), ('random_forest', rf)])

choose the right feature engineering for the job, and train your model

put these together in a pipeline

many of the problems I faced were different, though

then i got to my first job and realized there’s more to think about than splitting test/training/etc you need to figure out how to share this model with teammates in a way that DOESNT include emailing your model to other people, you need to integrate it into some larger application (am i going to write this IN MY SHINY APP?), you need to ensure that it still performs well a week from now, or a month from now

you start to realize that these practices are not always enough!

if you develop models…

you can operationalize them

the “real world” value of models often times comes from integrating

my advice for you, is to bring your models outside!!

you can learn to operationalize models using a set of practices called MLOps—

MLOps is…

a set of practices to deploy and maintain machine learning models in production reliably and efficiently

and these practices can be HARD.

these practices are to …

with Kubernetes-based + models. after a while where i STRUGGLED with the tools at hand, i felt that, as a data scientist, i needed to productionalize models, but i didn’t want to have to be a cloud architect as well a data scientist.

and while i don’t think we should be oblivious to everything in the dev ops world, i do think that there is space for tools to help data scientists more effectively communicate with their IT or devops teams, while still feeling ergonomic for data scientists.

• so i changed career paths to build new tools to make mlops feel like an extension of a data scientist’s current workflow. and this tool, is named vetiver.

i do work for posit, — this company was formerly known as RStudio, so it may come as no surprise that vetiver is a python package, but also an R package as well.

what are some tasks vetiver helps with?

• model versioning
• model deployment
• model monitoring

Each open source project falls into one (or sometimes multiple) different categories to fulfill these different tasks: all-in-one, data pipeline, infrastructure, modeling and training, monitoring, and serving

versioning

people think of versioning, its usually in the context of git! but we version lots of different things, and mostly badly

versioning

model

model_final

model_final_final

model_final_final_actually

model_final_final_actually (1)

versioning your model is the foundation for success in machine learning deployments…

we can already see here this is not going to scale for ONE MODEL lacks context

it would be nice if my models: - lived in a central location - were discoverable by my team - loaded right into memory

{pins}

pins package publishes data, models, and other R objects, making it easy to share them across projects and with your colleagues

• can be pinned in a variety of locations (AWS, GCP, GitHub, Posit Connect, and more!)
• good for sharing models, data, files, .Rds objects
• good for a pipelines where data is read and/or updated
• bad when multiple people are writing data at once

• Good use for pins: an ETL pipeline that stores a model or summarized dataset once a day
• Bad use for pins: a Shiny app that collects data from users, who may be using the app at the same time

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

train a model FIRST

inside this vetiver model object

• input data prototype

save a piece of your data to better debug when things go wrong

trying to figure out how to solve a puzzle is a lot harder when you don’t know what the finished product looks like

• what packages are needed to recreate this model
• metadata about the model

also with leveraging pins, it makes it easy to move data between R and Python at places where this is possible

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # can also be s3, azure, gcs, connect
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads")

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp( # create place for models to be stored
    allow_pickle_read = True)

v = VetiverModel(rf_pipe, "ads") # create deployable model object
vetiver_pin_write(model_board, v)

Meta(title='ads: a pinned Pipeline object',
    description="Scikit-learn <class 'sklearn.pipeline.Pipeline'> model", 
    created='20221102T094151Z', 
    pin_hash='4db397b49e7bff0b', 
    file='ads.joblib', 
    file_size=1087, 
    type='joblib', 
    api_version=1, 
    version=VersionRaw(version='65155'), 
    name='ads', 
    user={'required_pkgs': ['vetiver', 'scikit-learn']})

import pins
from vetiver import VetiverModel, vetiver_pin_write

model_board = pins.board_temp(
    allow_pickle_read = True)

v = VetiverModel(rf, "ads", ptype_data = X_train)
vetiver_pin_write(model_board, rf)

utilizing model cards

not only good models, but good models

• summary
• documentation
• fairness

from a team at google

kinda like writing down a recipe

this feels a bit different, since it is not as maybe TECHNICALLY involved

utilizing model cards

vetiver_pin_write(model_board, v)

utilizing model cards

Model Cards provide a framework for transparent, responsible reporting. 
 Use the vetiver `.qmd` Quarto template as a place to start, 
 with vetiver.model_card()

utilizing model cards

vetiver.vetiver_pin_write(model_board, v)
vetiver.model_card()

utilizing model cards

utilizing model cards

utilizing model cards

my model doesn’t have any ethical challenges, it’s predicting youtube likes

instead of deleting any such section because you have incomplete or imprecise information, you note your own process and considerations.

actually, my dad has a quote that he always tells me–“if you haven’t written it down, you haven’t thought it out”

model is saved in a central location, yay! BUT we want the model to be able to be used by others, without actually having to download it

deploy your model

deploy your model

creating model as a REST API endpoint

useful bc model can be used in memory just like you loaded it! without having to load it

also useful since API endpoints are testable, and work with json calls so you can call a Python model from R (or vice versa), if the input data is the same

deploy your model

deploy your model

my_api = VetiverAPI(v)
my_api.run()

deploy your model

vetiver.deploy_connect(
    connect_server = connect_server, 
    board = model_board, 
    pin_name = "ads", 
    version = "59869")

vetiver.prepare_docker(board=board, pin_name="ads")

generates the files needed to

app.py / plumber.R file Dockerfile requirement.txt or renv lockfile

monitoring

model is deployed a data scientist’s work is not done!

now, monitoring means somthing unique in MLOps– not necessarily looking at CPU usage, runtime, etc,

looking at statistical properties of input data or predictions

monitoring

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    [mean_absolute_error, r2_score], 
    "like_count", 
    "y_pred"
    )

vetiver.pin_metrics(
    model_board, 
    metrics, 
    "metrics_pin_name", 
    overwrite = True
    )
    
vetiver.plot_metrics(metrics)

we won’t look super deep at these functions right, but

monitoring

it is SO IMPORTANT TO TRACK your model’s performance metrics start decaying.

software engineering–when things went wrong, ERROR

models fail silently! and they can still run with no error, even if your accuracy is zero percent–

if you are not monitoring your model in some way, you are oblivious to decay.

Why should I be excited about vetiver?

Composability

• Internally, with VetiverAPI and VetiverModel
• Externally, leveraging the tools vetiver is built on

Ergonomics

• feels good to use
• works with the tools you like

few simple tools that you are able to compose together to make complex objects

since vetiverapi is built on fastapi/plumber, can build out to be quite complex also has methods to add other POST endpoints

also is composable with other tools to build out a custom framework that works well for your team

---

one thing vetiver has worked really hard on is to lower the barrier to entry on deploying models, making this feel like a natural extension of your current data science workflow

you are still able to use the tools you want

resources

• Operationalizing Machine Learning: An Interview Study
• vetiver.rstudio.com
• My talk about vetiver at rstudio::conf(2022)
• Julia Silge’s screencast on vetiver+Docker