Linking preprocessing with models [closed]

Question

We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go.

The people using the models for inference need to do the same preprocessing on their data as we used on our training data.

Is there any standard way of 'versioning' the preprocessing code so that it can be linked to the models that used that version? I could use GitHub versions or tags, and maybe that's the best way. Or is there some other technique that is widely used?

Answer 1

Using GitHub versions and tags is certainly one way to go. However, there are more sophisticated techniques and tools you might want to consider:

• Data Version Control (DVC): DVC is a popular open-source tool for versioning data and preprocessing code, specifically designed for machine learning projects. It provides a Git-like interface to manage both data and code, allowing you to link specific versions of your data and preprocessing pipeline with corresponding model versions. You can learn more about DVC at https://dvc.org/.

• MLflow: MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It includes tools for tracking experiments, packaging code into reproducible runs, and sharing and deploying models. With MLflow, you can track different versions of your preprocessing code, data, and models, and switch between them when needed. You can learn more about MLflow at https://mlflow.org/.

• Docker: Another way to ensure consistency in your preprocessing is by using Docker containers. You can create a Docker image with the required dependencies and preprocessing code, and then use this image for both training and inference. This ensures that the same code and environment are used at each stage. You can then version the Docker images using tags, and store them in a container registry such as Docker Hub or Google Container Registry. You can learn more about Docker at https://www.docker.com/.

Answer 2

Experiment tracking is so important, it's very hard to work together as a team on models without it. Sounds like you'd both want to track your data lineage and track your models. Here you can see my comparison of yolov5 and yolov8 models, everything is stored for all of these models; hyperparameters, metrics, confusion matrix, images, dependencies.. I didn't store the code, but it's very easy to do. https://www.comet.com/kristenkehrer/dogs-and-cats/view/new/panels