4
$\begingroup$

I have heard the term "pipeline" used in many different contexts. Now I'm trying to bring some clarity to the terminology: What exactly is a "pipeline" in machine learning?

$\endgroup$

3 Answers 3

4
$\begingroup$

A data pipeline consists of 3 main steps

  1. data collection (e.g. you collect images of cats from different sources)
  2. data transformation (e.g. you make the images all have the same dimensions and maybe you want to convert all of them to grayscale, etc)
  3. data storage (so that you don't lose the data and can later reuse)

In machine learning, you need data to train the models. So, you could adopt a data pipeline, but not necessarily. It depends on your use case. For example, maybe you don't need to collect the data because you can download it from the Internet (although we could consider this download the data collection itself), or maybe you don't need to store it (in a database) because you will use it only once. However, you will probably need to transform it.

Anyway, data pipelines are not specific to machine learning. You can also develop them for data analysis or visualisation (so without training any ML model). There may also be other types of pipelines (e.g. people may refer to the steps they followed to train a certain model as a pipeline - see e.g. how InstructGPT was created here - that's also a pipeline, in the sense that it's a chain of operations, where the output of the previous one is feed to the next one), but data pipelines are definitely the ones I am most familiar with, because I also work with them, and they have a more or less standard definition.

This IBM article provides a more detailed explanation.

$\endgroup$
3
$\begingroup$

A "pipeline" typically refers to a chain of methods where the output of the one is used as the input of another method.

This could be, e.g., a "preprocessing pipeline" where several data cleaning and data preparation steps are sequentially performed.

A more complete "ML Pipeline" could consist of data prepocessing, sampling of the data (e.g. for obtaining folds for cross validation or resampling) and a ML training and testing. Often, such a pipeline also records all used parameters.

Other than that there is no strict definition, as far as I know.

$\endgroup$
0
$\begingroup$

A "pipeline" in machine learning refers to a sequence of data processing and modeling steps that transform raw data into predictions or insights. Pipelines often include stages such as data preprocessing, feature extraction, feature scaling, model training, and model evaluation[1].

You may see in the context of the transformers library, the pipeline class simplifies the process of using pre-trained models for common NLP tasks like text classification, named entity recognition, and text generation. It provides an easy-to-use interface that handles tokenization, input processing, and output decoding in a boxed fashion.

Considering the Pipeline class in scikit-learn, the Pipeline class helps automate and organize machine learning workflows by allowing users to define a sequence of data transformations and models. It simplifies the process of applying multiple preprocessing steps and ensures that the same preprocessing is applied to both training and test data.

It is nothing distant from the definition in the context of generic computing.

is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion.

You might say that ML pipeline is a natural extension of data pipeline, which may have been more known and understood better.


[1] See: https://neptune.ai/blog/ml-pipeline-architecture-design-patterns

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .