What makes a 'good' dataset

Question

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.

While datasets from external sources are of course marketed as beeing 'good' or 'high-quality' (and in most cases not explained on how the authors come to this conclusion), creating a dataset yourself doesn't come with these labels.

This brings me to my questions: How can one (objectively) quantify the quality of a dataset for a given problem?

This, of course, includes some points which are more or less 'accepted' within the community, e.g. the dataset has to contain enough datapoints for every modeled state (which leads to the question of what is 'enough' ...), or that the datapoints for each modeled state should be roughly equally split (e.g. a dataset for cat/dog image discrimination would not work well with one dog image and 10k cat images) and so on.

I recognize that this is a rather open ended and maybe even philosophical question, but I believe that, given the importance of data for ML (and other disciplines), I am in need of an objective way to evaluate my datasets in relation to the task at hand and determine their quality in an objective way. Also my goal here is clearly ML oriented, but since this topic is not only valid in the ML context (and ML is in its core more or less complex statistics), I don't want to restrict this only to the ML, but to datasets overall

Answer 1

This is not a simple answer, and I think it really depends the goal, quality is not clearly defined and can vary. However there are some points that are commonly seen as positive.

The quantity

As you said, having enough data is important to train models, and it is even better if the dataset is balanced for classification. This means that it i spossible to build more complex models, and then learn more from the dataset.

Learn more does not mean to learn from more data, but from more diverse data. If you take the example of dog/cat classification, a larger dataset will probably have more angles (from above, behind or with an obscured view), and more cat/dog races which means that the algorithm can learn all these differences that needs a larger amount of data.

The main point of quantity is not how much data, but how much of the input space is covered.

A clean dataset

The first step of every project is often cleaning the data and preprocessing it, which can be avoided if the dataset is of better "quality". This means that all instances are correctly labeled, there is no missing value, no duplicates, the data types are the same...

There the data is synthetic and can be used nearly as is, without need of further work to process these.

The ambiguous points

If a dataset present the attributes above I'd say it's quality is already great, but there are some additional points that can be considered. Some attributes of the dataset can vary (for example image size/resolution, background noise for audios, typo in language corpus...).

The quality of the instances with such difference can be said to be low, if example if for example only the tail of the dog can be seen. However the issue is that real world data is not perfect.

In some cases you can pre-process these differences or perturbations and correct them, but sometimes you can't. There some people will say it's part of the challenge to deal with these, some will say it is useless. There it depends on your goal and subjective point of view to decide if these inconsistency is needed or more harmful.

To add a more personal thought, I'd say that when building a production model it is important to keep these as they reflect the real use and the perturbations brought by real data.

Answer 2

For the dataset to be useful for the community it should be publicly available in a stable domain and establish what the test set is (if you can have a hidden one and make a webpage to evaluate the submissions that's that's better). It's also useful to have baseline performance numbers.

The dataset doesn't have to be balanced. High-quality datasets are regarded as such if the label noise is small (note that annotators do fail), the data are not corrupted and care has been taken regarding potentially harmful biases. Of course, having the dataset checked and reviewed by experts in the field is desirable. Creating a good dataset involves lots of work.