In my daily machine learning / deep learning workflow, I often want to interact with a dataset in my code. Specifically, I would like to be able to load some module/package which can

  1. make sure that I have the dataset in a specific folder
  2. represent the structure of the dataset on a higher, possible object-oriented level
  3. is homogeneous across datasets of the same class, e.g. image classification (set of images with labels, possibly bounding boxes)
  4. open-source, such that I could contribute a specification to a previously uncovered dataset
  5. that lets me specify a set of commonly used licenses to which I can comply, to filter all compatible datasets

Currently, I mainly use Python, but other languages are also in the scope of this question (Matlab, Java). Does such an API exist? If not, which ones come closest to the requirements stated above?

To give you an example of how I would expect it to feel like, see the following python code

import dataset_api as da

in_2012 = da.get('ImageNet', version='2012')
in_2017 = da.get('ImageNet', version='2017')
coco = da.get('COCO')

image_classification = da.merge([in_2012, in_2017, coco])

images = image_classification.images

image = images[0]
image.path  # the absolute file path to the image file on my disk
image.objects  # an array of object that are visible in this image

scikit-learn has a small data sets API http://scikit-learn.org/stable/datasets/index.html I imagine one can add more data sets locally. Some data sets are for classification, other for regressions.

This is the only one I know about.


There are multiple python packages with inbuilt toy-datasets for testing purposes:


TensorFlow has a nice datasets package, which meets the requirements stated above.


Not the answer you're looking for? Browse other questions tagged or ask your own question.