In my daily machine learning / deep learning workflow, I often want to interact with a dataset in my code. Specifically, I would like to be able to load some module / package which can
- make sure that I have the dataset in a specific folder
- represent the structure of the dataset on a higher, possible object oriented level
- is homogeneous across datasets of the same class, e.g. image classification (set of images with labels, possibly bounding boxes)
- open source, such that I could contribute a specification to a previously uncovered dataset
- that let's me specify a set of commonly used licenses to which I can comply, to filter all compatible datasets
Currently I mainly use Python, but other languages are also in the scope of this question (Matlab, Java). Does such an API exist? If not, which ones come closest to the requirements stated above?
To give you an example of how I would expect it to feel like, see the following python code
import dataset_api as da in_2012 = da.get('ImageNet', version='2012') in_2017 = da.get('ImageNet', version='2017') coco = da.get('COCO') image_classification = da.merge([in_2012, in_2017, coco]) images = image_classification.images image = images image.path # the absolute file path to the image file on my disk image.objects # an array of object that are visible in this image
Note: this question is also posted on datascience.stackexchange.com. I wasn't sure which exchange better fits the question.