Questions tagged [datasets]
For questions related to sets of data and their use in AI.
229
questions
0
votes
1answer
22 views
An online editor that allows data labeling format
I have a set of students (~20) that will work on annotating data for an NLP project.
The annotation task will be as in the following:
...
1
vote
0answers
34 views
Batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch ...
1
vote
2answers
32 views
Train Validation Test Splitting After or Before Data Augmentation?
I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of ...
0
votes
1answer
13 views
How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?
I have a question
I have had a dataset below
...
0
votes
0answers
12 views
Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset
So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
0
votes
1answer
34 views
1
vote
1answer
25 views
How can I address missing values for LSTM?
I'm a student and writing my first paper for submission on conference. I have a question
there is a dataset below. this is temporal-spatial dataset.
...
0
votes
0answers
11 views
Simple example for side information
Many research papers contain the phrase "side information". After a search on side information, I got the following definition from here.
In many problems of machine learning and computer ...
0
votes
1answer
52 views
Can the developers of the libraries like TensorFlow have access to my dataset when I develop the model on my PC?
When I train a model with my dataset on my PC using Tensorflow, Keras, scikit-learn, Pandas or NumPy,
do the developers of these libraries can gain access to my dataset?
(May be when I train the model,...
1
vote
1answer
54 views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
0
votes
0answers
12 views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
1
vote
0answers
36 views
Identifying rotating and resizing letters with background noise
I'm trying to complete a captcha, and here is what it looks like:
Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (...
0
votes
0answers
17 views
Image regression - estimating sensors from images
I am trying to use images to predict the sensor data of a racing game.
Being a bit of a newcomer I have multiple questions. All help/suggestion is appreciated.
Dataset
The dataset looks something like:...
0
votes
0answers
10 views
Should we create a label mean group of small nearby objects in object detection?
I'm working on object detection models and my dataset sometimes has a lot of small objects (stay far from the scene) (overlapping and nearby) which is really annoying in annotating (it's too small and ...
1
vote
0answers
11 views
Split on dataset with some shared features?
I have a dataset with financial stock data, some of the features are shared, for example daily gold prices, while the stock price for each individual stock is different, the gold price would be the ...
1
vote
0answers
20 views
Pixel values of segmap in multi-class semantic segmentation
I'm preparing a dataset for a multiclass semantic segmentation using U-Net like architecture. To be precise, I've got it ready but a question came to my mind. How does pixel values of a segmentation ...
0
votes
1answer
32 views
Cnn for Combination of both digits and letters(small and capital) [closed]
Hi I am new to machine learning can anyone suggest open dataset consists of both digits and letters(small,capital)
I want images consisisting of both digits and letters to train my cnn model and make ...
0
votes
0answers
21 views
Is it possible to do object detection on an object classification dataset?
I'm new to computer vision, which I find fascinating. I wonder whether it is possible or if there has been any research into going from object recognition data to object detection. In other words, ...
0
votes
0answers
90 views
How does pairwise comparison training work in XGBoost's XGBRanker?
I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well.
I want to find out how the XGBRanker ...
2
votes
5answers
71 views
How do I increase the size of an (almost) balanced dataset?
I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
1
vote
1answer
34 views
How to source training data in ML for information security?
A company entrusts a Data Scientist with the mission of processing and valuing data for the research or treatment of events related to traces of computer attacks. I was wondering how would he get the ...
0
votes
0answers
10 views
Predicting training time using randomly generated datasets
Given a dataset, I need to predict the amount of time it will take to fit a model to it. I plan to do this by creating a csv containing the logs of previously fit models, and passing that data itself ...
1
vote
1answer
30 views
Can residual connections be beneficial when we have a small training dataset?
I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is ...
0
votes
0answers
16 views
Why doesn't U-Net work with images different from the dataset?
I have implemented a U-Net, similar to this implementation, but for a different dataset, this one, to segment roads.
It works fine using the test folder images, but, for example, when I pick a print ...
3
votes
1answer
61 views
Why different images of the same person, under some restrictions, are in a 50 dimension manifold?
In this lecture (starting from 1:31:00) the professor says that the set of all images of a person lives in a low dimensional surface (compared the the set of all possible images). And he says that the ...
1
vote
0answers
18 views
Theoretical limits on correlation between classification algorithm performances
Are there any known theoretical bounds, or at least heuristic approaches, regarding the relation or correlation between the performances of any two different classification algorithms?
For example, ...
2
votes
1answer
65 views
Would it be possible to determine the dataset a neural network was trained on?
Let's say we have a neural network that was trained with a dataset $D$ to solve some task. Would it be possible to "reverse-engineer" this neural network and get a vague idea of the dataset $...
3
votes
1answer
62 views
How to treat (label and process) edge case inputs in machine learning?
In every computer vision project, I struggle with labeling guidelines for border cases. Benchmark datasets don't have this problem, because they are 'cleaned', but in real life unsure cases often ...
1
vote
1answer
102 views
What dataset might Elon Musk's Dall-E have used?
Dall-E, it can generate many imaginative images from the description, even some peculiar images, how did they actually create this kind of dataset to train this AI , because there is not much of that ...
2
votes
0answers
56 views
In few-shot classification, should I use my custom dataset as the validation dataset and mini-ImageNet as the training dataset?
I am new to few-shot learning, and I wanted to get a hands-on understanding of it, using Reptile algorithm, applied to my custom dataset.
My custom dataset has 30 categories, with 5 images per ...
4
votes
1answer
132 views
How should I generate datasets for a SARSA agent when the environment is not simple?
I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
1
vote
0answers
40 views
Is training on single game each time appropriate for an agent to learn to play checkers
I was facing a problem I mentioned in a previous question but after a while, I realize that maybe the problem in the dataset not in the learning rate.
I build the dataset from white positions only i.e ...
0
votes
0answers
19 views
Where can a dataset of relationship between images be used?
I'm making a platform that will collect data about the relationship between different images. For example, if I have three images: a Christmas tree, a gift and Santa... then these will be connected by ...
1
vote
0answers
13 views
Is there a pre-trained network trained on RGB-D (4) channels? [closed]
The most used pre-trained networks for computer vision (e.g. ResNet50) are trained on 3 channels (RGB). At the same time, many cameras used in robotics return RGB-D outputs, that is including depth ...
2
votes
2answers
30 views
Why can't we combine both training and validation data, given that both types of data are used for developing the model?
Sorry if I sound confused. I read that data to be fed to a machine are divided into training, validation and test data. Both training and validation data are used for developing the model. Test data ...
4
votes
1answer
82 views
Can we use ML to do anything else other than predicting (in the case of mathematical problems)?
(The math problem here just serves as an example, my question is on this type of problems in general).
Given two Schur polynomials, $s_\mu$, $s_\nu$, we know that we can decompose their product into a ...
0
votes
0answers
41 views
Is there any rule of thumb to determine the amount of data needed to train a CNN
I am training an AlexNet Convolutional Neural Network to classify images in a dataset. I want to know if there is any general rule for using data augmentation in training a neural network. How can I ...
2
votes
1answer
72 views
How robust are deep networks to class imbalance?
Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling ...
0
votes
0answers
50 views
What could be a good $\mathcal{R}$ dataset in the article “Old Photo Restoration via Deep Latent Space Translation”?
There are three domains in this article: Old Photo Restoration via Deep Latent Space Translation.
The real old pictures noted by $\mathcal{R}$,
the artificial old pictures noted by $\mathcal{X}$, and
...
1
vote
1answer
73 views
Is it okay to think of any dataset in artificial intelligence as a mathematical set?
A dataset is a collection of data points. It is known that the data points in the dataset can repeat. And the repetition does matter for building AI models.
So, why does the word dataset contain the ...
1
vote
1answer
277 views
Should we also shuffle the test dataset when training with SGD?
When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples ...
1
vote
1answer
59 views
What is the reason for taking tuples as vectors rather than points?
Across the literature of artificial intelligence, especially machine learning, it is normal to treat the tuples of datasets as vectors.
Although there is a convention to treat them as data points. ...
0
votes
0answers
34 views
Transforming neural network target values before training
Consider the scenario in which I am measuring certain $f(a,x)$, which i want to be the target value for some related input $g(a,x)$. In other words, I am trying to map
$$g(a,x)\Rightarrow f(a,x)$$
I ...
0
votes
1answer
69 views
Is it possible to know the distance objects are from camera based on only knowing one object's height?
I am doing a project where I have to know distance a particular object is from camera. In the photo I only know one of the object's height, but I don't know how far away that object is and I don't ...
1
vote
0answers
29 views
How to find distance between 2 points when dimensions are all of different nature?
I have a dataset with four features:
the x coordinate
the y coordinate
the velocity magnitude
angle
Now, I want to measure the distance between two points in the dataset, taking into account the ...
0
votes
0answers
79 views
Multi-label dataloading bottleneck Pytorch
I am trying to write custom dataset and dataloader for pascal-voc-2007. It is a multi-label classification problem. There is csv file to hold the name of the images and their corresponding labels. I ...
1
vote
0answers
73 views
Which loss function to choose for imbalanced datasets?
For imbalanced datasets (either in the context of computer vision or NLP), from what I learned, it is good to use a weighted log loss. However, in competitions, the people who are in top positions are ...
1
vote
1answer
71 views
What are some examples of functions that machine learning models compute?
My simple understanding of AI is that it is based on a mathematical model of a problem. If I understood correctly, the model is a polynomial equation and its weights are calculated by training the ...
0
votes
1answer
23 views
Can a neural network be trained on a dataset containing only values for true output for a classification problem?
I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points ...
1
vote
0answers
45 views
Could the neural network automatically calculate and get different one-to-many quantities relative to their parent quantity?
Let's say I have a primary dataset that its secondary dataset is hundreds to match and group like an one-to-many relationship.
I'm new in this world of the AI but my problem is that many child groups ...
