Questions tagged [datasets]
For questions related to sets of data and their use in AI.
242
questions
0
votes
0answers
9 views
Positional Encoding for adjacency matrix
I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
3
votes
1answer
56 views
How to determine the quality of synthetic data?
I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums.
I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
1
vote
0answers
24 views
Recommended way to spilt image sequence for training/validation/testing
For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by.
Now what's the ...
1
vote
1answer
28 views
How do you handle unbalanced image datasets?
I have an image data set on which I am training a CNN. The data set is slightly unbalanced. So, my solution up till now was to delete some images of the majority class.
But I now realize that there ...
8
votes
3answers
704 views
Is it okay to use publicly available Instagram videos to train an AI?
Since I haven't found any good training data for my university project, I want to use pictures and videos from public Instagram profiles. Am I allowed to do that?
1
vote
0answers
45 views
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$...
1
vote
1answer
43 views
How can we "draw i.i.d" from any probability distribution?
Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges
...
0
votes
0answers
23 views
Modelling of output neuron for mixed features?
Dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using neural network. So, input to my ...
0
votes
1answer
39 views
Why disentangling the features of variation in representation?
Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al.
It has been hypothesized, ...
0
votes
0answers
36 views
What are mathematically the factors of variation in deep learning?
The following paragraph from an answer tells us about factors of variation
Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
0
votes
1answer
33 views
How to handle an unbalanced dataset when training object detection algorithms?
I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024.
Within those images I have the ...
0
votes
0answers
34 views
What's the best way to feed stories to a neural network?
I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because ...
0
votes
2answers
37 views
Why data required for hyperparameter tuning is considered as an additional data?
Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following ...
1
vote
0answers
30 views
Is it true that real world data is highly discontinuous?
A function $f$ is said to be continuous at a point $c$ if it satisfies three properties:
Should be defined at the point $c$
Left and right-hand limits at $c$ must be equal i.e., the limit must exist
...
0
votes
0answers
20 views
Adding data to training results in loss random peaks
I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what ...
0
votes
0answers
19 views
Which Neural Network Topology to choose, are Transformers suitable?
I have a regression problem and I am not quite sure which architecture to choose. I never worked with transformers before, but I generally understand how they work and I think they might be suitable. ...
0
votes
1answer
20 views
Data analysis before feeding to ML pipeline
I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the ...
0
votes
0answers
13 views
Weight for Samples on SVM (Support Vector Machine)
There is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library).
If I use the option ...
1
vote
1answer
229 views
Parallelised my training or the dataset
I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA, some training features might be the pivot price values and the set of the ...
1
vote
0answers
22 views
Training on the dataset in parts vs training on the whole dataset
What is the difference between these two situations? are they the same ?
#1 : train a model 20 epochs on the whole dataset
#2 : divide dataset into n-parts then train the model 20 epochs on each part
...
0
votes
1answer
27 views
An online editor that allows data labeling format [closed]
I have a set of students (~20) that will work on annotating data for an NLP project.
The annotation task will be as in the following:
...
1
vote
0answers
46 views
Batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch ...
1
vote
2answers
36 views
Train Validation Test Splitting After or Before Data Augmentation?
I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of ...
0
votes
1answer
16 views
How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?
I have a question
I have had a dataset below
...
0
votes
0answers
12 views
Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset
So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
0
votes
1answer
45 views
1
vote
1answer
28 views
How can I address missing values for LSTM?
I'm a student and writing my first paper for submission on conference. I have a question
there is a dataset below. this is temporal-spatial dataset.
...
0
votes
0answers
12 views
Simple example for side information
Many research papers contain the phrase "side information". After a search on side information, I got the following definition from here.
In many problems of machine learning and computer ...
1
vote
1answer
63 views
Can the developers of the libraries like TensorFlow have access to my dataset when I develop the model on my PC?
When I train a model with my dataset on my PC using Tensorflow, Keras, scikit-learn, Pandas or NumPy,
do the developers of these libraries can gain access to my dataset?
(May be when I train the model,...
1
vote
1answer
95 views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
0
votes
0answers
12 views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
2
votes
0answers
39 views
Identifying rotating and resizing letters with background noise
I'm trying to complete a captcha, and here is what it looks like:
Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (...
0
votes
0answers
19 views
Image regression - estimating sensors from images
I am trying to use images to predict the sensor data of a racing game.
Being a bit of a newcomer I have multiple questions. All help/suggestion is appreciated.
Dataset
The dataset looks something like:...
0
votes
0answers
10 views
Should we create a label mean group of small nearby objects in object detection?
I'm working on object detection models and my dataset sometimes has a lot of small objects (stay far from the scene) (overlapping and nearby) which is really annoying in annotating (it's too small and ...
1
vote
0answers
11 views
Split on dataset with some shared features?
I have a dataset with financial stock data, some of the features are shared, for example daily gold prices, while the stock price for each individual stock is different, the gold price would be the ...
1
vote
0answers
22 views
Pixel values of segmap in multi-class semantic segmentation
I'm preparing a dataset for a multiclass semantic segmentation using U-Net like architecture. To be precise, I've got it ready but a question came to my mind. How does pixel values of a segmentation ...
0
votes
1answer
34 views
Cnn for Combination of both digits and letters(small and capital) [closed]
Hi I am new to machine learning can anyone suggest open dataset consists of both digits and letters(small,capital)
I want images consisisting of both digits and letters to train my cnn model and make ...
0
votes
0answers
21 views
Is it possible to do object detection on an object classification dataset?
I'm new to computer vision, which I find fascinating. I wonder whether it is possible or if there has been any research into going from object recognition data to object detection. In other words, ...
0
votes
0answers
128 views
How does pairwise comparison training work in XGBoost's XGBRanker?
I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well.
I want to find out how the XGBRanker ...
2
votes
5answers
81 views
How do I increase the size of an (almost) balanced dataset?
I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
1
vote
1answer
34 views
How to source training data in ML for information security?
A company entrusts a Data Scientist with the mission of processing and valuing data for the research or treatment of events related to traces of computer attacks. I was wondering how would he get the ...
0
votes
0answers
10 views
Predicting training time using randomly generated datasets
Given a dataset, I need to predict the amount of time it will take to fit a model to it. I plan to do this by creating a csv containing the logs of previously fit models, and passing that data itself ...
1
vote
1answer
35 views
Can residual connections be beneficial when we have a small training dataset?
I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is ...
0
votes
0answers
16 views
Why doesn't U-Net work with images different from the dataset?
I have implemented a U-Net, similar to this implementation, but for a different dataset, this one, to segment roads.
It works fine using the test folder images, but, for example, when I pick a print ...
3
votes
1answer
61 views
Why different images of the same person, under some restrictions, are in a 50 dimension manifold?
In this lecture (starting from 1:31:00) the professor says that the set of all images of a person lives in a low dimensional surface (compared the the set of all possible images). And he says that the ...
1
vote
0answers
18 views
Theoretical limits on correlation between classification algorithm performances
Are there any known theoretical bounds, or at least heuristic approaches, regarding the relation or correlation between the performances of any two different classification algorithms?
For example, ...
2
votes
1answer
68 views
Would it be possible to determine the dataset a neural network was trained on?
Let's say we have a neural network that was trained with a dataset $D$ to solve some task. Would it be possible to "reverse-engineer" this neural network and get a vague idea of the dataset $...
3
votes
1answer
67 views
How to treat (label and process) edge case inputs in machine learning?
In every computer vision project, I struggle with labeling guidelines for border cases. Benchmark datasets don't have this problem, because they are 'cleaned', but in real life unsure cases often ...
1
vote
1answer
108 views
What dataset might Elon Musk's Dall-E have used?
Dall-E, it can generate many imaginative images from the description, even some peculiar images, how did they actually create this kind of dataset to train this AI , because there is not much of that ...
2
votes
0answers
68 views
In few-shot classification, should I use my custom dataset as the validation dataset and mini-ImageNet as the training dataset?
I am new to few-shot learning, and I wanted to get a hands-on understanding of it, using Reptile algorithm, applied to my custom dataset.
My custom dataset has 30 categories, with 5 images per ...
