Questions tagged [datasets]
For questions related to sets of data and their use in AI.
232
questions
0
votes
0answers
28 views
What are the "per image" annotations that are generally used for image datasets in AI?
Computer vision is highly benefited by AI algorithms. Image data is abundantly available. There are different varieties of tasks such as image classification, prediction, segmentation, generation, ...
-1
votes
0answers
24 views
How can I perform analysis with this given data
I'm currently working on an AI playing Mandarin using Minimax. I can extract this dictionary from a game where 2 AI with different depth play with each other:
...
0
votes
0answers
31 views
What is all necessary types of data for a bidirectional RNN to learn embeddings?
Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level.
In order to train a CNN for the classification tasks, images, and labels/outputs are ...
0
votes
0answers
6 views
Predicting single floats based on set of 2 feature arrays each of 100 values
I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples.
The problem is that my labels are single floats (...
1
vote
0answers
19 views
What are the types of data in which the order of instances does matter?
In general, the order of instances in the datasets that are used in machine learning is immaterial. But there are exceptions. Timeseries data is one such exception I know. Consider the following two ...
0
votes
0answers
9 views
What to predict in a limited transaction dataset?
I have been given a task with a real transaction dataset. The task is to predict something using either logistic regression or simple binary classification.
The columns are as follow:
Transaction ID
...
1
vote
0answers
25 views
Does the Frechet Inception Distance (FID) consider color?
I was wondering if the Frechet inception distance
for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale.
I know that it depends on the feature ...
1
vote
1answer
52 views
Is there any simple example for volumetric data except from physics and medicine?
Recently I heard about the term volumetric data. The definition for volumetric data is as follows
#1: Definition
Volumetric data is typically a set S of samples $(x, y, z, v)$,
representing the value ...
2
votes
0answers
19 views
Do any practical deep learning algorithms deal with tensors containing non-real entries?
In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers.
Other than both mentioned above, there may be some other real-...
0
votes
1answer
46 views
Generating a dataset from data with "assumed" lables
I've got a task similar to the following:
Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
1
vote
1answer
36 views
Can I use the transformers for the prediction of historical data?
Can I use the transformers for the prediction of wind power with the historical data?
Dataset
Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
0
votes
0answers
16 views
What are the pros and cons of using a normal positional encoding in an adjacency matrix?
I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
3
votes
1answer
71 views
How to determine the quality of synthetic data?
I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums.
I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
1
vote
0answers
26 views
Recommended way to spilt image sequence for training/validation/testing
For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by.
Now what's the ...
2
votes
1answer
36 views
How do you handle unbalanced image datasets?
I have an image data set on which I am training a CNN. The data set is slightly unbalanced. So, my solution up till now was to delete some images of the majority class.
But I now realize that there ...
8
votes
3answers
765 views
Is it okay to use publicly available Instagram videos to train an AI?
Since I haven't found any good training data for my university project, I want to use pictures and videos from public Instagram profiles. Am I allowed to do that?
0
votes
0answers
61 views
What are the different possible usages of the word "i.i.d" in machine learning?
The acronym "iid" stands for "independent and identically distributed". It is a property of a sequence of random variables. You can read here for more details. This question is ...
1
vote
1answer
80 views
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$...
1
vote
2answers
92 views
How can we "draw i.i.d" from any probability distribution?
Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges
...
0
votes
0answers
29 views
Modelling of output neuron for mixed features?
A dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using a neural network. So, input ...
0
votes
1answer
43 views
Why disentangling the features of variation in representation?
Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al.
It has been hypothesized, ...
0
votes
0answers
38 views
What are mathematically the factors of variation in deep learning?
The following paragraph from an answer tells us about factors of variation
Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
0
votes
1answer
41 views
How to handle an unbalanced dataset when training object detection algorithms?
I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024.
Within those images I have the ...
0
votes
0answers
40 views
What's the best way to feed stories to a neural network?
I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because ...
0
votes
2answers
45 views
Why data required for hyperparameter tuning is considered as an additional data?
Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following ...
1
vote
0answers
33 views
Is it true that real world data is highly discontinuous?
A function $f$ is said to be continuous at a point $c$ if it satisfies three properties:
Should be defined at the point $c$
Left and right-hand limits at $c$ must be equal i.e., the limit must exist
...
0
votes
0answers
23 views
Adding data to training results in loss random peaks
I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what ...
0
votes
0answers
22 views
Which Neural Network Topology to choose, are Transformers suitable?
I have a regression problem and I am not quite sure which architecture to choose. I never worked with transformers before, but I generally understand how they work and I think they might be suitable. ...
0
votes
1answer
24 views
Data analysis before feeding to ML pipeline
I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the ...
0
votes
0answers
15 views
Weight for Samples on SVM (Support Vector Machine)
There is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library).
If I use the option ...
1
vote
1answer
251 views
Is there a way to parallelise the RL training on multiple stocks to avoid the memory issue?
I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA
some training features might be the pivot price values and the set of the ...
1
vote
0answers
22 views
Training on the dataset in parts vs training on the whole dataset
What is the difference between these two situations? are they the same ?
#1 : train a model 20 epochs on the whole dataset
#2 : divide dataset into n-parts then train the model 20 epochs on each part
...
0
votes
1answer
30 views
An online editor that allows data labeling format [closed]
I have a set of students (~20) that will work on annotating data for an NLP project.
The annotation task will be as in the following:
...
1
vote
0answers
47 views
Batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch ...
1
vote
2answers
73 views
Train Validation Test Splitting After or Before Data Augmentation?
I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of ...
0
votes
1answer
20 views
How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?
I have a question
I have had a dataset below
...
0
votes
0answers
16 views
Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset
So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
1
vote
1answer
29 views
How can I address missing values for LSTM?
I'm a student and writing my first paper for submission on conference. I have a question
there is a dataset below. this is temporal-spatial dataset.
...
0
votes
0answers
12 views
Simple example for side information
Many research papers contain the phrase "side information". After a search on side information, I got the following definition from here.
In many problems of machine learning and computer ...
1
vote
1answer
131 views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
0
votes
0answers
12 views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
1
vote
0answers
40 views
Identifying rotating and resizing letters with background noise
I'm trying to complete a captcha, and here is what it looks like:
Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (...
0
votes
0answers
11 views
Should we create a label mean group of small nearby objects in object detection?
I'm working on object detection models and my dataset sometimes has a lot of small objects (stay far from the scene) (overlapping and nearby) which is really annoying in annotating (it's too small and ...
1
vote
0answers
11 views
Split on dataset with some shared features?
I have a dataset with financial stock data, some of the features are shared, for example daily gold prices, while the stock price for each individual stock is different, the gold price would be the ...
1
vote
0answers
28 views
Pixel values of segmap in multi-class semantic segmentation
I'm preparing a dataset for a multiclass semantic segmentation using U-Net like architecture. To be precise, I've got it ready but a question came to my mind. How does pixel values of a segmentation ...
0
votes
1answer
38 views
Cnn for Combination of both digits and letters(small and capital) [closed]
Hi I am new to machine learning can anyone suggest open dataset consists of both digits and letters(small,capital)
I want images consisisting of both digits and letters to train my cnn model and make ...
0
votes
0answers
22 views
Is it possible to do object detection on an object classification dataset?
I'm new to computer vision, which I find fascinating. I wonder whether it is possible or if there has been any research into going from object recognition data to object detection. In other words, ...
0
votes
0answers
158 views
How does pairwise comparison training work in XGBoost's XGBRanker?
I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well.
I want to find out how the XGBRanker ...
2
votes
5answers
94 views
How do I increase the size of an (almost) balanced dataset?
I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
1
vote
1answer
34 views
How to source training data in ML for information security?
A company entrusts a Data Scientist with the mission of processing and valuing data for the research or treatment of events related to traces of computer attacks. I was wondering how would he get the ...
