Questions tagged [datasets]

For questions related to sets of data and their use in AI.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Using logical rules in autonomous driving dataset(Argoverse, nuscenes etc) for trajectory prediction [closed]

I have seen some papers which convert traffic rules into logical ones (first order logic or temporal) as a kind of representation of rules knowledge. Like in the paper 'Formalization of Interstate ...
user avatar
0 votes
1 answer
47 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user avatar
  • 1
0 votes
0 answers
14 views

Is there any framework for generation of synthetic data where we can set how hard data is to classify?

I am comparing two different training methods of deep neural networks and I was wondering are there any frameworks that can generate x, y datasets but with some input parameter or parameters that can ...
user avatar
  • 1,039
7 votes
2 answers
746 views

Is there an argument against using the (reviewed) predictions of a model as ground truth to further train exactly this model?

I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight ...
user avatar
  • 73
0 votes
0 answers
15 views

Which existing model could be used for wind speed and direction prediction?

I am trying to predict the wind speed and wind direction in a graph network for a geographical area. The dataset includes the start and end nodes, the distance between them, and wind speed and ...
user avatar
  • 1
3 votes
1 answer
47 views

What is the total number of actions and rewards count

Reading this two articles about Reinforcement Learning: Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al. Human-level control through deep reinforcement learning by ...
user avatar
  • 133
0 votes
0 answers
16 views

How to use oxford5k for training?

Generally, we have training data with landmark IDs, their GTs (positive samples), and then separate query images and corresponding positive samples for evaluation. In the Oxford5k or ROxford5k, one ...
user avatar
  • 101
0 votes
3 answers
81 views

How to deal with an unbalanced dataset?

I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
user avatar
0 votes
1 answer
28 views

How to create a dataset for binary classification

I would like to classify whether a pot of water is boiling or not using a CNN. Is it enough to take pictures of boiling water using only one pot, or should I use different pots for this to generalize ...
user avatar
0 votes
1 answer
30 views

Having the negative cases in the same batch vs. shuffling the dataset

I am working on a model for an NLP task. The model encodes the text and has a regression output layer. In this task, from each instance (positive), I create several negative cases using a specific ...
user avatar
  • 123
0 votes
1 answer
70 views

Is there a standard term for the following flaw in the data?

I wonder if following characteristic of data has some standard "professional" or scientific term associated with it. Let's assume that I have a set of dog/cat images labeled 0 for a cat and ...
user avatar
  • 1,039
0 votes
0 answers
28 views

What are the "per image" annotations that are generally used for image datasets in AI?

Computer vision is highly benefited by AI algorithms. Image data is abundantly available. There are different varieties of tasks such as image classification, prediction, segmentation, generation, ...
user avatar
  • 3,099
0 votes
0 answers
33 views

What is all necessary types of data for a bidirectional RNN to learn embeddings?

Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level. In order to train a CNN for the classification tasks, images, and labels/outputs are ...
user avatar
  • 3,099
0 votes
0 answers
7 views

Predicting single floats based on set of 2 feature arrays each of 100 values

I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples. The problem is that my labels are single floats (...
user avatar
1 vote
0 answers
19 views

What are the types of data in which the order of instances does matter?

In general, the order of instances in the datasets that are used in machine learning is immaterial. But there are exceptions. Timeseries data is one such exception I know. Consider the following two ...
user avatar
  • 3,099
0 votes
0 answers
11 views

What to predict in a limited transaction dataset?

I have been given a task with a real transaction dataset. The task is to predict something using either logistic regression or simple binary classification. The columns are as follow: Transaction ID ...
user avatar
1 vote
0 answers
40 views

Does the Frechet Inception Distance (FID) consider color?

I was wondering if the Frechet inception distance for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale. I know that it depends on the feature ...
user avatar
  • 201
1 vote
1 answer
53 views

Is there any simple example for volumetric data except from physics and medicine?

Recently I heard about the term volumetric data. The definition for volumetric data is as follows #1: Definition Volumetric data is typically a set S of samples $(x, y, z, v)$, representing the value ...
user avatar
  • 3,099
2 votes
0 answers
19 views

Do any practical deep learning algorithms deal with tensors containing non-real entries?

In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers. Other than both mentioned above, there may be some other real-...
user avatar
  • 3,099
0 votes
1 answer
47 views

Generating a dataset from data with "assumed" lables

I've got a task similar to the following: Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
user avatar
  • 103
1 vote
1 answer
57 views

Can I use the transformers for the prediction of historical data?

Can I use the transformers for the prediction of wind power with the historical data? Dataset Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
user avatar
0 votes
0 answers
18 views

What are the pros and cons of using a normal positional encoding in an adjacency matrix?

I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
user avatar
3 votes
1 answer
127 views

How to determine the quality of synthetic data?

I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums. I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
user avatar
  • 33
1 vote
0 answers
27 views

Recommended way to spilt image sequence for training/validation/testing

For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by. Now what's the ...
user avatar
  • 33
2 votes
1 answer
81 views

How do you handle unbalanced image datasets?

I have an image data set on which I am training a CNN. The data set is slightly unbalanced. So, my solution up till now was to delete some images of the majority class. But I now realize that there ...
user avatar
8 votes
3 answers
844 views

Is it okay to use publicly available Instagram videos to train an AI?

Since I haven't found any good training data for my university project, I want to use pictures and videos from public Instagram profiles. Am I allowed to do that?
user avatar
0 votes
0 answers
66 views

What are the different possible usages of the word "i.i.d" in machine learning?

The acronym "iid" stands for "independent and identically distributed". It is a property of a sequence of random variables. You can read here for more details. This question is ...
user avatar
  • 3,099
1 vote
1 answer
93 views

Which of the following probability distribution is generating an iid dataset?

Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$ The probability distribution $p_1$ over $X_1, X_2$ is given by $$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$ $$...
user avatar
  • 3,099
1 vote
2 answers
118 views

How can we "draw i.i.d" from any probability distribution?

Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges ...
user avatar
  • 3,099
0 votes
0 answers
31 views

Modelling of output neuron for mixed features?

A dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using a neural network. So, input ...
user avatar
  • 3,099
0 votes
1 answer
51 views

Why disentangling the features of variation in representation?

Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al. It has been hypothesized, ...
user avatar
  • 3,099
0 votes
0 answers
39 views

What are mathematically the factors of variation in deep learning?

The following paragraph from an answer tells us about factors of variation Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
user avatar
  • 3,099
0 votes
1 answer
75 views

How to handle an unbalanced dataset when training object detection algorithms?

I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024. Within those images I have the ...
user avatar
  • 119
0 votes
0 answers
47 views

What's the best way to feed stories to a neural network?

I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because ...
user avatar
0 votes
2 answers
57 views

Why data required for hyperparameter tuning is considered as an additional data?

Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following ...
user avatar
  • 3,099
1 vote
0 answers
39 views

Is it true that real world data is highly discontinuous?

A function $f$ is said to be continuous at a point $c$ if it satisfies three properties: Should be defined at the point $c$ Left and right-hand limits at $c$ must be equal i.e., the limit must exist ...
user avatar
  • 3,099
0 votes
0 answers
25 views

Adding data to training results in loss random peaks

I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what ...
user avatar
  • 73
0 votes
0 answers
35 views

Which Neural Network Topology to choose, are Transformers suitable?

I have a regression problem and I am not quite sure which architecture to choose. I never worked with transformers before, but I generally understand how they work and I think they might be suitable. ...
user avatar
0 votes
1 answer
26 views

Data analysis before feeding to ML pipeline

I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the ...
user avatar
  • 103
0 votes
0 answers
16 views

Weight for Samples on SVM (Support Vector Machine)

There is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library). If I use the option ...
user avatar
1 vote
1 answer
261 views

Is there a way to parallelise the RL training on multiple stocks to avoid the memory issue?

I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA some training features might be the pivot price values and the set of the ...
user avatar
  • 13
1 vote
0 answers
26 views

Training on the dataset in parts vs training on the whole dataset

What is the difference between these two situations? are they the same ? #1 : train a model 20 epochs on the whole dataset #2 : divide dataset into n-parts then train the model 20 epochs on each part ...
user avatar
0 votes
1 answer
30 views

An online editor that allows data labeling format [closed]

I have a set of students (~20) that will work on annotating data for an NLP project. The annotation task will be as in the following: ...
user avatar
  • 123
1 vote
0 answers
60 views

Batch normalization for multiple datasets?

I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data. I was told that batch ...
user avatar
  • 201
1 vote
2 answers
325 views

Train Validation Test Splitting After or Before Data Augmentation?

I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of ...
user avatar
0 votes
1 answer
31 views

How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?

I have a question I have had a dataset below ...
user avatar
0 votes
0 answers
18 views

Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset

So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
user avatar
1 vote
1 answer
36 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
user avatar
0 votes
0 answers
47 views

Simple example for side information

Many research papers contain the phrase "side information". After a search on side information, I got the following definition from here. In many problems of machine learning and computer ...
user avatar
  • 3,099
1 vote
1 answer
240 views

What is the meaning or implications of the rank of a dataset for machine learning algorithms?

Consider a dataset with $n$ training examples and $d$ features. Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it. In matrices, rank $r$ is generally useful in Knowing the dimension ...
user avatar
  • 3,099

1
2 3 4 5