Skip to main content

Questions tagged [datasets]

For questions related to sets of data and their use in AI.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

(distributed) DataLoader when batches are correlated, pitfals?

I know how to implement Datasets and Dataloaders in pytorch. However, when it comes to distributed and correlated batch generation, I need advise. Background I have a network that generates a new ...
Klops's user avatar
  • 111
0 votes
1 answer
38 views

Size of Dataset object [closed]

I defined a dataset as below ...
COTHE's user avatar
  • 13
0 votes
0 answers
17 views

Deep Learning: Architecture vs. Features for Performance?

In deep learning, when aiming for peak metric performance, is a well-designed architecture with imperfect features/dataset generally preferable to a poorly designed architecture with high-quality ...
Muhammad Ikhwan Perwira's user avatar
1 vote
0 answers
17 views

Best method for finding centroid of 3D object?

I'm somewhat new to machine learning and want to implement a manufacturing application that finds the centroid of an irregular object so that as little material as possible is removed during ...
dstoddard's user avatar
0 votes
0 answers
20 views

Dependency in sequential data

I read from this article, We know that Whenever the points in the dataset are dependent on the other points in the dataset the data is said to be Sequential data. A common example of this is a ...
D. S.'s user avatar
  • 101
0 votes
0 answers
34 views

About copyright problem in creating benchmark dataset for pre-modern books

I'm a newbie so please answer me softly. Basically I want to create a benchmark dataset in Vietnamese pre-modern books (when the context, content, dialect, grammar, aged printed quality, printing mold ...
chickensoup's user avatar
0 votes
1 answer
29 views

Can I scale subsets of my dataset independently to handle different feature ranges?

I am currently making an LSTM model to predict the change in value of a stock for each day. My dataset consists of 4 years of data for 30 different stocks along with some financial metrics such as RSI,...
Daniel goldney's user avatar
0 votes
0 answers
22 views

Is it necessary that the number of samples of one class be balanced with other classes in a classification problem?

Consider a classification problem using machine learning techniques (e.g. malware detection). In such a problem, is it necessary that the number of samples from each class (in the mentioned example, ...
user16385455's user avatar
0 votes
1 answer
69 views

How do I know that my dataset is good enough for training a neural network?

Suppose I have a clean (no outliers and normalized) dataset for training a neural network. The training process is expected to take almost a week. So, before I start training, I want to know if this ...
user366312's user avatar
0 votes
0 answers
68 views

What are some tips of curating a dataset to fine-tune a code-completion LLM?

There is a new SDK that I am working on and I want to know what are some ways of automatically curating a dataset to train a code-completing LLM to deploy as a VSCode plugin? Hacky ways are ...
Levent Ozbek's user avatar
2 votes
1 answer
82 views

Do different camera angles affect the performance of the deep learning model?

I'm working on a project to build a face recognition system and I have a question: do different camera angles affect the performance of the deep learning model? For example, in CCTV, training data and ...
Naay's user avatar
  • 23
0 votes
0 answers
25 views

Could converting features into PCs, where #PCs = #features, improve model performance?

Let's say I have a dataset with some number of features (e.g., 10) and a target variable. I create 10 PCs from the dataset excluding the target variable. Then, I run a few classification algorithms (...
BigMistake's user avatar
1 vote
3 answers
76 views

Would maximizing (instead of minimizing) error of an LLM/HMM lead to complex behavior?

Imagine we have some sort of "next token predictor," either with transformer architecture, LSTM, or just a HMM (though the terminology I use here will be less aligned to HMMs, I believe the ...
BigMistake's user avatar
0 votes
0 answers
23 views

Is synthetic data just a placebo for immature models?

I apologize for the provocative question, but let me elaborate. I am trying to wrap my head around the logic of synthetic data. When you train a model what you are trying to do is to teach the ground ...
Pigna's user avatar
  • 295
0 votes
0 answers
13 views

Could you recommend action recognition dataset?

background I want to do action recognition for academic, but I can't find dataset what I want. so, I wonder Which dataset is suitable for me... What I want? I want to find dataset such as.. labeling ...
Yang's user avatar
  • 57
0 votes
0 answers
32 views

Training YOLO on YOLO results

I have a large unlabelled dataset. What if I use YOLO to label it? Will this dataset be useful to train a better YOLO model? What if I then finetune it on a smaller labelled dataset? My usecase ...
Karol Idaszak's user avatar
0 votes
1 answer
32 views

What are the differences between loss surfaces that "derive"from different observations?

If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum. How different those surfaces one from another? Would it be ...
Igor's user avatar
  • 183
1 vote
1 answer
671 views

What role does data quality plays in the LLM scaling laws?

DeepMind released the Training Compute-Optimal Large Language Models paper in 2022 which describe some scaling laws for LLMs. As far as I understand this is the most accredited reference to estimate ...
Blue Nebula's user avatar
0 votes
1 answer
55 views

AI and Machine Learning Prediction Algorithms for predicting outcome results of Hypothetical poll

Can artificial intelligence and Machine Learning Prediction Algorithms assist in deciding the Outome Results of a Hypothetical Online Poll? Poll: Selecting favorite American President till date. ...
Prashant Akerkar's user avatar
1 vote
1 answer
108 views

Creating a Dataset from Time Series Data

Context I'd like to build a regression model for this data to predict a user's test scores given their study habits. Basically, the variables are in two separate csv tables similar to the ones below. ...
LittleLulatsch's user avatar
0 votes
1 answer
59 views

What are possible reasons for the validation loss increasing with more data?

I trained a neural network on an NLP problem and compared the loss and BLEU score on the validation data with the same training parameters in two scenarios: a) when I trained on 25% of the data, b) ...
postnubilaphoebus's user avatar
1 vote
2 answers
69 views

Is data useless for a neural network if some inputs are derivatives of other inputs?

That is, if some of the inputs to a neural network can be calculated by a pre-determined function whose variables are other inputs, then are those specific inputs useless? For example, suppose there ...
BlueSnake's user avatar
1 vote
0 answers
230 views

How to get ZINC 500k dataset?

I have been using the ZINC graph regression dataset available through pytorch geometric datasets for a while now in two of its modes (12k examples and 250k examples). However, in the PapersWithCode ...
Angelo's user avatar
  • 211
1 vote
1 answer
248 views

How can imitation learning data be collected?

How can imitation learning data be collected? Can I use a neural network for that? It might be noisy. Should I use manual gathering?
dato nefaridze's user avatar
1 vote
1 answer
3k views

Fine-tune GPT-Neo with prompt and completion?

I'm new to AI and machine learning. To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("...
SoftTimur's user avatar
  • 111
0 votes
0 answers
37 views

Do Neural Networks tend to have Zero Mean Errors in each Output?

My NN (a few linear layers with ReLUs + batch normalization, no activation in the last layer) learns to approximate vector-valued labels $y_z$ from data $z\sim\rho_z$ in a supervised way, i.e. net$(z)=...
joinijo's user avatar
  • 101
0 votes
1 answer
1k views

Split dataset into Train/Validation/Test for Object Detection

I have a dataset for Object Detection with YOLO format labels, each imagine can have occurences of different classes and multiple occurences of the same class. How can the dataset be divided into ...
1stTimeStackOverflow's user avatar
10 votes
2 answers
2k views

How can I encode angle data to train neural networks?

I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$). I am looking for study material on how to encode this data. Can you supply me with a ...
user366312's user avatar
1 vote
1 answer
74 views

Can independent datasets be artificially combined for multimodal learning (semi-synthetic data generation)?

BACKGROUND: To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., chest X-ray (modality 1) and ...
Snehal Patel's user avatar
2 votes
1 answer
73 views

Are the "artifacts" in select Keras MNIST training images really there or is my download corrupt?

I'm having fun with a ludicrously well known and used dataset: mnist. I am doing it with a huge and well known tool: keras. Please excuse the red dots, something else I was doing. I have otherwise ...
EngrStudent's user avatar
0 votes
1 answer
244 views

Why does data augmentation using synthetic data generated by one model improve the performance of another model?

I understand from articles like this one that synthetic data generated by one model based on real data can improve the performance of a second model. Can anyone help me understand the intuition behind ...
Fijoy Vadakkumpadan's user avatar
0 votes
1 answer
139 views

Machine Learning Methods commonly used when data are scarse

It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification ...
postnubilaphoebus's user avatar
-1 votes
1 answer
246 views

Generating synthetic time series data with limited data

I would like some opinions on my current situation. I have a set of time series data that I want to forecast. The data however is not very long (around 500 rows) so I was looking into generating many ...
codinator's user avatar
0 votes
1 answer
58 views

To train a mushroom vs. non-mushroom image classifier, which images should comprise the negative class?

I want to train a model that will identify if the image is mushroom and non-mushroom. If the image is mushroom, I will use another existing model to categorize if its poisonous or edible. I want to ...
Jorge Gabriel's user avatar
3 votes
4 answers
2k views

How is MNIST only providing the training and the test sets? What about the validation?

I was taught that, usually, a dataset has to be divided into three parts: Training set - for learning purposes Validation set - for picking the model which minimize the loss on this set Test test - ...
tail's user avatar
  • 157
2 votes
1 answer
226 views

Is there an image classification dataset where the class depends on spatial relations?

My question is pretty much the one asked above. To clarify a bit further: I have only found datasets that do object localization and that also have relations between the objects annotated (like: "...
Johannes R.'s user avatar
1 vote
0 answers
25 views

Can I use a dataset with real-world images and corresponding actions that the expert took to train an IRL algorithm?

Offline Reinforcement Learning approaches like Inverse Reinforcement Learning/ Batch RL/ imitation learning/ behavior cloning allow us to use previous demonstrations by an expert to learn a policy. ...
a_razzaq's user avatar
1 vote
2 answers
1k views

Does more data increase training accuracy in neural networks?

I was wondering what is the performance benefit of feeding more data to a machine learning model like a neural network? Like I know one of the benefits is that it increases generalization - testing ...
CoderMath's user avatar
1 vote
1 answer
361 views

Feature Engineering on transactional dataset clustering

I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows: client_id Sex Age ...
Juan Ignacio Rojo's user avatar
1 vote
2 answers
122 views

Master theorem about polynomial classifiers?

Does anyone know if there is a theorem or counterexample establishing whether or not for any given binary classification task in some finite (possibly large) dimensional vector space of attributes, ...
letsmakemuffinstogether's user avatar
0 votes
1 answer
97 views

Meaning of Large Dataset for machine learning

Some online answers about parameters in machine learning mention that it is dependent on the size of dataset we have (if it is a large dataset or not). Is this size related to the number of samples we ...
Batoul Diab's user avatar
0 votes
1 answer
229 views

combine two features in dataset?

I have a data set containing the number of security gaps and the level of that gap for a specific website. Now suppose I have 2 features in this data set, the first feature is the number of a ...
Issa Mansour's user avatar
1 vote
1 answer
98 views

Datasets input at model.fit produce unexpected results of training loss vs validation loss

Im trying to train a neural network (VAE) using tensorflow and Im getting different results based on the type of input in the model.fit. When I input arrays I get normal difference between the ...
user avatar
1 vote
1 answer
210 views

Should I include overlapping (input) Data in my training data

If I have time dependent data and want to predict the relative change for a future time. Should I separate the data so that the input times don't overlap? With an example: I have hourly temperature ...
KarlTheGreat's user avatar
0 votes
1 answer
85 views

What should be taken as random variables in the distributions of datasets?

Consider the following two paragraphs taken from the paper titles Generative Adversarial Nets by Ian J. Goodfellow et.al #1: Abstract We propose a new framework for estimating generative models via ...
hanugm's user avatar
  • 3,930
1 vote
2 answers
288 views

What makes a 'good' dataset

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own. ...
JakobS's user avatar
  • 11
0 votes
1 answer
75 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user avatar
7 votes
2 answers
805 views

Is there an argument against using the (reviewed) predictions of a model as ground truth to further train exactly this model?

I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight ...
thzu's user avatar
  • 73
0 votes
0 answers
20 views

Which existing model could be used for wind speed and direction prediction?

I am trying to predict the wind speed and wind direction in a graph network for a geographical area. The dataset includes the start and end nodes, the distance between them, and wind speed and ...
bsha's user avatar
  • 1
4 votes
1 answer
322 views

What is the total number of actions and rewards count

Reading this two articles about Reinforcement Learning: Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al. Human-level control through deep reinforcement learning by ...
Jigberto's user avatar
  • 143

1
2 3 4 5 6