5

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model ...


5

There are various dataset available such as Pascal VOC dataset: You can perform all your task with these. size of the dataset is as follows ADE20K Semantic Segmentation Dataset: you can perform only segmentation here COCO dataset: This is rich dataset but a size larger then 5 GB so you can try downloading using google colab in your drive and then make ...


4

I presume this question was prompted by the paper Geometric deep learning: going beyond Euclidean data (2017). If we look at its abstract: Many scientific fields study data with an underlying structure that is a non-Euclidean space. Some examples include social networks in computational social sciences, sensor networks in communications, functional ...


4

In computer science, if you say "A is a proxy for B", then it means that "A replaces B" (temporarily or not), or that "A is used as an intermediary for B". The term "proxy" usually refers to a server, i.e. there are the so-called proxy servers, which intuitively do the same thing (i.e. they are used as intermediaries). ...


3

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the occurrence of another sentence $s_j$. If they are not independent, they can also not be independent and identically distributed (which is a stronger condition). The same ...


3

Your data set would be what is called "unbalanced' and this can lead to problems in developing an accurate classifier. The best thing to do (which you might not be able to do) is to find more images for those classes with a smaller number of images. Another alternative is to synthetically produce more images. One way to do that is to use the Keras ...


3

Yes, you can weight the loss function for each example, so that instead of your cost function being $$J = \sum_i \mathcal{L}(y_i, \hat{y}_i)$$ It will be $$J = \sum_i w_i\mathcal{L}(y_i, \hat{y}_i)$$ Where $i$ iterates over your data set, $\mathcal{L}$ is the loss function you are using, $y_i$ is ground truth for each example and $\hat{y}_i$ is ...


3

Of course, it's possible to define a problem where there is no relationship between input $x$ and output $y$. In general, if the mutual information between $x$ and $y$ is zero (i.e. $x$ and $y$ are statistically independent) then the best prediction you can do is independent of $x$. The task of machine learning is to learn a distribution $q(y|x)$ that is as ...


3

Look at Google's Open Image Dataset @ https://storage.googleapis.com/openimages/web/index.html They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships. Here is the link for the traffic signs dataset.


3

Yes it should be possible. You may have a bug in your code, or the wrong hyperparameters. Training ResNet-50 will take a long time. Try training on other sets of images and see what accuracy you get to check if your approach is correct. Or, try loading a pretrained model, and training from that.


2

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life. If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the ...


2

You can't label things you don't know. The goal of labeling is to label the things you want the classifier to learn so that when you run it in inference mode you can discover what is in your data (new data that you didn't use for training, validating, or testing). It is not a good idea to label small objects like the 'blue water' unless it is important to ...


2

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes, you successfully conducted hyper-parameter optimization. However, the AUC you received for the best set of hyperparameters found (by manual tuning) is not ...


2

In this paper: Unsupervised Machine Translation Using Monolingual Corpora Only the authors proposed a novel method. Intuitively it is an autoencoder, but the Start Of Sentence token is set to be the language type. One other advanced method is to use the pre-training model. In this paper: Cross-lingual Language Model Pretraining researchers proposed an ...


2

Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a significantly smaller or less representative validation dataset, e.g., distribution of the validation dataset can be skewed towards ...


2

Simply stated, you use your validation set to regularize your model for unseen data. Test data is completely unseen data, on which you evaluate your model. Various validation strategies are used to improve your model to perform for unseen data. So strategies like k-fold cross-validation are used. Also, the validation set helps you in tuning your ...


2

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering algorithms. sklearn provides functions that implement k-means (and an example), hierarchical clustering algorithms, and other clustering algorithms. To assess ...


2

One problem with clustering algorithms is that they will typically find you a solution, ie they will split your data set into clusters, but it will find you a structure even if there isn't one. Your data looks like it could consist of about 5 to 7 clusters, but it could equally well just be 2 or only 1. What you need to do after the clustering is to assess ...


2

This question raises a lot more questions. It seems like a solution looking for a problem, instead of the other way round. How do you measure the fitness of a feature? What would one of the "evolved datasets" mean? What does it represent? What would your overal purpose be? If you just wish to generate simulated datasets, there are easier ways to ...


2

The paper Evolutionary Dataset Optimisation: learning algorithm quality through evolution (2019), by Henry Wilde et al., proposes a method to generate datasets with a genetic algorithm. Their goal is to generate data for which a particular algorithm performs well, in terms of a certain metric, so that to get more insights about this algorithm and why it ...


2

If I understood correctly, the model is a polynomial equation No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a polynomial, but, for example, in a neural network, you can combine many sigmoids to build complicated functions that may not necessarily be polynomials. We usually ...


2

They are equivalent. When we consider a particular instance as a vector, we are not literally imagining it as an arrow with it's head at the point coordinates and tail at the origin. It's just when you are working with a tuple of numbers in a mathematical context, it is conventional to call it a vector. This language follows into machine learning which is ...


2

Short answer Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you shuffle or not the test or validation data (unless you are computing some metric that depends on the order of the samples), given that you will not be computing ...


2

It's true that your original dataset can contain duplicates, so it should not be called a set, in order to be consistent with the mathematical definition of a set. There are mathematical objects known as multi-sets that can contain duplicates, but the order of the elements is still not relevant. There are also tuples and sequences, where the order of the ...


1

In machine learning, we can use all the datasets as training data in a model. But if there are too many data sets, or too much data, and we do not split them up, our model may be not produce acceptable results. Why? Because if the model studies too much training data, it may be overfitted. (Just like when you cram for a test, and get overloaded with ...


1

Your assumption about the test data is not correct completely. Maybe you use the test data to tune your learning algorithm to work better on the test data, but it's not the whole thing. Sometimes you need to know that the ML method is working or not and have a sense about how much does it work! You have other scenarios that you want to evaluate your method: ...


1

Batch size and epochs are independent parameters - they serve very different purposes. Your main question as I understand it (and for general, non-library specific consumption) is what is an epoch and how is the data used for each epoch? Simply put, an epoch is a single iteration though the training data. Each and every sample from your training dataset ...


1

You can find the dataset in the following links: Pomegranate Disease Detection Using Image Processing fruits 360 datasets


1

I believe this is covered under Section 107 of the Copyright Act states: the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, ...


1

If you want to evaluate on real thermal image dataset, you can use this one. Thermal Image dataset is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7") Mean Average Precision is the relevant metric but it's more technical. You can start explaining with False Positives ...


Only top voted, non community-wiki answers of a minimum length are eligible