5

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the values of these features, and then training a model for each bin. They are then examining the differences between the models. Usually this is done to learn ...


4

The most common way people deal with inputs of varying length is padding. You first define the desired sequence length, i.e. the input length you want your model to have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired ...


4

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy? In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and ...


4

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones. Your weights should be both positive and negative. In addition, once trained, they tend to follow a certain size distribution. It helps if you start ...


3

I answered a similar question earlier and here is a piece of my answer that i think covers your question: Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network ...


3

Your data set would be what is called "unbalanced' and this can lead to problems in developing an accurate classifier. The best thing to do (which you might not be able to do) is to find more images for those classes with a smaller number of images. Another alternative is to synthetically produce more images. One way to do that is to use the Keras ...


3

Yes. Skewed data is one of the biggest problems in AI applications. As you rightly identified, the real world distribution is skewed. Doing a random sampling results in one major issue of an uneven sampling (like in your case). Even worse could happen, all of the samples may fall into a single class and other classes may not even be recognized by your ...


3

In the article Playing Atari with Deep Reinforcement Learning, Mnih et al, 2013, which was a major outbreak in Deep Reinforcement learning (especially in Deep Q learning), they don't feed only the last image to the network. They stack the 4 last images : For the experiments in this paper, the function φ from algorithm 1 applies this preprocessing to the ...


3

You should look into "missing values". This is an entire research field in itself. First, you need to identify the type of missing values: They can be missing purely at random. Whether they are missing or not is itself a useful feature, and should be treated as a class of its own. (Those two are the best case scenarios.) Whether they are missing ...


3

There are different questions and even different lines of thought here. Let's go through them On resizing Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN) What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small ...


2

Fourier transform is used to transform audio data to get more information (features). For example, raw audio data usually represented by a one-dimensional array, x[n], which has a length n (number of samples). x[i] is an amplitude value of the i-th sample point. Using the Fourier transform, your audio data will be represented as a two-dimensional array. ...


2

Depends on how your game is played. Is there any meaning assigned to the order of cards, or are all 5 played simultaneously? If order matters, use 5 one-hot vectors so you can choose how to order them, otherwise use a single 5-hot input vector. I would also add that if temporal order matters, you could also use a recurrent net with a 52-element input and ...


2

Here's a list of some of the best python libraries for natural language processing. Natural Language Toolkit (nltk) Covers all the basic functions and NLP tools such as tokenization etc. TextBolb This is a good library of beginners, it provides the nltk toolkit in a simplified format. Spacy It is an advanced library and can be used in production code. You ...


2

manual feature engineering started becoming obsolete That is wrong. Any suggestion on when to use manual feature engineering, feature learning or a combination of the two? Deep learning is awesome for natural signals like images, audio or large amounts of unstructured text (e.g. arbitrary crawled websites) There are some basic steps that make almost ...


2

You can use the function inverse_transform of the created MinMaxScaler object. See also this Stack Overflow question for other answers and examples.


2

While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from? I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some ...


2

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy). The book Data Preprocessing in ...


2

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK There is a DataCamp course that walks you through most of the process but instead of a classifier they use deep learning to reconstruct brain images. They ...


2

It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link , if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be spectacular. In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". Also, you should remember that BERT ...


2

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each ...


2

It depends, as mentioned in the comments, on your model and labels. For example, how would you use standardization on a multi-classification problem? Generally, standardization is more favorable for input data as its mean is around 0. I assume you have a regression model and in that case, using standardization could be better than normalization.


2

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part ...


2

Mathematically, the convolution is an operation that takes two functions, $f$ and $g$, and produces a third function, $h$. Concisely, we can denote the convolution operation as follows $$f \circledast g = h$$ In the context of computer vision and, in particular, image processing, the convolution is widely used to apply a so-called kernel (aka filter) to an ...


2

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC ...


2

There are many papers on this but the following is a good start: How to unwrap wine labels programmatically. The author includes source code in Python. You mentioned you do not want to do a panoramic view but that has more than one meaning. If I assume you mean you do not want to rotate the can while taking multiple photos, or you don't want to take ...


2

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.


2

You can use stratified cross-validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same, and therefore the test data will always be "realistic". However, when training a model (using ...


2

You can claim to use a real-world dataset, you would just need to specify that some values were interpolated. Do you have to have the inter-mediate values though? By the looks of it, each "region" was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably ...


2

Padding is indeed the easiest solution. And if no bias is used then masking the extra values during the loss computation is also not necessary, since it's enough to use zero as padding value. You might be interested though in checking Spatial Pyramid Pooling. This pooling method allows to combine fully convolutional modules and dense layers, i.e, it can be ...


Only top voted, non community-wiki answers of a minimum length are eligible