5

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the values of these features, and then training a model for each bin. They are then examining the differences between the models. Usually this is done to learn ...


4

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy? In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and ...


4

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones. Your weights should be both positive and negative. In addition, once trained, they tend to follow a certain size distribution. It helps if you start ...


3

The most common way people deal with inputs of varying length is padding. You first define the desired sequence length, i.e. the input length you want your model yo have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired ...


3

Your data set would be what is called "unbalanced' and this can lead to problems in developing an accurate classifier. The best thing to do (which you might not be able to do) is to find more images for those classes with a smaller number of images. Another alternative is to synthetically produce more images. One way to do that is to use the Keras ...


3

It is somewhat risky to discuss data independently with your learning mechanism. There is actually no such thing as good data or a good learner. There is only data that is good WITH a particular learner. That is even true of human intelligence after all the standardized education and testing done today. There are also exceptional learners that find data ...


3

I answered a similar question earlier and here is a piece of my answer that i think covers your question: Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network ...


3

You should look into "missing values". This is an entire research field in itself. First, you need to identify the type of missing values: They can be missing purely at random. Whether they are missing or not is itself a useful feature, and should be treated as a class of its own. (Those two are the best case scenarios.) Whether they are missing ...


2

Here's a list of some of the best python libraries for natural language processing. Natural Language Toolkit (nltk) Covers all the basic functions and NLP tools such as tokenization etc. TextBolb This is a good library of beginners, it provides the nltk toolkit in a simplified format. Spacy It is an advanced library and can be used in production code. You ...


2

manual feature engineering started becoming obsolete That is wrong. Any suggestion on when to use manual feature engineering, feature learning or a combination of the two? Deep learning is awesome for natural signals like images, audio or large amounts of unstructured text (e.g. arbitrary crawled websites) There are some basic steps that make almost ...


2

While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from? I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some ...


2

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy). The book Data Preprocessing in ...


2

You can use the function inverse_transform of the created MinMaxScaler object. See also this Stack Overflow question for other answers and examples.


2

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK There is a DataCamp course that walks you through most of the process but instead of a classifier they use deep learning to reconstruct brain images. They ...


2

It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link , if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be spectacular. In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". Also, you should remember that BERT ...


2

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each ...


2

It depends, as mentioned in comments, on your model and labels. For example how would you use standardisation on multi classification problem? Generally, standardisation is more flavourable for input data as its mean is around 0. I assume you have a regression model and in that case using standardisation could be better than normalisation.


2

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part ...


2

Mathematically, the convolution is an operation that takes two functions, $f$ and $g$, and produces a third function, $h$. Concisely, we can denote the convolution operation as follows $$f \circledast g = h$$ In the context of computer vision and, in particular, image processing, the convolution is widely used to apply a so-called kernel (aka filter) to an ...


2

There are different questions and even different lines of thought here. Let's go through them On resizing Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN) What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small ...


2

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC ...


2

There are many papers on this but the following is a good start: How to unwrap wine labels programmatically. The author includes source code in Python. You mentioned you do not want to do a panoramic view but that has more than one meaning. If I assume you mean you do not want to rotate the can while taking multiple photos, or you don't want to take ...


2

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.


2

You can use stratified cross validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same and therefore the test data will always be "realistic". However, when training a model (using ...


2

You can claim to use a real-world dataset, you would just need to specify that some values were interpolated. Do you have to have the inter-mediate values though? By the looks of it, each "region" was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably ...


1

Yes! This is crucial. If you rotate your input images for segmentation, you need to rotate the output masks as well. Otherwise the loss of your network will not be correctly calculated and your network will not learn how to generalize to rotated input images. If you use keras, you can use two ImageDataGenerator classes, one for the images and one for the ...


1

Your question are missing some details and i will assume some scenarios. If you have a classification problem: you can try group the values in intervals that make sense (you should analyze and decide for this setup), if its possible. For example: 0.000-0.250 (0), 0.251-0.500 (1), 0.501-0.750 (2) and so on. Note that neural networks are sensible for distance ...


1

An interesting question. I would start by finding n nearest neighbors of each data point, then calculate their center of mass c and the point's distance d to its nth nearest neighbor. The smaller the d is the larger the density is around a given point. You could then iteratively step every point away from their c in an inverse proportion to the distance d ...


1

As you mentioned in the comments about a possible problem of using mean, median type of imputations naively could lead to wrong predictions. In such cases, you need to first check whether you have enough data. If you have enough data You can try using MICE (Multivariate Imputation By Chained Equations) algorithm on your missing data. The method is based ...


1

It depends how you recognise the entities. If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with. However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if [work] at {...


Only top voted, non community-wiki answers of a minimum length are eligible