5

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the values of these features, and then training a model for each bin. They are then examining the differences between the models. Usually this is done to learn ...


3

You should look into "missing values". This is an entire research field in itself. First, you need to identify the type of missing values: They can be missing purely at random. Whether they are missing or not is itself a useful feature, and should be treated as a class of its own. (Those two are the best case scenarios.) Whether they are missing ...


3

The most common way people deal with inputs of varying length is padding. You first define the desired sequence length, i.e. the input length you want your model yo have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired ...


3

I answered a similar question earlier and here is a piece of my answer that i think covers your question: Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network ...


3

Your data set would be what is called "unbalanced' and this can lead to problems in developing an accurate classifier. The best thing to do (which you might not be able to do) is to find more images for those classes with a smaller number of images. Another alternative is to synthetically produce more images. One way to do that is to use the Keras ...


3

It is somewhat risky to discuss data independently with your learning mechanism. There is actually no such thing as good data or a good learner. There is only data that is good WITH a particular learner. That is even true of human intelligence after all the standardized education and testing done today. There are also exceptional learners that find data ...


2

manual feature engineering started becoming obsolete That is wrong. Any suggestion on when to use manual feature engineering, feature learning or a combination of the two? Deep learning is awesome for natural signals like images, audio or large amounts of unstructured text (e.g. arbitrary crawled websites) There are some basic steps that make almost ...


2

Here's a list of some of the best python libraries for natural language processing. Natural Language Toolkit (nltk) Covers all the basic functions and NLP tools such as tokenization etc. TextBolb This is a good library of beginners, it provides the nltk toolkit in a simplified format. Spacy It is an advanced library and can be used in production code. You ...


2

While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from? I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some ...


2

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy). The book Data Preprocessing in ...


2

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each ...


2

It depends, as mentioned in comments, on your model and labels. For example how would you use standardisation on multi classification problem? Generally, standardisation is more flavourable for input data as its mean is around 0. I assume you have a regression model and in that case using standardisation could be better than normalisation.


2

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part ...


2

It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link , if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be spectacular. In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". Also, you should remember that BERT ...


2

Mathematically, the convolution is an operation that takes two functions, $f$ and $g$, and produces a third function, $h$. Concisely, we can denote the convolution operation as follows $$f \circledast g = h$$ In the context of computer vision and, in particular, image processing, the convolution is widely used to apply a so-called kernel (aka filter) to an ...


2

There are different questions and even different lines of thought here. Let's go through them On resizing Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN) What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small ...


1

Removing the overlayed text might increase accuracy, but you'd need to train a different model to do this, and that is an entirely different task as it is no longer classification, but generation. There are easier ways to augment your data and probably get similar benefits to your accuracy. However, if you would still like to do this, there is a lot of ...


1

There are multiple ways to get word embedding from a corpus. Count Vectorizer: You can use the CountVectorizer() from sklearn.feature_extraction.text and then use the fit_transform() if the corpus has been converted into a list of sentences TF-IDF Vectorizer: You can use the TfidfVectorizer from sklearn.feature_extraction.text and then again use the ...


1

This Cross Validated post might answer your question. In a nutshell: A single batch (that is all your data in one batch) will result in a smooth trajectory on the loss surface. The drawback is that all your data might not fit into your memory. Which is highly likely for ~100k images. One image per batch (batch size = no. examples) will result in a more ...


1

Yes! This is crucial. If you rotate your input images for segmentation, you need to rotate the output masks as well. Otherwise the loss of your network will not be correctly calculated and your network will not learn how to generalize to rotated input images. If you use keras, you can use two ImageDataGenerator classes, one for the images and one for the ...


1

Your question are missing some details and i will assume some scenarios. If you have a classification problem: you can try group the values in intervals that make sense (you should analyze and decide for this setup), if its possible. For example: 0.000-0.250 (0), 0.251-0.500 (1), 0.501-0.750 (2) and so on. Note that neural networks are sensible for distance ...


1

As you mentioned in the comments about a possible problem of using mean, median type of imputations naively could lead to wrong predictions. In such cases, you need to first check whether you have enough data. If you have enough data You can try using MICE (Multivariate Imputation By Chained Equations) algorithm on your missing data. The method is based ...


1

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK There is a DataCamp course that walks you through most of the process but instead of a classifier they use deep learning to reconstruct brain images. They ...


1

It depends how you recognise the entities. If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with. However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if [work] at {...


1

In fact, choosing smartly the values of the image augmentation can help the performance of your system. Where I work we developed an object detector for cars. We had the following image augmentation parameters: Apect ratio distorsion: it changed the cars dimensions Additive noise: it blurred the image Change colorspace: change the cars colors Saturation ...


1

You can use append function: final = df1.append(df2, ignore_index=True) To set the last column as labels, you set them as so by: labels = np.array(final["will_buy"]) So, when calling the fit method on the model you build, you set labels = labels.


1

You can use the function inverse_transform of the created MinMaxScaler object. See also this Stack Overflow question for other answers and examples.


1

I developed a python script to crop faces using MTCNN. I found this to be the most accurate of all the face cropping algorithms at the expense of being somewhat slower. The function I developed is on the kaggle website at https://www.kaggle.com/gpiosenka/detect-align-resize-rename-facial-images. The markup first cell explains how to use it. In a nutshell ...


1

I have found a script that does what i need: https://github.com/leblancfg/autocrop


1

I don't know of a tool but you could write a simple script to detect faces and crop it. It's quite simple with the Haar cascade in openCV to detect faces and use inbuilt functions to crop your image based on the size of the detected face. Hope that helps !


Only top voted, non community-wiki answers of a minimum length are eligible