New answers tagged

0

I think serali answered this question well, though I wanted to give some extra reading for those interested. There are many ways of deciphering what a neuron in a NN is doing. This lecture does a fantastic job at covering some of these methods and is an incredibly interesting watch. This covers more advanced methods of visualising what a model is doing.


2

In TensorFlow Playground, the horizontal line show where each class is separated for each neuron. What happens when you take any intermediate neuron to make the decision? You can see the answer by the line provided by that neuron. And this decision is a result of the weighted sum from the decisions of the previous neurons (up to activation). Take the middle-...


0

There's similar boosting classes in XGBoost for regression. You can implement their built-in classes for your problem, rather than implementing from scratch. You can read more about it from their website. You can also take a look at catboost, which implements a different approach.


0

very interesting questions: 1) what exactly is happening when training and validation accuracy change during training The accuracy change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of ...


0

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.


1

Hello and welcome to the community. There are multiple ways you can train a neural network: stochastic, mini-batch and batch. What you explained is the stochastic mode, where you input one training example 01 for example, calculate the gradients and update the networks weights before the next training example is fed. You could also select multiple such ...


0

The question is related to "feature extraction". Firstly, to tackle a regression problem like both the problems stated by you, you need to provide the neural network with the most relevant inputs that have a effect on the output. Eg. If you want your network to add x and y, you need to provide it training examples like input(x=1, y=3) and output (sum=4). ...


3

On paper, one expects a complex enough network to determine any complicated function of a limited number of inputs, given a large enough dataset. But in practice, there is no limit to the possible difficulty of the function to be learnt, and the datasets can be relatively small on occasion. In such cases - or arguably in general - it is definitely a good ...


0

The main reason of overfitting in any neural network is having too many unrestricted trainable degrees of freedom in the model. Methods similar to dropout reduce the number of neurons at each training run which effectively means having a smaller network. On the other hand in $l_1$ and $l_2$ regularization, a term added to the loss function which put a ...


1

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.: (Image adapted from Wikipedia entry on overfitting) It is clear that this does not happen in your diagram, hence your model does not overfit. A difference between a training and a validation score by itself does not ...


2

You can use CNN in timeseries data. Convolutional Recurrent Neural Network(RCNN) is one of the examples. Convolutional layers basically extract feature from image, It is not related to time series data passing, Neither of them you mention on the question. CNN therefore use some recurrent concept to improve their prediction such as in ResNet, Highway Networks,...


3

Usually you need to ensure that your convolutions are causal, meaning that there is no information leakage from the future into the past. You could start by looking at this paper which compares Temporal Convolutional Networks (TCN) with vanilla RNNs models.


1

I found my answer in a different post: How to evolve weights of a neural network in Neuroevolution?. Note that the genetic algorithm is a subcategory of the neuroevolution algorithm. Short answer, my original thoughts were correct.


0

I would recommend learning about Reinforcement learning first. You don't need a dataset as you train your network by letting it play the game over and over again. but knowing how to do so doea mean finding out about markov decision process and how you can use the neural network to solve this.


2

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss of one almost definitely means there's something off. I'd recommend before doing anything thoroughly go through your data or see if there's anything to debug ...


3

Yes, this is actually a limitation known as catastrophic forgetting. A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on the weights important for those tasks". See Overcoming catastrophic forgetting in neural networks for details. Another approach is Learning without forgetting. ...


3

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, probably something is not right. But remember that there are 2 parts contributing to the overall loss. The mse loss and the l2 penalty loss. (Also remember that ...


1

A robust ML model is one that captures patterns that generalize well in the face of the kinds of small changes that humans expect to see in the real world. A robust model is one that generalizes well from a training set to a test or validation set, but the term also gets used to refer to models that generalize well to, e.g. changes in the lighting of a ...


1

There are so many different versions of spiking neural networks out there. I think it is mainly due to the fact that there has been no dominant successful SNN model with proper learning algorithm like CNN with BP. However, there have been several recent papers(e.g. SuperSpike, SLAYER) on SNN that may lead to the standard framework for SNN. It happened within ...


3

tl;dr The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter. A deeper look... Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute ...


2

Why AI is (or not) a good option for the generation of random numbers? AI approaches are generally not good for generating random numbers, for these reasons: Similar to why they are not good for adding numbers, there already exist many strong pseudo-random and "true" random sources, possible without using any AI approach, and demonstrably good enough for ...


0

According to your description, you already know your function $f$ to be optimized. So you should use it directly instead of the standard loss functions. In this other post there is an explanation of how to use $f$ as a custom loss function in Keras.


1

Artificial Intelligence is a simulation of neurons interacting to each other. It's a very good copy of the model of a Neuron, where the input shows the acting of dendrites, the cell body (soma) represents the Neuron class itself. And the output is the axon. In general, the answer to your question would be - not yet. There are many aspects that are needed ...


1

You need to have access to the 696th hour (or successive hours), otherwise, you cannot test your model. An alternative would be, for example, to train your model on the first 693 hours, validate it on the 694th hour, and test it on the 695th hour.


2

You results show signs of overfitting at around epoch 40. In order to overcome this you can either simplify the model somewhat or increase regularization. You do not share what values you are using for dropout regularization so you can try increasing that. But to be honest, I am not sure if that is going to help. You are using dropout in a pure CNN ...


2

Is the image taken from a constant distance? If yes, you'd need to scale the images to the same dimensions first of all. For few images say 100-500 images (more the better) you'd need to label the dataset by proper scaling. Once labeled, use it to train a CNN (Although best would be training a ResNet). Once trained with decent accuracy, test it for the ...


2

If you have stero pairs, and you can identify the objects in the scene, you do not need a neural network, you can just use triangulation. If you need to identify which objects in the scene are the same, you have an image segmentation problem. Depending on your problem and the amount of data you have access to, you may be able to use simple techniques like ...


2

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25. The ...


4

In language theory, there are generally several admitted levels that can be studied in relation with one another or independently. The semantic level is the one dealing with the meaning of the text ("semantic" comes from the greek and means "to signify"). The semantic level is therefore generally independent from the syntax and even the language used to ...


0

Some ideas out the top of my head: In the case of $dy/dx_2>0$ you could compute the gradient using the chain rule and limit the weights so that the constrain holds In the case of $y + x_5 + x_7 < K$ you could use a clipping function on the output layer?


-1

When training our neural network, you need to scale your dataset in order to avoid slowing down the learning or prevent effective learning. Try normalizing your output. This Tutorial might help


1

First of all, you should add the argument workers = n in the fit generator call. n should be bigger than 1 to prefetch data. As your data processing requires the data be taken from a server or port, you should do pre fetching data as that would fetch the next data while GPU is processing. If you call fit_generator with workers > 1 , use_multiprocessing=...


3

For the first question, RMSE and Euclidean distance have no difference, not that i know of. For the second question, you only need the common loss function for normal tasks. MSE is a common loss function used in linear regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For ...


3

When someone is able to do a causative attack it means there is a mechanism by which they are able to input data into the network. Maybe a website where people can input their images and it outputs a guess on what is in the picture and then you click if it got it right or not. If you continue to input images and lie to it it will obviously get worse and ...


2

You can try using a multi-input model. Here is a recent post with a similar discussion, with the required architecture defined in the answer. Instead of combining the separate models, you can create a model which uses image and numerical data side by side. Keras allows you to use different types of data using multi input structure via functional API. And ...


1

The first neural network machine was the stochastic neural analog reinforcement calculator (SNARC), built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we ...


4

Here is a paper with the mathematical definition of each term: Let Nt,n,σ,L be all target functions that can be implemented using a neural network of depth t, size n, activation function σ, and when we restrict the input weights of each neuron to be |w|1 + |b| ≤ L.


1

I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then $min_{\phi}max_{\theta}L(G_\phi, D_\theta)$ starts to look a lot like $max_{\theta}min_{\phi}L(G_\phi, D_\theta)$. This is problematic because in the latter case the ...


1

Read on Fully Convolutional Networks (FCN). There is a lot of papers on the subject, first was "Fully Convolutional Networks for Semantic Segmentation" by Long. The idea is quite close to what you describe - preserve spatial locality in the layers. In FCN there is no fully connected layer. Instead there is average pooling on top of last low-resolution/high-...


1

Lets start with question 1) how does JS-divergence handles zeros? by definition: \begin{align} D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\ &= \frac{1}{2}\sum_{x\in\Omega} [p(x)log(\frac{2 p(x)}{p(x)+q(x)}) + q(x)log(\frac{2 q(x)}{p(x)+q(x)})] \end{align} Where $\Omega$ is the union of the domains of $p$ and ...


3

I have had similar thoughts about neural networks before. Convolution layers are layers of two dimensional nodes effectively passing the spacial data so why don't we use two dimensional hidden layers to receive information out of them. I'm sure someone has used this type of implementation before. I believe the papers bellow are using this. Part of the ...


1

The facenet model is just the head of the model. The architecture is similar to the enocdr part of an autoencoder, but it uses supervised learning instead of unsupervised learning. The network is called a siamese network The triplet loss helps make the embeddings more representative of the input image/person, with the embedding distance being as large as ...


1

Here are two review articles: Elsken, Metzen, Hutter: Neural Architecture Search: A Survey (2019), Journal of Machine Learning Research 20, 1-21 He, Zhao, Chu: AutoML: A Survey of the State-of-the-Art (2019)


2

The problem originated because of the nature of the code. Code: https://github.com/AISangam/Facenet-Real-time-face-recognition-using-deep-learning-Tensorflow/blob/master/classifier.py model = SVC(kernel='linear', probability=True) model.fit(emb_array, label) class_names = [cls.name.replace('_', ' ') for cls ...


0

If you're really just trying to find long contiguous flat regions in a sequence, you do not need machine learning. Your PI is mistaken. You would be better off simply writing a short data processing program. Your program could find the finite differences between adjacent datapoints, and then count whether a long string of them are below some threshold to ...


1

TL;DR: You'll need to store a little bit more to perform backward passes. You'll need to store data from the forward pass. This stored information is used for calculating the gradient. Overview (warning: not trivial) I know the weights can just be stored in an array You'll need a little more: To update the weights you need to keep a "cache" of the ...


0

In image processing CNNs are usually used to create weighted filters for focusing in on the image features which are most important for making predictions. Keras is one of the libraries used to examine images in this way. With this type of analysis you will need labeled and unlabeled data you want to create a network that inputs a photo extracts the flat ...


1

https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro Is relevant. based on my reading when you have a NN and do Binary crossentropy on what you might call 'Linked category data' the accuracy can tend to be better than in a Categorical crossentropy model. The binary aspect implies ...


2

Facenet is a Siamese network. It's basic architecture is this: The input(a face) is fed through a deep convolutional neural network and also a fully connected layer at the end. The fully connected layer at the end output an embedding of the input image which is a predefined size. The embedding can contain feature that human understand or maybe not. The ...


4

The following articles Ising models for networks of real neurons (2006) by Gasper Tkacik et al. Deep neural networks for direct, featureless learning through observation: The case of two-dimensional spin models (2018) by Kyle Mills et al. Inverse Ising inference by combining Ornstein-Zernike theory with deep learning (2017) by Soma Turi, Alpha A. Lee et al. ...


Top 50 recent answers are included