Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

Filter by
Sorted by
Tagged with
4 votes
1 answer
433 views

What are the differences between Bytenet and Wavenet?

I recently read Bytenet and Wavenet and I was curious why the first model is not as popular as the second. From my understanding, Bytenet can be seen as a seq2seq model where the encoder and the ...
0 votes
1 answer
47 views

How special tokens in BERT-Transformers work?

"[SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate ...
0 votes
1 answer
8 views

Machine Learning book for fundamentals - Simon Haykin vs. Christopher M. Bishop

Since I started studying Machine Learning, I was torn between two books in this area, and I could never decide which one is the best to follow. The first book is widely used and known: Pattern ...
0 votes
1 answer
19 views

Shuffling vs Non-shuffling train/test set yields drastically different results

I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input ...
0 votes
1 answer
35 views

Having the negative cases in the same batch vs. shuffling the dataset

I am working on a model for an NLP task. The model encodes the text and has a regression output layer. In this task, from each instance (positive), I create several negative cases using a specific ...
0 votes
0 answers
10 views

Why is the derivative of activation function all positive?

All the activation functions I see have positive derivatives. Will negative ReLU work as well as its positive counterpart or will it lead to instability?
0 votes
0 answers
8 views

multi-agent RL training schemes

I would like to do a project in multi-agent reinforcement learning. I have a quite simple self-made environment which is ready for RL training in single agent fashion. It includes an agent which can ...
0 votes
0 answers
6 views

Shuffle data inside learning sample in order independet transformer model

Does it make sense to create new samples with shuffled items "tokens" inside a learning sample for the order independent (no positional encoding) transformer model to improve model accuracy?
0 votes
0 answers
30 views

Hot to calculate Maximum Normalized log Probability for Active Learning with BERT

I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al.. With n being the sequence length, yi the label of word i. Xij is the ...
1 vote
0 answers
13 views

How to use NN to generate a model which produces given distributions?

For a non-Markovian random walk, each step can go up or down. And for the $n-th$ step, its step size $s(n)$ may depend on the path of walk, and the probability for going up or down may also depend on ...
1 vote
1 answer
24 views

Why do we subtract logsumexp from the outputs of this neural network?

I'm trying to understand this tutorial for Jax. Here's an excerpt. It's for a neural net that is designed to classify MNIST images: ...
0 votes
1 answer
32 views

Autoencoders: Where does the encoder end and the decoder begin?

Consider a simple Autoencoder neural net: ...
-4 votes
0 answers
24 views

Can you show me my future success? [closed]

I've been struggling alot with understanding what is meant for me because I cannot see it. Alot of spiritual facts say that I am meant for greatness but I find it hard to perceive what is being said ...
2 votes
1 answer
98 views

How does backpropagation know which weights to change?

I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the ...
0 votes
2 answers
296 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
2 votes
1 answer
482 views

Could the normalisation of the inputs make the neural network insensitive to changes in the inputs?

When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features ...
2 votes
2 answers
73 views

Maximize loss on non-target variable

I have a neural network that should be able to classify documents to target label A. The problem is that the network is actually classifying label B, which is an easier task. To make the problem more ...
2 votes
1 answer
70 views

Is there a way to update the neural network to fit the new data without the time required for retraining?

I built a basic neural network in MATLAB. The neural network classifies points on the X-Y axis system into two classes (0 and 1). (I try to get the function that represents a shape from this photo) ...
0 votes
0 answers
6 views

linear layer bert sentence embedding

I have a special situation where I need to embed a sentence with bert-sentence transformer and get a numeric value for it. This is not possible with this model, ...
1 vote
1 answer
120 views

Computation of initial adjoint for NODE

I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model ...
0 votes
0 answers
6 views

How to obtain the graph in the video tutorial

I am watching this video tutorial https://www.youtube.com/watch?v=gmjzbpSVY1A&t=1002s . At 16:34, the author show the variation of the line when weight change from 0.97 to -1, the below graph is ...
1 vote
1 answer
42 views

Why and when do we use ReLU over tanh activation function?

I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
1 vote
1 answer
19 views

How to use information on a function to design a neural network learning that function?

I have a function $g$ that takes a vector $x$ of size $n$ and an integer $k$ in $1, \ldots, n$. I know this function is of the form $$g(x,k) = G\left(\sum_{i=1}^k f(x_{i})\right),$$ where $f$ and $G$ ...
0 votes
0 answers
16 views

Difference between training algorithms

I'm using GPS Data for my Total electron Content (TEC) Prediction, for which I'm using Non-linear Autoregressive with External (Exogenous) Input (NARX) Model. My question is what's the difference ...
0 votes
0 answers
10 views

How state is combined with action in crtitic networks?

Actor-critic networks are present in deep reinforcement learning algorithms. Actor-network takes a state as input and gives action as output. Critic-network takes state and action as input and gives a ...
0 votes
1 answer
26 views

In a neural network's neuron that has no activation function, to calculate the delta for the neuron during back propagation do you use a derivative?

I have a neural network that is composed of an input layer, two hidden layers and an output layer. The topology is [151, 200, 100, 1] I am using ReLU activation function on the neurons that are in the ...
86 votes
4 answers
62k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
1 vote
1 answer
185 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
3 votes
1 answer
72 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
0 votes
1 answer
24 views

Do we feature scale one hot encoded variables in neural networks? [closed]

If I have a categorical variable in my neural network which I encode using one hot encoding, do I need to feature scale it along with other features before training the artifical neural network? or do ...
1 vote
0 answers
14 views

What is the next step in top-down brain simulation after spiking neural networks?

This paper from Yamazaki et al. describes a 68 billion spiking neural network model of the cerebellum. The simulation was about 600 times slower than real time, and the cerebellum is perhaps one of ...
0 votes
1 answer
13 views

U-Net Maxpooling vs Convolution

Hello I'm implementing a CycleGAN and most of the other implementations I've seen on the internet use Convolution with stride 2 instead of a Maxpoolinglayer for downsample. On to my question, why ...
2 votes
2 answers
149 views

Validation Loss Fluctuates then Decrease alongside Validation Accuracy Increases

I was working on CNN. I modified the training procedure on runtime. As we can see from the validation loss and validation accuracy, the yellow curve does not fluctuate much. The green curve and red ...
0 votes
1 answer
35 views

How can I demonstrate a novel ML classification algorithm has value?

I designed a ML classification algorithm that's simple, efficient, and effective. It's not perfect, but seems to be widely applicable across domains. I'd like to submit it for publication, but I don't ...
1 vote
1 answer
22 views

fondamental question about regularization techniques to solve overfitting problem in neural networks

I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset. Using some classical regularization ...
0 votes
0 answers
7 views

Simple dimension unmatch problem of a simple neural network

In this simple neural network: the derivative for the cost function J when assuming binary cross entropy loss would be If we assume that the dimension of X is 2x1, then wouldn't A1 be 2x1 and A2 be ...
0 votes
1 answer
14 views

How to handle anomaly detections with multiple different timeseries' from network traffic?

I would like to implement an anomaly detection algorithm on multiple timeseries' from different network users. Since each user has different behavior and network traffic usage, my question is how can ...
0 votes
0 answers
7 views

Are there any general guidelines for the architecture of critic network based on actor network?

Suppose the actor-network looks like the following ...
1 vote
2 answers
336 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
1 vote
0 answers
18 views

Combining Different Inputs in a Neural Network for Numerical Integration

I am building a NN that numerically integrates a non-linear differential equation. Given a DE: $$ \frac{d}{dt}x(t) = f(x, p) $$ with solution $x \in \mathbb{R}^n$ and parameters $p \in \mathbb{R}^m$, ...
0 votes
1 answer
18 views

Question regarding matlab computer vision application and color recongnition [closed]

I am thinking of choosing a computer vision project for my school project(detect crack on surface) and the duration I have is roughly 4 months. With no prior knowledge in neural network, is matlab ...
0 votes
0 answers
9 views

Extracting behavior (switch On/Off) of an electric load from unlabeled time series data

Following are the details of my dataset: sampling frequency: 1 Hz No. of useful features: 10 The time series dataset is from household wherein I'm required to find ...
0 votes
1 answer
42 views

Entirely linear neural network learning non-linear function

I have a neural network that's trained on a sine wave. It uses a lookback of 20 to see what the last 20 predictions were and predict the next value. This network has only a single Linear layer (input ...
2 votes
1 answer
226 views

How can we get a differentiable neural network to count things?

Imagine I have images with apples in them. I want to train a neural network which can count the number of apples in each image. BUT, I don't want to use a detector, then count the number of bounding ...
1 vote
1 answer
26 views

Can the output layer be connected to multiple layers?

Normaly, the output layer is only connected to the second last layer. Is there any model that the output layer is connected to multiple layers (For example, the second last layer AND the layer before ...
0 votes
1 answer
17 views

How does a sigmoid neuron act like a perceptron in this scenario?

I have been reading Michael Nielsen’s book online on his website at http://neuralnetworksanddeeplearning.com/chap1.html. I am struggling to understand the second exercise: When c approaches infinity, ...
0 votes
0 answers
25 views

Why Is There The Term 1/m In Backpropagation

In backpropagation the gradients are used to update the weights using the formula $$w = w - \alpha \frac{dL}{dw}$$ and the loss gradient w.r.t. weights is $$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} ...
0 votes
1 answer
30 views

Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com

I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he ...
0 votes
1 answer
31 views

How are Neural Networks protected from false training data?

Suppose the training data there exist an element of some data being misleading and some being right, how could the Neural network be trained so that it could filter the right data from the wrong one? ...
0 votes
0 answers
33 views

Neural Networks in molds industry

I recently began an internship at a moldmaker, where I'm supposed to learn about NN and how to use them (as you can imagine, I don't know much). Each mold is composed of many pieces, and for each ...

1
2 3 4 5
46