Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

Filter by
Sorted by
Tagged with
4
votes
3answers
116 views

What are the typical sizes of practical/commercial artificial neural networks?

I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others. The only thing I found about Tesla is ...
0
votes
1answer
14 views

Tensforflow schedule - does not change boundaries [closed]

I'm trying to manipulate the learning rate with tf PiecewiseConstantDecay. I can easily check if the algorithm switches learning rate values, because one rate is extremely low 1e-20 !! However, NO ...
-1
votes
0answers
28 views

what are recirculation neural networks? [closed]

I want to know about re circulation neural networks. Are they even real or hypothetical? I tried to gather information but I didnt succeed. Resources for further reading will be appreciated.
1
vote
0answers
27 views

Is there literature on Neural Network with activation functions of bounded domain?

I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
1
vote
0answers
27 views

Neural Network for Picking Parameters of a Nonlinear Function to Data Points

I'm trying to make a neural network in pytorch that picks the parameters of a nonlinear function, the radius and (x,y) center of a circle in the example below, based on a sample of values from the ...
1
vote
1answer
39 views

In practice, are perceptrons typically implemented as objects?

I'm fairly new to ANNs. I know the general structure, the math, and the algorithms behind them. I figured the logical next step on my journey to fully understanding them would to be implement one ...
-1
votes
0answers
21 views

train and validation in a multi model pipeline

I have a pipeline which contains two NN blocks one after the other (the second gets as input the first output). I was wondering how to train and validate the two blocks. Split to train, val and test. ...
0
votes
1answer
49 views

How to build a word recognizer with text as images using CNN? [closed]

I am new to machine learning. I tried to use this CNN model, which was originally used for handwritten character recognition, for handwritten word recognition, but it is not working. Can anyone share ...
0
votes
1answer
50 views

Is a genetic algorithm efficient for a snake game?

I am working on a DIY project in which I want to be able to train a neural network to play Snake. Is a genetic algorithm an efficient way of training a network for this application? For a GA, what ...
2
votes
1answer
77 views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
1
vote
0answers
25 views

How to prove that a regularisation method simplified a neural network?

There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural ...
0
votes
0answers
56 views

What could be a good way to visualise the feature extraction process with MobileNet?

I am trying to create a visualisation for how transfer learning (feature extraction in particular) works with MobileNet. With the ml5.js library, you can extract a ...
1
vote
1answer
27 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
1
vote
1answer
44 views

How do transformers understand data and answer custom questions?

I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly ...
0
votes
1answer
33 views

What is the correct formula for updating the weights in a 1-single hidden layer neural network?

I'm creating a neural network with 3 layers and no bias. On internet I saw that the expression for the derivative of the weights between the hidden layer and the output layer was: $$\Delta W_{j,k} = (...
1
vote
0answers
13 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
0
votes
0answers
9 views

Choosing the size of the network for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system. After doing some hyperparameter tuning with various sizes for embedding and dense layers sizes, from ...
0
votes
0answers
25 views

How to understand the phrase “conditioning on an input” for a neural network?

Suppose I have a dataset $D_1$ with size $n$ and each training sample has $m$ attributes/features. So, my neural network has $m$ neurons at input layer, i.e., $D_1$ has $n$ samples of size $m$. ...
0
votes
0answers
17 views

When a deep learning paper mentions projections does that mean no bias?

Sometimes a paper mentions a projection layer where some dimensionality is projected onto another to enforce some downstream dimensionality matching requirement. I actually don't have any links at ...
0
votes
1answer
32 views

How does Keras BatchNormalization work?

I have read some articles and watched some videos by Andrew Ng stating that it makes more sense to use batch normalization before applying the activation function. ...
1
vote
0answers
10 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
0
votes
1answer
59 views

What type of ANN architecture to choose?

I have $N$ number of teachers each of which has an input feature vector ($25$ dimensional) consisting of positive numerical values for different quality of aspects (for example: lecturing ability, ...
0
votes
0answers
14 views

Monotonically increasing Siamese neural network

I want to design a Siamese neural network for which there are n inputs which are all positive and there is one output which is also positive. How can I enforce the condition that the input/output ...
0
votes
1answer
26 views

How can abstract graphs be recognized by neural nets?

Recognition of optical patterns (as pixel maps) by neural networks is standard. But optical patterns may be only slightly distorted or noisy, and may not be arbitrarily scrambled – e.g. by ...
2
votes
1answer
23 views

What is the difference between multi-head and normal output?

Let's say that I have a neural network with 2 heads. The first consists of X neurons. The second consists of Y neurons. I have these 2 heads because I want to predict 2 different variables. And I can ...
0
votes
0answers
25 views

How does pairwise comparison training work in XGBoost's XGBRanker?

I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well. I want to find out how the XGBRanker ...
2
votes
1answer
71 views

Which ANN structure to use?

Let $\mathcal{S}$ be the training input data set where each input $u^i \in \mathcal{S}$ has $d$ features. I want to design a ANN so that the cost function below is minimized (the sum of square of ...
-1
votes
1answer
29 views

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
2
votes
1answer
63 views

How can we get a differentiable neural network to count things?

Imagine I have images with apples in them. I want to train a neural network which can count the number of apples in each image. BUT, I don't want to use a detector, then count the number of bounding ...
0
votes
0answers
11 views

How can I prune BERT layers

I would like to finetune BERT on SQuAD and then evaluate the output from each layer (so from using 1 layer to using all 12). I know you can prune heads using Huggingface but was wondering how could ...
3
votes
2answers
59 views

Why some neural network models in the 1980s shown as circuit models

I am familiar with the currently popular neural network models that have weights and are trained with backpropagation and gradient descent. However, I came across a different type of neural network ...
1
vote
0answers
18 views

Which methods for weight initialization in Neural Networks are currently common practice?

I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
0
votes
0answers
7 views

Should I L-2 Normalise outputs in Siamese Neural Neural Network for distance computation for Triplet Loss or not?

I am building a Siamese Neural Network for Images (CNN) which uses the FaceNet's Triplet Loss as its loss function. I found a good Implementation here where we build a model and the outputs from the ...
0
votes
0answers
16 views

Adversarial Attacks and interpolation methods

I am attacking a model. The model is a simple CNN and PGD is used. The model runs on 112x112 ImageNet dataset. So I first load images as 224x224 and use interpolation function to downsample it to ...
1
vote
0answers
34 views

Which approach best suits vector encodings?

I want to build a model that when given two vectors, outputs the probability of one vector being the encoded form of the other. I have 2 strategies for this: (Dataset available) I can directly feed ...
0
votes
4answers
47 views

Oversampling of Balanced Dataset

I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling but they work best for ...
0
votes
0answers
16 views

Word2vec - CBOW and Skipgram comparative study for big data

Word2vec - CBOW and Skipgram comparative study As a part of my thesis I am creating a comparative study of CBOW and Skipgram for big data. My input is a Wikipedia dump and I have created a single .txt ...
1
vote
1answer
22 views

How do I take the correct classification predictions of an ml algo (i.e. random forest/neural net) and sort the instances in each category?

I am trying to sort the instances within each of 5 classification categories in a dataset that has been put through both a random forest classifier and a neural network with 99% accuracy on each. ...
0
votes
0answers
20 views

Backpropagation - what does rate of change calculated from the partial derivatives actually relate to?

I understand conceptually how backpropagation works according to the chain rule, and I understand that partial derivatives calculate the rate of change of a function containing multiple variables with ...
2
votes
2answers
88 views

Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the "The board is oriented to the perspective of the current player." I ...
0
votes
1answer
48 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
0
votes
0answers
10 views

How to write activation function with a higher order tensor in Keras? [migrated]

I want to create a paricular neural network in Keras. In this neural network I use layers given by $$ f(x) = C_k(\underbrace{x,\dots,x)}_{\times k}+\phi(w^\intercal x+b) $$ The expression $\phi(w^\...
3
votes
1answer
60 views

Are neural networks invertible?

I am interested in learning about the inverse of neural networks and I would like to understand about the invertibility of neural networks, as for example described in On the Invertibility of ...
0
votes
0answers
15 views

Can transformers be used to improve regression?

I was recently reading a bit about transformers and I don't understand them very much but I was wondering if anyone knows if any of their techniques such as attention mechanism or anything has been ...
0
votes
0answers
10 views

Why is my LSTM model predicting accurately for only a few values and showing drastic aberration later?

I am training an LSTM model using stock data for time series forecasting and the results are a little confusing to me. This is the prediction I get after 5 epochs. And this after 100 epochs. Why the ...
1
vote
1answer
34 views

How to design fitness function for multiple objectives?

I am currently building a neural network with genetic algorithms that learns to fly a 2D drone to a target. My goal is that it achieves all tasks as fast as possible, but I want the drone to also fly ...
0
votes
0answers
25 views

Identifying if a model is over or under-fitting via graphs

I am working on a Neural Network and have plotted the performance of my model. However the plots seem not to fit the "trends" (which help you identify the issue with your model) presented in ...
1
vote
0answers
31 views

Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?

In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
0
votes
0answers
40 views

What are the math theorems regarding the Multilayer Perceptron?

I've come across a theorem "Convergence theorem Simple Perceptron" for the first time, here-> https://zaguan.unizar.es/record/69205/files/TAZ-TFG-2018-148.pdf, page 27, (is in Spanish) ...
4
votes
2answers
57 views

What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), ...

1
2 3 4 5
42