# Tag Info

4

One of the things you may have missed out in your design is some arrow going from the last layer in the layer side back to the first layer. e.g. If you're thinking some thoughts you'll want those to keep going round and round in your head. At the moment while your design would have some way to learn and react to the environment, it wouldn't have any short ...

4

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases) If you are using gradient descent as a training method, then differentiability ...

4

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers) The better question is ...

3

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail. Using backpropagation to compute the gradient is an ...

3

I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS. It's usually called the loss function (and, in general, objective function) and often denoted as $\mathcal{L}$ or $L$ (or something like that, i.e. it is not really important how you denote it). The specific function used as a ...

3

Welcome to AI Stack exchange! You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing. So what do we ...

2

A neural network is composed of continuous functions. Neural networks are regularized by adding an l2 penalty on the weights to the loss function. This means the neural network will try to make the weights as small as possible. The weights are also initiallized with a N(0, 1) distribution so the initial weights will also tend to be small. All of this means ...

2

From here: Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.

2

A couple of points: Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high. Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea. Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations. Model ...

2

It depends. It could give you a boost or it could not. Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches). Ideal World: We optimize the network as a whole to gain better course grained features over the ...

2

Mathematical Exploration let $\Theta^+$ be the pseudo-inverse of $\Theta$. Recall, that if a vector $\boldsymbol v \in R(\Theta)$ (ie in the row space) then $\boldsymbol v = \Theta^+\Theta\boldsymbol v$. That is, so long as we select a vector that is in the rowspace of $\Theta$ then we can reconstruct it with full fidelity using the pseudo inverse. Thus, ...

2

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value. For ...

1

Maybe you are looking for a combination of a version control system (like git and Github) and a tool like comet.ml. In the past, I used comet.ml to keep track of different experiments performed with different hyper-parameters or different versions of the code. There are other alternatives to comet.ml, such as sacred, but they may also have different features ...

1

To emphasize (and this is not emphasized in this answer), in the case of neural networks, the biases or, more precisely, the connections (or weights) between biases and other neurons are also learnable parameters, so the back-propagation algorithm calculates a gradient of the loss function that contains the partial derivatives with respect to the connections ...

1

In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a ...

1

So, if I go the opposite way, start with my y and predict an x, and then ask for the inverse of that - I get really good results (actually - 100% accuracy). i.e. model = Sequential([ Dense(784, input_shape=(10,), activation='sigmoid'), ]) model.compile(loss=keras.losses.binary_crossentropy, optimizer=keras.optimizers.Adam(0.01), ...

1

The perceptron convergence theorem states that any architecture will lead to a correlation between the data. Yes, you can!

1

It depends on the data. If it is structured like form data, then you might not need AI at all — simple regular expression patterns might be fine. This would apply for example to address data. If you have the word street followed by a colon, followed by some text, it seems fairly obvious that this is the name of a street, and possibly also a house ...

1

There are two weight-initializing methods for neural networks: 1-Zero initializing 2-Random initializing https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78 If you choose zero initalizing method in every train loop, you may get same results OR you can use transfer learning according to your problem, it allows ...

1

I don't think you can. Say a NN with 3 layers gives an accuracy of 95.3% and another NN with 4 layers gives an accuracy of 95.4%. Then there is no guarantee that the 4 layer NN is better than the 3 layered NN. Since with different initial values the 3 layer NN might perform better. You could run multiple times and probabilistically say that this is better, ...

1

Simply said, predicting pseudo random number is just not possible for now. Pseudo random numbers generated now have a high enough "randomness" so that it cannot be predicted. Pseudo random numbers is the basis of modern cryptography which is widely used in the world wide web and more. It may be possible in the future through faster computers and stronger AI, ...

1

Getting the intent of the sentence is not an easy task. To get you started on what to do, have a look on word vectors. You can also download pre-trained word2vec models. They help in getting similarity of words and reasoning with words. To get the intent of a sentence, you can use LSTM. Fun fact most NLP algorithms strip away punctuation with is sufficient ...

1

There are probably multiple different explanations and reasonings, but I can offer you one. If your output vector contains negative values, to get something that's related to probabilities (all components positive, summing to $1$) you cannot do what you suggested because you can possibly get a negative probability which doesn't make sense. Good property of ...

1

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you only need one hidden layer to approximate the multiplication on a compact, note that you need to apply a non linear activation on the hidden layer to do this.

1

Here's a link to my answer on CV Stack Exchange, where I have mentioned about latent spaces and some deep learning models that learn these representations: https://stats.stackexchange.com/questions/442352/what-is-a-latent-space/442360#442360 In short, deep learning models for Domain Adaptation, Computer Vision, Natural Language Processing, Recommendation ...

1

If is a truly a random number, and you could guess each of the next successive five in sequence, then you could win the lottery consistently. This is one of the first tasks many people try to do when first learning machine learning. If the lottery is truly a random physical process with fair, i.e., balanced ping pong balls, then you cannot predict which ...

1

A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions. The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one ...

Only top voted, non community-wiki answers of a minimum length are eligible