New answers tagged

1

We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also ...


4

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed ...


4

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms. In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient ...


0

You can also pose your problem as co-reference resolution. Try Huggingface's neuralcoref library.


3

Sorry cannot directly reply to your comment as I posted without an account, and you were right! I replaced transposed layers with Upscale1D+Conv1D and that solved the issue. gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen) should become (notice that strides=2 becomes strides=1): gen = Upscale1D()(gen) ...


0

Basically, what you want to do is an anomaly detection at the fastest speed possible. You need to sample the image at two timesteps and if there is a difference you click. The smaller the difference between your timesteps the better it is. But, you would not need a neural network for it. Since, it is a single colour that will change. A color is represented ...


1

A perceptron is a linear threshold function. That means it has a weight vector $w$, and it outputs $w \cdot x > t$, where $x$ is the input vector and $t$ the threshold. Naïve Bayes makes the assumption that all features are independent (hence the term naïve). It predicts the most likely class by using Bayesian probability, for each class multiplying the ...


0

Naive Bayes is a generative algorithm while Perceptron is a discriminative algorithm. That is the main difference.


0

neural networks can solve all taylor series polynomials meaning a NN is an generalized linear model. Most function f(y) can be solved with neural networks. However, many matrix operations can not be generalized for a neural network to solve like determinants. Operations like rotation, scale, and transform also can not be generalized. you can solve all ...


-2

Yes, RNN can work on the functions you have mentioned. In fact, neural networks can approximate anything (Universal Approximation Theorem). This question also reminds me of Neural Turing Machine. But, it would be a complete waste to use RNNs or NNs for such a task.


0

It is indeed true that neural networks are just another ways of curve fitting. In fact, as I learned regression after I learned neural networks, I shout out "neural networks are just more sophisticated curve fitting!". However, as Dave said, it can approximate any function in practice. See Google's neural net playground for an interesting animation....


4

You can already do this with some neural networks, such as GANs and VAEs, which are generative models that learn a probability distribution over the inputs, so they learn how to produce e.g. images that are similar to the images they were trained with. Now, if you're interested in whether there is a black-box method, i.e. a method that, for every possible ...


1

I suggest you look into link prediction. I have had good luck with the StellarGraph library. They have several algorithms implemented, including GCN. Link prediction is a binary classification problem. Given two nodes, $v_i$ and $v_j$, does there exist a link between them? Using a library like StellarGraph will also produce node embeddings while performing ...


1

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here. The hinge loss function is defined as follows $$ \ell(y) = \max(0, 1-t \cdot y) \tag{...


0

I have one possible solution for you inspired by real-time face recognition system. It is a similar case to you. Create embeddings for each class i.e. person or in your case, a class. (Using Siamese network with ArcFace loss) When a new image comes, take L1 or cosine distance with the embeddings. You will need a threshold above which you create a embedding ...


1

The link you have mentioned is using Dense layers. One thing to start with would be to use 1D CNNs (they will capture some local information). Also, since sequence matters in your case, refrain from one-hot encodings (just 1, 2, 3, 4). And for the 2D matrix, use a 2D CNN. Then, flatten your encoding for both 1D CNN and 2D CNN, then, finally combine them. ...


2

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention. Some examples include: In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by $$ f(x) = \sigma(Wx + b)$$ ...


1

I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you. The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log ...


6

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of ...


1

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete? Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in range $[0,1]$ and the sum across all allowed actions is $1$. The two ...


0

There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a DQN with multiple outputs. The other, less efficient way, includes taking in an state-action as input and outputting a single real value - this approach is ...


1

As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the squared bellman error you need first calculate the scalar target $y = r(s, a) + \max_a Q(s', a)$. Then, to train the network we take a vector $\mathbf{x'} = \...


0

Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue On eq(6) of the relevant blog post, they identify the weight matrix of a discrete, binary Hopfield network as $$ \boldsymbol{W} = \sum_i^N x_i ...


3

If you look into the top conferences on machine learning and neural networks, such as NeurIPS, ICLR, and ICML, you will find many papers related to neural networks and deep learning, given that these are still very hot/promising topics. However, occasionally, you will find accepted papers that do not involve neural networks. Here's a small list of them that ...


0

I am not aware of any empirical results regarding this question. But in theory, adding a regularization term shall make the learning task actually even harder, since there is suddenly a second loss term that the network has to be optimized for, which is not even directly related to achieving the original task of fitting the model to the data. It is true that ...


1

From your question there is no indication that there is any pattern to these digits. If there were, the recommendation for an LSTM or RCNN would make sense. In the case of random values, I have found that a two or three layer CNN that then descends through two parallel dense networks does an excellent job identifying CAPTCHA style random characters. One ...


1

After some research on the internet, I realized that using VOSK toolkit in python, it can be found (detect) any particular word in audio file or real time audio streaming. https://alphacephei.com/vosk/


1

Your task is text recognition, however your code is for classification task. So you need to use different approach for that. You mentioned that you're going to give model 123 and get 123. But you can not do that with just convolutional networks. Images with text are sequential, so you need to use CRNN(Convolutional-Recurrent-Neural-Networks), LSTM(Long-Short-...


Top 50 recent answers are included