7

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-...


6

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of ...


4

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...


3

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can find two very good papers here, and here). During training, to ensure the same size of the input and output they can just search for the longest sentence they ...


3

Yes, there are different ways. What I think you are looking for is under the research field of Localization and Mapping. Which divides in the following subfields: For getting current (the robot) position and trajectory go to models for Odometry Estimation For getting a representation of the world around the robot go to models for Mapping If you want both of ...


3

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than ...


3

I'll answer in a couple of stages. I feel somewhat lost as to what the input for the NN should look like. Your choices boil down to two options, each with their own multitude of variants: Vector Representation: Your input is a vector of the same size as your vocabulary where the elements represent the tokens in the input example. The most basic version of ...


3

There are tasks in computer vision where recurrent neural networks (RNNs) can be useful because there's some sequential sub-task in the main task. For instance, in the paper Long-Term Recurrent Convolutional Networks for Visual Recognition and Description, the authors investigate the use of a neural network that is both recurrent and convolutional to solve ...


2

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step. $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation) According to to section 6.1 of the paper, the input is a tuple represented ...


2

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer. Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (...


2

Unfortunately, this is not possible. The normal equation can only directly optimise a single layer that connects input and output. There is no equivalent for multiple layers such as those in any neural network architecture.


2

after reading some literature in the area I'd recommend the following: Try using Convolution Neural Networks (CNNs) this paper outlines some really good points on why should you use CNNs. Try a Combination of the different layers in the same model. Start with some Convolution Layers, then some LSTMs and then a couple of Dense Layers followed by Dropout


2

Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers less than 1 and gradually it will become weaker and weaker and when it arrives to the first layers, it's so weak that makes almost no change in initial layers ...


2

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method: For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the ...


2

Hopfield networks, a special case of RNNs, were first proposed in 1982: https://www.pnas.org/content/79/8/2554 Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found here: https://blog.exxcellent.de/ki-machine-learning


2

Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are: $$ h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h) $$ $$ y_t = \sigma_y(W_y h_t + b_y) $$ Here $W_h$ is input to hidden state matrix, and $U_h$ is hidden state to hidden state, $W_y$ is hidden state to ...


2

In short, repetition with feedback. You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work around this when desired input and output is more variable. The most common approaches are: Padding: Give the ML model capacity to cope with the largest expected ...


2

Not quite sure about RNN & LSTM (and it always depends on the task), but for CNN the answer is clearly no; CNN routinely include FC layers. Quoting from the highly popular (and recommended) Stanford course CS231n: Convolutional Neural Networks for Visual Recognition: ConvNet Architectures We have seen that Convolutional Networks are commonly made up of ...


1

This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$ Code Sample for $f_t$: import numpy as np import math def sigmoid(values): sigmoid_applied = [] for value in values: result = 1 / (1 + math.pow(math.e, -value)) ...


1

I finally grasped the concept of word embedding. Thanks to @nbro, after reading the 2 articles s/he recommended What Are Word Embeddings for Text? and Word embeddings the 1st article gives me a good idea about the big picture of the Word Embeddings; whereas the 2nd article is actually the one which clears my mind. I am an visual person, I understand ...


1

The specific term you are looking for is "word embedding" and not just "embedding". How to numerically represent textual data? Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or ...


1

This is a difficult problem. First, how do you define 'subject'? Do you have a (closed) lists of labels you want to assign? What about subjects that overlap, or don't occur in your list? What even is a subject? This is a non-trivial issue. Second, and this is even harder, how do you want to recognise subjects? A simple solution could be using a list of ...


1

Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.


1

I have not implement the backprop of a bi-directional RNN from scratch so I can't be sure my answer is correct but I hope it helps. You can see how bi-directional RNN works from this video from Andrew NG. I got the image below from that video: For more clarity: So if you know how to backprop through a simple RNN, you should be able to do so for bi-...


1

This sounds like a problem that's best solved with a simple non-AI algorithm. If you just enumerate the coordinates in a regular order (rows, colums, zig-zag, hilbert curve) and emit the coordinates where the image has a '1' you're meeting your requirements. Is there any specific reason you want to use an AI algorithm which is most likely worse than this?


1

It doesn't drops rows or columns, it acts directly on scalars. The Dropout Layer keras documentation explains it and illustrates it with an example : The Dropout layer randomly sets input units to 0 with a frequency of rate After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, ...


1

One of the essential pre-processing we do on the corpus involves treating the variable-length sentences to a fixed length. There are various ways in which we can do this: Truncate This involves reducing the length of all the sentences to the length of the shortest sentence in the corpus. This is generally not done as it reduces the amount of information ...


1

You want to look at recurrent neural networks.


1

I'm currently working with Temporal Convolution Networks (TCNs) for making predictions with time series data (link to article here: https://medium.com/@raushan2807/temporal-convolutional-networks-bfea16e6d7d2). These types of networks, like other types of convolutional networks for time series, use a dilated convolution operation, which, unlike the standard ...


1

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN. Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used ...


Only top voted, non community-wiki answers of a minimum length are eligible