# Tag Info

13

This is a question of time series forecasting, since your numbers form a sequence. You may want to take a look at the "forecasting" tag at CrossValidated. If you have only 700 data points, ML/AI methods will likely not be very useful. Whatever you do, I would recommend you benchmark your chosen method against very simple approaches, like the ...

8

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious patterns", there are some kind of hidden patterns in your data. If not, what you have is not very different than random walk in 3 dimensions - which makes the ...

7

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...

6

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase). What exactly is parallelized? Also, it's worth ...

6

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/related works and conclusion. Find out why the method is proposed, what the well-known approaches are, what the intuition for the proposed methods are. Find out the ...

4

Since you only have only 700 observations, I would not try a deep learning approach. I think it is very unlikely that any Deep Learning approach will learn a non-obvious relationship with that little data. What you could try is create a set of features based on lags. Create a feature, that is lagged by 1, by 2, by 3, and so on. Also moving average of lagged ...

3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...

3

You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML. Anyway if you approximates these polygons as images you could maybe train a CNN. Look at the image classification networks which provide bounding boxes.

3

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals: \begin{align*} p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\ =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i}) \end{align*} This can be sampled by samping each of the conditional is ascending ...

3

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM. Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to ...

2

(Old question, I know...) It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often. Example for Sequence-to-sequence without two RNNs To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-...

2

One renowned example for the specified case is SeqGAN Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the ...

2

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer. Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (...

2

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as ...

1

To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length. However, the computational complexity comes from the matrix operations in the attention layer. Those are not trained; There are no trained parameters in the attention ...

1

Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech Recognition with the Transformer Model), and text-to-speech (Neural Speech Synthesis with Transformer Network).

1

You can do this similar to the BIDE approach. It can be done like this: class TreeNode: def __init__(self, element, depth, count=0, parent=None): self.count= count self.element= element self.depth= depth self.subnodes= dict() self.parent= parent def __repr__(self): return f'{self.__class__....

1

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference. I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short. Either way, when a model is trained a certain way,...

1

When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens. OpenAI did some ...

1

In computational learning theory, the VC dimension is a formal measure of the capacity of a model. The VC dimension is defined in terms of the concept of shattering, so have a look at the related Wikipedia article, which briefly describes the fundamental concept of shattering. See also my answer to the question How to estimate the capacity of a neural ...

1

If you give it a name, it will probably be almost perfect model, as the number of hidden units are definitely enough to store the raw data of a name. However as a neural network it will still not be perfect. There may also be serious overfitting if you do this. A Seq2Seq model have way more parameters than necessary to just "remembers" all the possible names,...

1

He likely found this to a be a best practice to avoid over fitting, with a small data set if you only use small and easy to learn (less words -> less degrees of freedom) sequences then you open your model to the risk of over fitting that data set where as on a large data set that has alot more total information you can train on small sequences without being ...

1

Shouldn't the hidden state of the RNN from the previous timestamps be enough? It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras: Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training ...

1

In section 2.1 of the paper, the authors state that the goal of character-level language model is to estimate the following joint probability distribution $$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$ which is a joint probability distribution of all characters of a sequence of $T+1$ characters, where \$\...

1

Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. This typically produces worse accuracy scores because there are interdependencies within the text being produced - a model translating "thank you" could ...

1

I know it's too late to answer your query after 1.5 years The inference algorithm is explained at https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference Implementation of the algorithm is available at: https://github.com/pytorch/fairseq. One could get a complete understanding from their source code.

Only top voted, non community-wiki answers of a minimum length are eligible