1

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial. What you are seeing is most likely the result of a greedy sampling - the most probable next word is chosen from the ...


1

How about a Temporal Convolutional Network? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences. There are also claims that TCNs are comparable to RNNs in performance on common tasks, so ...


1

The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with "John went to..." and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model. As mentioned in the paper, the most common method to combat this (at the time) was using an ...


1

It seems this paper defines a "clock period ${T}_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T_i$ is smaller than period $T_j$.". This definition is, however, only in this paper, as far as I know....


Only top voted, non community-wiki answers of a minimum length are eligible