I recently read the paper Time-Series Generative Neural Networks and found the results that they reported quite promising (https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf). While looking at the code provided on the GitHub repo by one of the authors (https://github.com/jsyoon0823/TimeGAN) I however find a number of parts of code which do not seem to be explained in the paper, and whose motivation evades me.
Firstly, in the paper for both the embedder and generator sub-networks they condition the output of both of these networks at one time step on the network's previous outputs:
This is not done, however, in the code; the embedder and generator networks as implemented in the github repo are simple recurrent networks, whose output at a given timestep is determined by the current input and the hidden state. I wouldn't necessarily consider this a problem, as I would think that one can assume the relevant information from all the previous inputs is passed forward to the current output via the hidden state, but the fact that the networks are explicitly conditioned on their previous output, opposed to implicitly via the hidden state, is utilized in the training process for TimeGAN:
In TimeGAN, the generator is exposed to two types of inputs during training. First, in pure open-loop mode, the generator—which is autoregressive—receives synthetic embeddings (i.e. its own previous outputs) in order to generate the next synthetic vector .... we also train in closed-loop mode, where the generator receives sequences of embeddings of actual data (i.e. computed by the embedding network) to generate the next latent vector.
This discrepancy is ultimately tied then to another problem that the supervised loss function as defined in the paper cannot be evaluated since the generator can't be conditioned on the real embedder output:
Instead what is labeled as the supervised loss in the code is defined utilizing the output of a network known as the supervisor which is not mentioned in the original paper and has the following loss function:
G_loss_S = tf.losses.mean_squared_error(H[:, 1:, :], H_hat_supervise[:, :-1, :])
Whereby the variable H is the real output after being passed through the embedder network (H = E(x)) and H_hat_supervise is the real output after being passed through the embedder and the supervisor networks (H_hat_supervise = S(H) = S(E(x))). This new supervised loss seems unusual to me because it seems that we calculate the mse between the input and output of the supervisor network whereby we compare the last T-1 timesteps of input information to the first T-1 timesteps of output information. While I do believe that this should generally encourage the supervisor transformation to more or less obey the stepwise dynamics of the data, the form of this supervised loss, even after reading the authors explanation in the github repo (https://github.com/jsyoon0823/TimeGAN/issues/15) remains unclear.
There remains other issues with unifying my understanding of the code with that of the paper: The discriminator is not a bidirectional network (as stated in the paper), the discriminator seems to attempt to be tasked with classifying both latent representations of our generated sequence (following the transformation G(Z)) as well as the fully generated sequence (S(G(Z))), an entire T-length sequence is generated from one sampled noise vector from a normal distribution instead of a T-length sequence following T-samplings from a wiener process, furthermore it is unclear to me how k-step predictions would be made by the model as implemented in the Github repo, although the network introduced in the paper seemed perfectly capable of making them (by iteratively conditioning the generator on its previous outputs).
All in all I would greatly appreciate if someone with perhaps a bit more knowledge in the field, on the paper or just in general could maybe help to clarify some of the "discrepancies" I've identified above, or perhaps let me know if they had similar issues understanding the paper and comparing it with the available code. Thanks in advance for any help anyone can provide.