4
In the past, I have used TensorFlow (1 and 2), Keras and PyTorch, so I will give an answer based on my experience. Currently, I use TF 2 and Keras (the version shipped with TF 2).
In my (but not only) opinion, TF 1 is really ugly and painful, given that it involves sessions, placeholders and, in general, you need to define the computational graph before ...
3
Your statement that researchers build their network from the ground-up using C++ or some other low level library couldn't be further from the truth.
You could take a look at this analysis showing the popularity of these two frameworks in the top ML conferences. The following Figure is taken from there.
In CVPR-2020, for example, TensorFlow and pytorch ...
3
TensorFlow was developed by Google and is based on Theano (Python library), while Facebook developed PyTorch using the Torch library. Both frames are useful and have a great community behind them. Both provide machine learning libraries to accomplish various tasks and do the job. TensorFlow is a powerful and deep learning tool with active visualization and ...
2
I understand your question as: "How did the author select the number of neurons in their hidden layer?"
The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more ...
2
There are a few things you could do to improve this NN, but are probably worth covering in different questions.
Your main problem though is that you forgot to reset the gradient after each training batch. You need to call optim.zero_grad() in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep ...
2
You cannot do this:
$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$
That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation ...
2
Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A ...
2
You want to compute the mean loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!
In your case:
You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So ...
2
Here is the commit
I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.
In [2]: histories = [deque(maxlen=5)] * 4
In [3]: histories ...
2
Yeah, it seems like it's a wrong implementation.
vals_ref_v is a matrix of 1 row, and 128 columns.
value_v.detach() is a matrix of 128 row
2
Trajectory size can be fixed, but in this case problem would be formulated as something similar to the multi-armed bandit problem where there is a single state and a set of actions to choose from. There is no sequential decision making since samples are not correlated, they are picked at random. So, if you take a batch of 20 examples then you would basically ...
2
If you know it is symmetric, then you could do a couple things.
Zero out a half.
Don't bother learning both halves of the image. Just put a zero mask over the upper or lower half of the output matrix and just have the network regress the other half. Just don't make the network do more work than it needs to do.
Learn both, but add symmetric loss
In your ...
1
Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call
1
This is not quite the loss that is stated in the paper.
For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\theta}(s_0)$. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic.
In general, we can write
$$v_\pi(s) = \mathbb{E}_{a\sim\pi}[...
1
As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN.
Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used ...
1
The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of mask to -inf (when it should be $0$).
Also, just an FYI, although you haven't focused on evaluation/prediction yet, I will explain the evaluation/prediction here as well for completeness, since it works so differently than ...
1
To answer the question in the title, your enclosed method is a valid way to use 2d convs after a flattened feature vector. However, the bad results you experience could come from the structure of your model or from the way you train it. Regarding you last question, it is very hard to give you an advice without knowing your intentions in detail. Regardless, ...
1
So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.
Now its important you do the split before entering the fit, and not do it for each epoch or something alike.
Once you do that and the fit method should be something like:
def fit(self, ...
1
If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, ...
1
You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words.
For each important ...
1
I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply $Q(a, s) - V(s)$, where $Q(a, s)$ and $V(s)$ with the same shape.
The call to detach() is important here as we don't want to propagate the PG into ...
1
In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!).
algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function
algorithms that ...
1
It seems that decaying the learning rate solved my problem. I changed learning_rate from 0.001 to 0.0001
1
I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked.
So think about your input guys
1
If your interest is positional information, encode it!
This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need
an example of a paper that encodes the 2D positional info: ...
1
The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the ...
1
BCELoss ( Binary Cross Entropy Loss) is used for binary classifier, which is a neural network that have a binary output, 0 or 1. It is not used for multi-output neural network like your case. For that kind of networks, you can use MSELoss or CrossEntropyLoss as your loss for the network.
For the calculation of BCE, it is shown on pytorch documentation.
...
1
Your forward function is not using the previous hidden state.
observe: you pass hidden but never use it.
def forward(self, input, hidden):
layer = self.hiddenWx1(input)
layer = self.hiddenWx2(layer)
a_next = self.tanh(layer)
z = self.z1(a_next)
z = self.z2(z)
y_next = self.softmax(z)
return y_next,a_next
1
I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and 9000th epoch's loss is greater about 0.2 alternate of 0.09
I also tried to change optimizer and learning rate but no loss didn't improve.
you can explore the modified code may help ...
1
How to export Pytorch Deep Neural Networks trained model to C++ program to use it from C++? [closed]
PyTorch now has a C++ frontend. I haven't tried it, but I'm sure you could use that.
Another option, which is more production-tested, is using a message passing framework such as ZeroMQ to communicate requests and results between Python and C++ executables.
Only top voted, non community-wiki answers of a minimum length are eligible
