# Tag Info

3

In the past, I have used TensorFlow (1 and 2), Keras and PyTorch, so I will give an answer based on my experience. Currently, I use TF 2 and Keras (the version shipped with TF 2). In my (but not only) opinion, TF 1 is really ugly and painful, given that it involves sessions, placeholders and, in general, you need to define the computational graph before ...

3

TensorFlow was developed by Google and is based on Theano (Python library), while Facebook developed PyTorch using the Torch library. Both frames are useful and have a great community behind them. Both provide machine learning libraries to accomplish various tasks and do the job. TensorFlow is a powerful and deep learning tool with active visualization and ...

2

I understand your question as: "How did the author select the number of neurons in their hidden layer?" The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more ...

2

There are a few things you could do to improve this NN, but are probably worth covering in different questions. Your main problem though is that you forgot to reset the gradient after each training batch. You need to call optim.zero_grad() in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep ...

2

You cannot do this: $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$ That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation ...

2

Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A ...

2

Here is the commit I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue. In [2]: histories = [deque(maxlen=5)] * 4 In [3]: histories ...

2

Yeah, it seems like it's a wrong implementation. vals_ref_v is a matrix of 1 row, and 128 columns. value_v.detach() is a matrix of 128 row

1

You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words. For each important ...

1

This is not quite the loss that is stated in the paper. For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\theta}(s_0)$. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic. In general, we can write v_\pi(s) = \mathbb{E}_{a\sim\pi}[...

1

I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked. So think about your input guys

1

I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply $Q(a, s) - V(s)$, where $Q(a, s)$ and $V(s)$ with the same shape. The call to detach() is important here as we don't want to propagate the PG into ...

1

In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!). algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function algorithms that ...

1

It seems that decaying the learning rate solved my problem. I changed learning_rate from 0.001 to 0.0001

1

The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the ...

1

BCELoss ( Binary Cross Entropy Loss) is used for binary classifier, which is a neural network that have a binary output, 0 or 1. It is not used for multi-output neural network like your case. For that kind of networks, you can use MSELoss or CrossEntropyLoss as your loss for the network. For the calculation of BCE, it is shown on pytorch documentation. ...

1

Your forward function is not using the previous hidden state. observe: you pass hidden but never use it. def forward(self, input, hidden): layer = self.hiddenWx1(input) layer = self.hiddenWx2(layer) a_next = self.tanh(layer) z = self.z1(a_next) z = self.z2(z) y_next = self.softmax(z) return y_next,a_next

1

You want to compute the mean loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches! In your case: You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So ...

1

If your interest is positional information, encode it! This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need an example of a paper that encodes the 2D positional info: ...

1

I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and 9000th epoch's loss is greater about 0.2 alternate of 0.09 I also tried to change optimizer and learning rate but no loss didn't improve. you can explore the modified code may help ...

1

PyTorch now has a C++ frontend. I haven't tried it, but I'm sure you could use that. Another option, which is more production-tested, is using a message passing framework such as ZeroMQ to communicate requests and results between Python and C++ executables.

1

Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.

Only top voted, non community-wiki answers of a minimum length are eligible