Recent Questions - Artificial Intelligence Stack Exchange most recent 30 from ai.stackexchange.com 2022-07-05T02:47:09Z https://ai.stackexchange.com/feeds https://creativecommons.org/licenses/by-sa/4.0/rdf https://ai.stackexchange.com/q/36195 0 Why do we use gradient descent to minimize the loss function? Proton https://ai.stackexchange.com/users/56170 2022-07-05T02:41:04Z 2022-07-05T02:41:04Z <p>The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method.</p> <p>But in Calculus, if we want to find the global minimum of a multivariable function, we usually first calculate the partial derivatives of this function with respect to its variables, and then set these partial derivatives to zeros, and then find the solutions of these equations. Usually we get a bunch of points. We can use the second derivative method(involving Hesse Matrix) to determine whether these points give local minimum values, or we can directly evaluate the values of the function at these points and compare them to find the minimum.</p> <p>So, I'm curious why don't we use this classical method in Calculus to find the global minimum value of the loss function and instead using gradient descent? Is it hard to for computer to find zeros of multivariable equations?</p> https://ai.stackexchange.com/q/36194 0 Why and when do we use ReLU over tanh activation function? Struggling_In_Final https://ai.stackexchange.com/users/56156 2022-07-04T23:57:56Z 2022-07-04T23:57:56Z <p>I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how come ReLU is very popular in most NNs (which is even more odd when the authors didn't mention about ReLU at all)</p> https://ai.stackexchange.com/q/36192 0 How does GPT use the same embedding matrix for both input and output? SRobertJames https://ai.stackexchange.com/users/56167 2022-07-04T20:35:57Z 2022-07-04T20:46:08Z <p>My understanding is that GPT uses the <strong>same embedding matrix</strong> for both inputs and output: Let <span class="math-container">$V$</span> be the vocab size, <span class="math-container">$D$</span> the number of embedding dimensions, and <span class="math-container">$E$</span> be a <span class="math-container">$V \times D$</span> embedding matrix:</p> <ul> <li>On input, if <span class="math-container">$x$</span> is a one-hot <span class="math-container">$V$</span>-dimensional vector, GPT uses <span class="math-container">$Ei$</span>.</li> <li>On output, if <span class="math-container">$\hat y$</span> is a <span class="math-container">$D$</span>-dimensional prediction vector, GPT uses softmax(<span class="math-container">$E^\top{\hat y}$</span>) as its predictions.</li> </ul> <h2>Q1. Is the above correct?</h2> <p>I cannot find this stated clearly in the paper, but it is stated explicitly <a href="https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens" rel="nofollow noreferrer">here</a>. It's also clearly implied by the parameter count listed <a href="https://jalammar.github.io/illustrated-gpt2/" rel="nofollow noreferrer">here</a>, and argued for as best practice <a href="https://paperswithcode.com/method/weight-tying" rel="nofollow noreferrer">here</a>. Yet, for example, Karpathy's mini-GPT implementation seems to use two different matrices:</p> <pre><code>self.tok_emb = nn.Embedding(config.vocab_size, config.n_embd) # &lt;--- This would be E self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd)) self.drop = nn.Dropout(config.embd_pdrop) # transformer self.blocks = nn.Sequential(*[Block(config) for _ in range(config.n_layer)]) # decoder head self.ln_f = nn.LayerNorm(config.n_embd) self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False) # &lt;--- This has the same dimensions as Etranspose but is clearly a different matrix </code></pre> <h2>Q2. If it is correct, how does can it work?</h2> <p>This seems to be tasking <span class="math-container">$E$</span> with two very different, even opposing, functions:</p> <ul> <li>Map vocab to their <em>meaning</em> on the input side; higher magnitude indicates &quot;more meaning&quot;</li> <li>Map meaning to the <em>most likely</em> vocab on the output side; higher magnitude indicates greater likelihood</li> </ul> <p>When outputting, we want the softmax to be highest when the word is most likely; magnitude of the output matrix should be roughly proportional to how likely the word is two appear.</p> <p>Yet, when inputting, magnitude has <em>nothing to do with likelihood</em>. Magnitude on the input side captures some element of meaning: perhaps how extreme or intense the meaning is, perhaps another aspect (not necessarily easily interpreted).</p> https://ai.stackexchange.com/q/36191 1 How to calculate a meaningful distance between multidimensional tensors Hadar Sharvit https://ai.stackexchange.com/users/52355 2022-07-04T18:17:50Z 2022-07-04T22:36:28Z <p><strong>TLDR</strong>: given two tensors <span class="math-container">$t_1$</span> and <span class="math-container">$t_2$</span>, both with shape <span class="math-container">$(c,h,w)$</span> what metric should be use to measure their distance?</p> <hr /> <p><strong>More Info</strong>: I'm working on a project in which I'm trying to distinguish between an anomalous sample <span class="math-container">$A$</span> (specifically from <code>MNIST</code>) and a &quot;regular&quot; sample <span class="math-container">$R$</span>. The solution I chose is to consider the feature maps that are given by <code>ResNet</code> and use <em>kNN</em>. More specifically:</p> <ul> <li>I embed the entire <code>CIFAR10_TRAIN</code> data to achieve a dataset that consists of activations with dimension <span class="math-container">$(N,c,h,w)$</span> where <span class="math-container">$N$</span> is the size of <code>CIFAR_TRAIN</code></li> <li>For <span class="math-container">$2$</span> new test samples <span class="math-container">$t_C$</span> and <span class="math-container">$t_M$</span> from <code>CIFAR10_TEST</code> and <code>MNIST_TEST</code> respectively (both with shape <span class="math-container">$(c,h,w)$</span>) I embed them both to receive the activations (same as for the training data)</li> <li>(<strong>!</strong>) I find the <em>k-Nearest-Neighbours</em> of <span class="math-container">$t_C$</span> and <span class="math-container">$t_M$</span> w.r.t the embedding of the training data</li> <li>I calculate the mean distance between the <span class="math-container">$k$</span> neighbors</li> <li>Given some predefined threshold, I classify <span class="math-container">$t_C$</span> and <span class="math-container">$t_M$</span> as regular or anomalous, hoping that the distance for <span class="math-container">$t_M$</span> would be higher, as it represents O.O.D sample.</li> </ul> <p>Notice that in (<strong>!</strong>) I need some distance measure, but this is not trivial as these are tensors, not vectors.</p> <hr /> <p><strong>What I've Tried</strong>: a trivial solution is to flatten the tensor to have shape <span class="math-container">$(c\cdot h\cdot w)$</span> and then use basic <span class="math-container">$\ell_2$</span>, but the results turned out pretty bad. (could not distinguish regular vs anomalous in this case). Hence: <em>Is there a better way of measuring this distance</em>?</p> https://ai.stackexchange.com/q/36189 -1 Choices of activation functions Struggling_In_Final https://ai.stackexchange.com/users/56156 2022-07-04T15:35:56Z 2022-07-04T15:35:56Z <p>The authors of Efficient Backprop repeatedly stressed the importance of having symmetric sigmoid/activation function so that the outputs (inputs for the next layer) are averaged at 0. If that's the case, then why does ReLU a standard choice for most ANN?</p> <p><a href="https://i.stack.imgur.com/NPuPB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NPuPB.png" alt="enter image description here" /></a></p> <p>Bonus question: What did the authors mean when they said &quot;One of the potential problems with symmetric sigmoids is that the error surface can be very flat near origin&quot;?</p> https://ai.stackexchange.com/q/36187 0 How does covariance = 1 reorganise input pattern like in figure 3? Struggling_In_Final https://ai.stackexchange.com/users/56156 2022-07-04T14:02:48Z 2022-07-04T14:05:44Z <p>Page 8 - <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noreferrer">http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf</a></p> <p>TLDR contexts:</p> <p>(a) By setting covariance = 1, input patterns from different training sets reorganised shown in bottom left in the figure below</p> <p><a href="https://i.stack.imgur.com/OnQFc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OnQFc.png" alt="" /></a></p> <p>(b) Convergence is faster not only if inputs are shifted to average at 0 but also scaled to have the same covariance</p> <p><a href="https://i.stack.imgur.com/hruNj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hruNj.png" alt="" /></a></p> <p>(c) Covariance is sensitive to scaling but interpretation of covariance value isn't clear for that very reason. Overall, covariance only tells if multiple data has positive trend (covariance &gt; 0); or negative trend (covariance &lt; 0); or no trend (covariance = 0) - <a href="https://www.youtube.com/watch?v=qtaqvPAeEJY" rel="nofollow noreferrer">https://www.youtube.com/watch?v=qtaqvPAeEJY</a></p> <p>(d) The sigmoid function that authors referred to is this one on the right <a href="https://i.stack.imgur.com/Veg5E.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Veg5E.png" alt="" /></a></p> https://ai.stackexchange.com/q/36186 0 Exercise 3.21 Sutton Barto: Draw or describe the contours of the optimal action-value function for putting, $q_{*}(s, putter)$, for the golf example user https://ai.stackexchange.com/users/45492 2022-07-04T13:31:01Z 2022-07-04T14:56:26Z <p>I am doing exercise 3.21 in Sutton and Barto. Here's the exercise:</p> <blockquote> <p>Draw or describe the contours of the optimal action-value function for putting, <span class="math-container">$q_{*}(s, putter)$</span>, for the golf example.</p> </blockquote> <p>Here's the golf example:</p> <blockquote> <p>To formulate playing a hole of golf as a reinforcement learning task, we count a penalty (negative reward) of 1 for each stroke until we hit the ball into the hole. The state is the location of the ball. The value of a state is the negative of the number of strokes to the hole from that location. Our actions are how we aim and swing at the ball, of course, and which club we select. Let us take the former as given and consider just the choice of club, which we assume is either a putter or a driver. The upper part of Figure 3.3 shows a possible state-value function, <span class="math-container">$v_{putt}(s)$</span>, for the policy that always uses the putter. The terminal state in-the-hole has a value of 0. From anywhere on the green we assume we can make a putt; these states have value 1. Off the green we cannot reach the hole by putting, and the value is lower. If we can reach the green from a state by putting, then that state must have value one less than the green’s value, that is, 2. For simplicity, let us assume we can putt very precisely and deterministically, but with a limited range. This gives us the sharp contour line labeled -2 in the figure; all locations between that line and the green require exactly two strokes to complete the hole. Similarly, any location within putting range of the -2 contour line must have a value of -3, and so on to get all the contour lines shown in the figure. Putting doesn’t get us out of sand traps, so they have a value of <span class="math-container">$\infty$</span>. Overall, it takes us six strokes to get from the tee to the hole by putting. <a href="https://i.stack.imgur.com/og83J.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/og83J.png" alt="enter image description here" /></a></p> </blockquote> <p>Here's my solution:</p> <p>I will describe my solution: Suppose you are in the green zone. Then if you use the putter, you get it into the hole. So in the green it's <span class="math-container">$-1$</span>. Outside the green: If we are in the areas of <span class="math-container">$-3$</span> and <span class="math-container">$-2$</span> for <span class="math-container">$v_{putt}$</span>, then the values stay the same, because if we use the putter in the area of <span class="math-container">$-3$</span>, we get to <span class="math-container">$-2$</span> (my reasoning for this is because from the <span class="math-container">$v_{putt}$</span> contours, it is evident that using a putter will get you into the zone which is 1 step less) and we use either putter or driver from this point - doesn't matter. If we are in the area of <span class="math-container">$-2$</span> (for <span class="math-container">$v_{putt}$</span>), we use putter and get to green, following a hole after the next stroke. So that area is <span class="math-container">$-2$</span> for <span class="math-container">$q_{*}(s,putter)$</span>. Now, I assume that the area of <span class="math-container">$-4$</span>, <span class="math-container">$-5$</span>, and <span class="math-container">$-6$</span> of <span class="math-container">$v_{putt}$</span> coincides with the area of <span class="math-container">$-3$</span> for <span class="math-container">$q_*(s,driver)$</span>. So if we are in area <span class="math-container">$-4$</span> for <span class="math-container">$v_{putt}$</span>, we use putter one time and we get to the are of <span class="math-container">$-3$</span> for <span class="math-container">$v_{putt}$</span>, which we assume to be the area of <span class="math-container">$-2$</span> for <span class="math-container">$q_*(s,driver)$</span>. So this means that we can use driver and get to the hole in <span class="math-container">$2$</span> steps, so this means that in total we used <span class="math-container">$3$</span> steps, so <span class="math-container">$-4$</span> for <span class="math-container">$v_{putt}$</span> is <span class="math-container">$-3$</span> for <span class="math-container">$q_*(s, putter)$</span>. For the <span class="math-container">$-5$</span> are in <span class="math-container">$v_{putt}$</span>, we use a putter and then we are in the area of <span class="math-container">$-3$</span> for <span class="math-container">$q_*(s, driver)$</span>. This means we can get in <span class="math-container">$4$</span> steps. So the area of <span class="math-container">$-5$</span> for <span class="math-container">$v_{putt}$</span> is <span class="math-container">$-4$</span> for <span class="math-container">$q_*(s, putter)$</span>. The area of <span class="math-container">$-6$</span> for <span class="math-container">$v_{putt}$</span> is <span class="math-container">$-4$</span> for the same reason.</p> <p><strong>Question:</strong> Would this be correct reasoning?</p> https://ai.stackexchange.com/q/36184 0 What is a "continuous vector"? The Pointer https://ai.stackexchange.com/users/16521 2022-07-04T11:50:51Z 2022-07-04T12:08:47Z <p>I have seen the concept of a &quot;continuous vector&quot; described in the context of embeddings. For instance, <a href="https://datascience.stackexchange.com/a/54045/49134">this answer</a> to a question on embeddings in the context of deep learning. I obviously know what a vector is, but it isn't clear to me what a <em>continuous</em> vector is as a mathematical object. What is a &quot;continuous vector&quot;?</p> https://ai.stackexchange.com/q/36183 0 How to encode time embeddings in a diffusion model for 1D vectors? James Arten https://ai.stackexchange.com/users/47551 2022-07-04T10:42:26Z 2022-07-04T10:42:26Z <p>For a project I'm working on, I'd like to try a diffusion model like illustrated in the <a href="https://arxiv.org/pdf/2006.11239.pdf" rel="nofollow noreferrer">paper</a> by Ho et Al to generate 1D vectors. What I'm trying to figure out at the moment, is what kind of architecture should I use for the 1D case, given that those people use a U-net-based model including different layers like attention, residual, group norm, etc. The only requirement that such a model should satisfy is that <strong>the input and output shapes should be the same</strong>, as we're interested in predicting the <strong>noise</strong> associated with a particular timestep.</p> <p><span class="math-container">$$\mathcal{L}(\theta) := \mathbb{E}_{\textbf{x}_0, \, t, \, \varepsilon}\left[\|\varepsilon - \varepsilon_\theta(\sqrt{\bar{\alpha}_t}\textbf{x}_0 +\sqrt{1-\bar{\alpha}_t}\varepsilon,t)\|^2\right]$$</span></p> <p>So for now, I'm considering a simple <em>autoencoder</em> architecture but I'm not sure <strong>how and where</strong> I should inject knowledge of timesteps using some sort of positional encoder as proposed in the paper by Ho et Al.</p> <p>Additionally, I'm not even sure about they inject such knowledge into the model? Do they sum the obtained time embeddings over channels? Or over image dimensions?</p> https://ai.stackexchange.com/q/36182 0 What other Machine Learning techniques other than Neural Networks are there? akastack https://ai.stackexchange.com/users/56148 2022-07-04T09:22:03Z 2022-07-04T17:36:47Z <p>I know that there are three types of machine learning algorithms, supervised, unsupervised, and reinforcement learning, and that often neural networks are used to implement them. However, neural networks is said to be a subfield of machine learning, and deep learning a subfield of neural networks.</p> <p>Thus, I would like to know:</p> <p>Which specific algorithms/models belong in Machine Learning but are not considered neural networks? Any specific names for these fields?</p> https://ai.stackexchange.com/q/36181 0 Is the objective of VAE misaligned with the goal of generative modelling? Mike Land https://ai.stackexchange.com/users/42533 2022-07-04T07:24:19Z 2022-07-04T07:24:19Z <p>Recall that the maximised objective in the VAE is as follows: <span class="math-container">$$-L := \log P(X) − \mathop D (Q(z|X)∥P(z|X)) = \mathbb E_{z\sim Q(z \mid X)} \log P(X|z) − \mathop D (Q(z|X)∥P(z)) .$$</span> It is well-known the second term on the right rewrites, if <span class="math-container">$k$</span> is the dimension of the latent space, <span class="math-container">$P(z) = \mathcal N(z; 0, I)$</span>, and <span class="math-container">$Q(z \mid X) = \mathcal N(z; \mu(X), \Sigma(X))$</span> for some <span class="math-container">$\mu, \Sigma$</span>, into <span class="math-container">$$L_z := \mathop D(\mathcal N (\mu(X), \Sigma(X))∥ \mathcal N (0, I)) = \frac 1 2 \left ( \mathrm{tr} \, \Sigma(X) + \lVert \mu(X) \rVert^2 − k − \log \det \Sigma(X) \right ).$$</span> If we assume <span class="math-container">$P(X \mid z) = \mathcal N(X; f(z), \sigma^2 I)$</span> for some <span class="math-container">$f, \sigma$</span> (as is usually done), the first term on the right (the negation of which is usually termed the <em>reconstruction loss</em>) takes the form <span class="math-container">$$-L_r := \mathrm{const} − \frac 1 {2\sigma^2} \lVert X − f (z) \rVert ^2.$$</span> <span class="math-container">$\sigma$</span> is usually a fixed parameter.</p> <p>Consider the following two extremes: (1) if <span class="math-container">$\sigma \to 0$</span>, a model which acts as an identity on <span class="math-container">$X$</span> is optimal with respect to the loss <span class="math-container">$L \to L_r$</span>; (2) if <span class="math-container">$\sigma \to \infty$</span>, the model with an encoder that simply samples from the standard normal distribution <span class="math-container">$\mathcal N(0, I)$</span> and whatever decoder is optimal. In any case, the model with a encoder which maps all of <span class="math-container">$X$</span> into <span class="math-container">$\mathcal N(0, I)$</span> and a decoder which maps all values of the latents into the 'correct' value of <span class="math-container">$X$</span> is optimal with respect to the loss, but such a one cannot obviously exist unless the distribution of <span class="math-container">$X$</span> is degenerate.</p> <p>The original purpose of VAE, as far as I understand, was to enable mapping samples from <span class="math-container">$\mathcal N(0, I)$</span> into realistic values of <span class="math-container">$X$</span>. Now, this probably formalises as follows: given a sample <span class="math-container">$X_1, \dots, X_n$</span> of such values, we want the decoder to map them into such <span class="math-container">$z_1, \dots, z_n$</span> so that the distribution of this sample <span class="math-container">$z_\square$</span> 'approaches' <span class="math-container">$\mathcal N(0, I)$</span> in the limit <span class="math-container">$n \to \infty$</span> (with the distributions <span class="math-container">$P(z_i \mid X_i)$</span> not necessarily all being equal to <span class="math-container">$\mathcal N(0, I)$</span>). Then, sampling latents from <span class="math-container">$\mathcal N(0, I)$</span>, we will hopefully always get realistic values of <span class="math-container">$X$</span>. But, in contrast, a VAE model is taught something completely different, which is not quite aligned with that which we want to achieve in the end.</p> <p>Due to the age of VAEs, I can suppose that such ideas have probably been already explored. Are there any works that attempt to change the formulation in such a way as to mitigate the discrepancy which I have tried to describe above?</p> https://ai.stackexchange.com/q/36180 0 the best case time complexity for the minimax algorithm with alpha-beta pruning user288609 https://ai.stackexchange.com/users/56143 2022-07-04T04:10:35Z 2022-07-04T04:10:35Z <p>It is well-known that the node ordering plays an important factor in <code>minimax algorithm</code> \alpha-\beta pruning. Especially the worst case time complexity is O(b^m) while the best case time complexity is O(b^(m/2)). The worst case time complexity corresponds to the scenario that we have to traverse all the nodes in the game tree. The best case time complexity comes from the scenario <strong>where children of Max nodes are searched in greatest-value-first order, children of Min nodes are searched in least-value-first order</strong> But I am not clear how does the O(b^(m/2)) is achieved under this scenario.</p> https://ai.stackexchange.com/q/36179 0 Why can't I train like a dataset of samples instead of maintaining replay buffer? hanugm https://ai.stackexchange.com/users/18758 2022-07-04T00:12:36Z 2022-07-04T04:24:36Z <p>On observing the <a href="https://arxiv.org/pdf/1509.02971.pdf#page=5" rel="nofollow noreferrer">DDPG algorithm</a>, we notice that the updation of neural networks is happening <em>during</em> the episode.</p> <p>But, it seems there can be no issue if we allow the completion of an episode and then treat the <em>buffer as a dataset</em> to update the neural networks if I don't have any memory constraints. Isn't it?</p> <p>Even if the episode is too long, I think we can update neural networks once an episode terminates if I have enough memory. I don't think memory constraint is a reason for updating neural networks during episodes, as we can see large datasets in deep learning competitions and research. One possible reason I am thinking is that an episode <a href="https://ai.stackexchange.com/a/30087/18758">may not have a goal state</a> and can continue indefinitely.</p> <blockquote> <p><strong>In general MDPs do not have goal states</strong>. Although using the agent's actions to achieve certain desirable end states, such as winning a game or completing a puzzle, is a very common design, there is no requirement for this. The more general requirement is to maximise some aggregate of the reward at each time step - usually either a discounted sum of rewards or the mean reward.</p> </blockquote> <p>If 'having no goal state(s)' is a reason for using buffer, why do the authors claim it as an <em>algorithm</em> instead of a procedure? Am I missing something?</p> https://ai.stackexchange.com/q/36178 0 How to calculate similarity between the two graphs? Exploring https://ai.stackexchange.com/users/41187 2022-07-03T23:44:32Z 2022-07-03T23:44:32Z <p>I have to learn similarity between graphs using deep learning. I have many samples (500k) of graphs.</p> <p>How can I compute similarity score between two graphs? I am thinking:</p> <ul> <li>convert graphs into vectors using Graph2Vec embedding</li> <li>then compare them using various similarity calculating techniques like cosine similarity.</li> </ul> <p>But I am no deep learning expert and would love to know whether this is a proper way to learn graph embedding.</p> <p>I know there are different GNNs like GCN, GGNN or such. So I would really appreciate if I can get some feedback using Graph2Vec would be a good fit.</p> https://ai.stackexchange.com/q/36177 1 Why don't we use diffusion for non-graph CNNs? xuq01 https://ai.stackexchange.com/users/55890 2022-07-03T22:17:20Z 2022-07-03T22:17:20Z <p>I'm pretty new to graph neural networks, so please forgive me if this is a silly question.</p> <p>Diffusion is a method used to improve graph CNNs, however it seems to me that general CNNs can also benefit from taking into consideration diffusion-like processes (for example, in CV, one may want information to diffuse from a small neighborhood, then to a bigger neighborhood, etc.). Also, PDE-based methods have been used in CV traditionally (before deep learning became a thing), so there's at least some evidence that this might work. But I did search extensively, and I can't find any information on using diffusion for general CNNs!</p> <p>Therefore, what I'm curious about is: why didn't I find any information about this? Is it already explored under a different name? Or does it not really work because of some reason I don't know (or didn't think about)? Or maybe it does work, but there are some technical difficulties?</p> https://ai.stackexchange.com/q/36169 0 How to establish baselines, with different training loops Yash_Bit https://ai.stackexchange.com/users/56129 2022-07-03T16:50:07Z 2022-07-03T21:57:00Z <p>My objective is to test out a new algorithm that I designed. However, I am confused whether my methodology to train the networks is correct.</p> <p>I am just concerned about the training loops:</p> <p>In the first algorithm (DIAYN, SAC Based Algorithm), the pseudocode follows a high-level pseudocode of:</p> <pre><code>Run for N-STEPS: 1. Run for around 5000 steps 2. Add in Replay Buffer 3. After each step, after 5K, choose action using policy 4. Step in env, collect reward, next_obs .. 5. Update networks by sampling from replay buffer, batch size of 1025 </code></pre> <p>In the new algorithm, I update the same networks, but in a new manner (which is required for the algorithm to do some other stuff.</p> <pre><code>Run for n epochs: 1. Run and collect the 1000 samples of next_obs, reward .. by choosing the action from the policy. 2. Then, run some algorithm (this the new addition) to the replay buffer. 3. Run a training loop, which runs 1000 times, which updates the networks of a batch size of 128. </code></pre> <p>As you can see, in the first algo_1, the batch size is 1024, and we collect new samples after each update. Whereas, in algo_2 we update the network 1000 times with replay buffer samples, then we collect new samples.</p> <p>However, in algo_2, we collect 1000 new samples again. In algo_1, only one sample is added to the replay_buffer after each update. So one new data point is generated from a new updated policy. In algo_1, 1000 samples are generated using the new policy updated 1k times from old replay_buffer.</p> <p>My question is this, if I wanted to establish a baseline using algo_1, and say that my algo_2 is better as it does X better. Can I do so, if I make sure that the N-STEPS in algo_1 are equal to epochs*1k_training_loop in algo_2?</p> <p>I apologise for not making this post succinct.</p> https://ai.stackexchange.com/q/36167 -1 How to recreate fine-tuning experiment in Paper: DIAYN [closed] Yash_Bit https://ai.stackexchange.com/users/56129 2022-07-03T16:01:47Z 2022-07-03T22:00:49Z <p>This is my first post here, I apologise if there are any mistakes in how I have formatted the question.</p> <p>I am trying to recreate experiment from Figure 5. in the paper linked below (Image Provided). Pg # 6/22</p> <p>[Diversity is all You Need]<a href="https://i.stack.imgur.com/e2XTc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/e2XTc.png" alt="]" /></a>(<a href="https://arxiv.org/pdf/1802.06070.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1802.06070.pdf</a>)</p> <p>The reward is plotted on the y-axis, vs the time in the x. I am confused whether if this is the diversity reward as mentioned in the paper, or the env reward, which is received by env.step(action). It says, that it is, &quot;task-specific reward function.&quot;, I am not sure however.</p> https://ai.stackexchange.com/q/36166 0 Is the accuracy the best metrics to evaluate the performance of Deep Learning model? [closed] Dilip C M Dept of MCA https://ai.stackexchange.com/users/55943 2022-07-03T16:01:18Z 2022-07-04T02:55:07Z <p>Consider a model <code>A</code> that achieved an test accuracy of 99% on dataset-A with the size of 200 images and a model <code>B</code> that achieved only 50% on dataset-B with a size of 50,000 images. Also consider both the datasets split into train,validation and test sets in the ratio of 0.8,0.1,0.1.</p> <p>But on test data of dataset-B,the model A is failed to attain same accuracy in fact it is giving lesser accuracy that of model B.</p> <p>So, is the accuracy always the best measure to evaluate the performance of the DL model? Or any other better performance metrics available?</p> https://ai.stackexchange.com/q/36163 2 How to sample the tuples during the initial time steps of the DDPG algorithm? hanugm https://ai.stackexchange.com/users/18758 2022-07-03T12:38:29Z 2022-07-04T00:13:04Z <p>I am facing an issue in understanding the following line from the <a href="https://arxiv.org/pdf/1509.02971.pdf#page=5" rel="nofollow noreferrer">pseudocode of the DDPG algorithm</a></p> <blockquote> <p>Sample a random minibatch of <span class="math-container">$N$</span> transitions <span class="math-container">$(s_i, a_i, r_i, s_{i+1})$</span> from <span class="math-container">$R$</span></p> </blockquote> <p>Here <span class="math-container">$N$</span> is a hyperparameter that is equal to the number of transitions or samples that need to be present in the minibatch.</p> <p>Traditionally, we take a minibatch of samples from a dataset, which contains all samples, and pass them to a neural network, but if we observe the DDPG pseudocode, we are storing transition after transition into the buffer <span class="math-container">$R$</span>. So, I think, it needs several steps before sampling from the buffer <span class="math-container">$R$</span>. But, if we observe the pseudocode, it says that we need to sample from the first timestep of the first episode.</p> <p>How is it possible? Where am I missing?</p> https://ai.stackexchange.com/q/36162 1 Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'? hanugm https://ai.stackexchange.com/users/18758 2022-07-03T08:22:05Z 2022-07-04T00:13:17Z <p>In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch of unit transitions to train neural networks as it is known that neural networks work well for iid data.</p> <p>But in games, experience trajectory is important as it contains temporal dynamics. Am I true? If not, all the knowledge required to learn the policy function can be obtained from (out of sequence or randomly sampled) unit transitions alone.</p> <p>Which one among the both is correct?</p> <p>Note that unit transition in this question refers to <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span></p> https://ai.stackexchange.com/q/36160 1 Has there been an instance of an AI agent breaking out of its sandbox? 2080 https://ai.stackexchange.com/users/42922 2022-07-02T17:34:06Z 2022-07-04T22:23:29Z <p>There have been instances of agents using edge cases like bugs in <a href="https://www.youtube.com/watch?v=Lu56xVlZ40M" rel="nofollow noreferrer">physics engines</a>, <a href="https://www.gwern.net/Tanks#alternative-examples" rel="nofollow noreferrer">repetitive behavior in games</a> or word repetition in text prediction to cheat their reward function. However, these agents are arguably still contained, as while they explore the extremes of the state space of the simulation they don't expand their action space beyond what is possible in the simulation.</p> <p>The <a href="https://www.youtube.com/watch?v=p5T81yHkHtI" rel="nofollow noreferrer">Pokémon Yellow Total Control Hack</a> shows that in some systems, it is possible to gain full control of a computer by exploiting bugs in the hardware or software (here: memory corruption), enabling the agent to even <em>completely</em> reprogram the system 'from within', just using the normal inputs.</p> <p>Do you know of similar extreme examples where an AI agent went far beyond what was intended with the simulation environment?</p> https://ai.stackexchange.com/q/36117 0 How to interpret the output plan of the fast-downward planner Bilal https://ai.stackexchange.com/users/36869 2022-06-29T11:23:53Z 2022-07-04T07:05:51Z <p>I'm using this <a href="http://editor.planning.domains/#read_session=7yXhxKBnwe" rel="nofollow noreferrer">domain/problem</a> with the <a href="https://www.fast-downward.org" rel="nofollow noreferrer">fast-downward</a> planner like this:</p> <p><code>./fast-downward.py --plan-file plan.out ../test_domain.pddl ../test_problem.pddl </code></p> <p>The issue here is that the <code>output.sas</code> contains hundreds of thousands of lines without a clear reference to the solution plan!</p> <p>Here is the full output log of the planner:</p> <pre><code>\$ ./fast-downward.py --plan-file plan.out ../test_domain.pddl ../test_problem.pddl INFO planner time limit: None INFO planner memory limit: None INFO Running translator. INFO translator stdin: None INFO translator time limit: None INFO translator memory limit: None INFO translator command line string: /usr/bin/python3 /media/belal/WD//Planning/downward/builds/release/bin/translate/translate.py ../test_domain.pddl ../test_problem.pddl --sas-file output.sas Parsing... Parsing: [0.000s CPU, 0.001s wall-clock] Normalizing task... [0.000s CPU, 0.000s wall-clock] Instantiating... Generating Datalog program... [0.000s CPU, 0.000s wall-clock] Normalizing Datalog program... Normalizing Datalog program: [0.000s CPU, 0.001s wall-clock] Preparing model... [0.000s CPU, 0.000s wall-clock] Generated 9 rules. Computing model... [0.000s CPU, 0.000s wall-clock] 73 relevant atoms 28 auxiliary atoms 101 final queue length 157 total queue pushes Completing instantiation... [0.000s CPU, 0.001s wall-clock] Instantiating: [0.000s CPU, 0.003s wall-clock] Computing fact groups... Finding invariants... 5 initial candidates Finding invariants: [0.000s CPU, 0.001s wall-clock] Checking invariant weight... [0.000s CPU, 0.000s wall-clock] Instantiating groups... [0.000s CPU, 0.000s wall-clock] Collecting mutex groups... [0.000s CPU, 0.000s wall-clock] Choosing groups... 4 uncovered facts Choosing groups: [0.000s CPU, 0.000s wall-clock] Building translation key... [0.000s CPU, 0.000s wall-clock] Computing fact groups: [0.000s CPU, 0.001s wall-clock] Building STRIPS to SAS dictionary... [0.000s CPU, 0.000s wall-clock] Building dictionary for full mutex groups... [0.000s CPU, 0.000s wall-clock] Building mutex information... Building mutex information: [0.000s CPU, 0.000s wall-clock] Translating task... Processing axioms... Simplifying axioms... [0.000s CPU, 0.000s wall-clock] Translator axioms removed by simplifying: 0 Computing negative axioms... [0.000s CPU, 0.000s wall-clock] Processing axioms: [0.000s CPU, 0.000s wall-clock] Translating task: [0.000s CPU, 0.001s wall-clock] 24 effect conditions simplified 0 implied preconditions added Detecting unreachable propositions... 0 operators removed 0 axioms removed 2 propositions removed Detecting unreachable propositions: [0.000s CPU, 0.000s wall-clock] Reordering and filtering variables... 6 of 6 variables necessary. 4 of 6 mutex groups necessary. 24 of 24 operators necessary. 0 of 0 axiom rules necessary. Reordering and filtering variables: [0.000s CPU, 0.000s wall-clock] Translator variables: 6 Translator derived variables: 0 Translator facts: 16 Translator goal facts: 2 Translator mutex groups: 4 Translator total mutex groups size: 12 Translator operators: 24 Translator axioms: 0 Translator task size: 204 Translator peak memory: 31884 KB Writing output... [0.000s CPU, 0.000s wall-clock] Done! [0.000s CPU, 0.007s wall-clock] translate exit code: 0 INFO Running search (release). search needs --alias, --portfolio, or search options </code></pre> <p>Can you please tell me how can I interpret the generated plan?</p> https://ai.stackexchange.com/q/36019 0 Correct reward for trading applications. RL agent learns according to immediate reward instead of cumulative reward Leibniz https://ai.stackexchange.com/users/49718 2022-06-22T11:26:19Z 2022-07-04T11:30:09Z <p>I have coded a RL environment for trading. The action space is discrete with 3 components [0,1,2]; where 0 corresponds to selling an arbitrary amount of shares; 1 corresponds to holding; and 2 corresponds to buying. I am using the immediate profit, the amount of money made by selling or spent by buying, as reward. If the amount of shares is 0, selling has the same effect as holding. Training on historic data, I have realized that the agent ends up predicting selling at all times. If I try to penalize the agent for choosing selling when the number of shares is 0 then it gets stuck in holding. What could be a suitable reward for this application?</p> https://ai.stackexchange.com/q/35696 0 What is a "canonical space"? Trong-Thang Pham https://ai.stackexchange.com/users/26364 2022-05-29T11:51:46Z 2022-07-04T06:05:20Z <p>I am reading the paper on 3D reconstruction, <a href="https://proceedings.neurips.cc/paper/2021/file/a11f9e533f28593768ebf87075ab34f2-Paper.pdf" rel="nofollow noreferrer">ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction</a>, and I encountered the term &quot;canonical space&quot;.</p> <p>What is a &quot;canonical space&quot;? Is it widely used? Is there other use of the term?</p> <p>In my understanding, a canonical space is a standard, norm space. And we want to convert sample points into that space so that we can perform standard procedures. But I don’t know if that interpretation is correct or not. And I am also interest if it is a common term or very few people use it.</p> https://ai.stackexchange.com/q/34403 0 time series analysis: predict number and type of service Alfonso https://ai.stackexchange.com/users/20780 2022-02-03T14:37:38Z 2022-07-04T21:04:15Z <p>I have temporal data regarding the number of customers who requested a specific service in a given period (month and year). Below is a small excerpt from the dataset:</p> <p><img src="https://i.stack.imgur.com/PpWqq.png" alt="1" /></p> <ul> <li>Month-year: month and year when the service has been requested/offered</li> <li>Service Description: the tipology of the service request by the customer</li> <li>occurences: how many times the customers in that period requested that service</li> </ul> <p>I have monthly data from 2003 to 2020 and I would like to carry out a predictive analysis to predict the number of events from 2021 to 2023 and also predict the type of services. For the first I know that I have to face the problem using the analysis of the time series, I have doubts about the second part ... how to predict the type of service in addition to the number of requests? Can you give me some suggestions?</p> https://ai.stackexchange.com/q/28173 0 How to reject boxes inside each other with Non Max Suppression A Tyshka https://ai.stackexchange.com/users/47814 2021-06-09T22:30:28Z 2022-07-05T02:04:10Z <p>I’m working on an object detection cnn, and having some issues with non max suppression. When I have a small box inside a large box, NMS is not rejecting the smaller, incorrect box, because its IOU is small (large union, small intersection). How is this scenario typically dealt with? When using out of the box pretrained models for object detection I don’t seem to get boxes completely inside other boxes. Example here: <a href="https://i.stack.imgur.com/BiOB4.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BiOB4.jpg" alt="enter image description here" /></a> green is ground truth, blue is prediction. The center box has a tiny blue box inside that’s not getting rejected by NMS</p> https://ai.stackexchange.com/q/28149 0 Can you use a graph as input for a neural network? CatSpaghetti https://ai.stackexchange.com/users/47747 2021-06-08T14:36:52Z 2022-07-04T13:07:30Z <p>We want to try and distinguish real voices from (deep)fake voices using the graphs generated by a discrete fourier transform (generated from .wav audio files). We know from each image if it is a real or a fake voice, so it's a supervised classification problem. An image would look like this:</p> <p><a href="https://i.stack.imgur.com/vbqjJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vbqjJ.png" alt="enter image description here" /></a></p> <p>We think that real voices generate a graph with clear spikes, whereas fake voices have more noise resulting in less clear spikes. For this reason, we thought of using a CNN to take such an image as input (with x and y-axes ommited), and classify it as real or fake. Our concern is that it's actually a graph and not an image of an object, so we're not sure if this would be a good approach. We could also use the arrays generated from the fourier transform, but we're not sure how we could use that as input as we want to classify if it's real or fake, and not predict y for each x.</p> https://ai.stackexchange.com/q/28126 1 Selecting features for a neural network: is it redundant to have a feature that is an average (or max, or min) of some other features Arthur Song https://ai.stackexchange.com/users/47715 2021-06-07T03:46:03Z 2022-07-03T23:03:45Z <p>I'm trying to create a neural network that would able to look at the current price of a crypto asset and classify between a &quot;BUY&quot;, &quot;SELL&quot; or &quot;HOLD&quot;. So far for my input features, I've decided to go with the past 40 opens, closes, highs, lows, turnover, and volumes (240 features + the current price so 241 total features).</p> <p>Would it be redundant/not ideal if I had another feature that was the average of the past 40 opens for example? What about the max/min of the past opens?</p> <p>My thinking was that with only the raw prices data of the past 40 days, the neural network would be able to &quot;detect&quot; and create the most optimum features like the average or max in the hidden layers. And therefore, having the avg. or the max/min of some existing features would be unnecessary or perhaps worsen the performance of the model.</p> <p>Or is there no clear answer and would this be something I'd only be able to figure out by testing against data?</p> <p>Thanks for your help!!</p> https://ai.stackexchange.com/q/15463 0 How could I compute in real-time the similarity between tickets? Alfonso https://ai.stackexchange.com/users/20780 2019-09-16T12:29:45Z 2022-07-04T12:07:25Z <p>I'm dealing with a &quot;ticket similarity task&quot;.</p> <p>Every time new tickets arrive at the help desk (customer service), I need to compare them and find out about similar ones.</p> <p>In this way, once the operator responds to a ticket, at the same time he can solve the others similar to the one solved.</p> <p>I expect an input ticket and all the other tickets with their similarity in output.</p> <p>I thought about using <strong>DOC2VEC</strong>, but it requires training every time a new ticket enters.</p> <p>What do you recommend?</p> https://ai.stackexchange.com/q/2810 4 Using Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment Tayyebi https://ai.stackexchange.com/users/5398 2017-02-11T11:22:59Z 2022-07-04T10:51:30Z <p>I'm here to ask you for a solution on this problem which is: how to use <em>Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment</em>.</p> <blockquote> <p>Reinforcement Learning is a sub-area of Machine Learning in which an active entity called an agent interacts with its environment and learns how to act in order to achieve a pre-determined goal. The Reinforcement Learning had no prior model of behaviour and the participants no prior knowledge that their task was to move to and stay in a specific place. The participants were placed in a virtual environment where they had to avoid collisions with virtual projectiles. Following each projectile the agent analysed the movement made by the participant to determine paths of future projectiles in order to increase the chance of driving participants to the goal position and make them stay there as long as possible.</p> </blockquote> <p>The purpose of this question is to find a direct answer from the community with help of a paper which is already published on <a href="https://www.sciencedirect.com/science/article/abs/pii/S1071581916301513" rel="nofollow noreferrer">science direct</a> and the text above is exactly quoted from that source (<a href="http://discovery.ucl.ac.uk/1539195/1/Slater_elsarticle-template-harv.pdf" rel="nofollow noreferrer">PDF version</a>).</p> <p>How can we approach solving this problem?</p>