2

Neural networks are not invariant to translations, but equivariant, Invariance vs Equivariance Suppose we have input $x$ and the output $y=f(x)$ of some map between spaces $X$ and $Y$. We apply transformation $T$ in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of ...


2

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. More recent papers used 512, 768, 1024. One of the factors, influencing the choice of embedding is the way you would like different vectors to correlate with each ...


2

You're trying to learn a cubic function that explodes in values and your issue is scaling. I have been able to learn a better approximation by scaling data and using tanh as activation function. Code and result are as below: Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result ...


2

I'd like to add more information to Neal's answer. A network implemented in the example does not include activation for the output, because it will be applied during the training: # ... def training_step(self, batch): images, labels = batch out = self(images) # apply activation and calculate loss loss = F.cross_entropy(out, labels) ...


2

A neural network layer with no activation function is the same as "linear activation" i.e. $f(x) = x$ This is often used for the output layer in regression problems, where a constrained output like that of sigmoid, hyperbolic tangent or ReLU may not be appropriate. For an output layer, this is fine, and does not conflict with any theory behind ...


1

In machine learning, a tensor is a multi-dimensional array (i.e. a generalization of a matrix to more than 2 dimensions), which has some properties, such as the number of dimensions or the shape, and to which you can apply operations (for example, you can take the mean of all elements across all dimensions). So, a scalar is a 0-d tensor (no dimensions), a ...


1

Imagine the tensor as a some generalized $n$-dimensional hyperrectangle sliced into $n$-dimensional hypercubes. Each element of the tensor is labeled by the position along the given axis, say $(x_1, x_2, \ldots)$. Axis is not a property of tensor, rather the tensor is embedded in a $n$-dimensional space, where the axes are chosen along the sides of the ...


1

I think this is best explained using an analogy. Also you seen to have the misconception that you don't tune hyper-parameters for training data. You want to increase the accuracy of the training set AND validation set at the same time, but the validation set is more important so you want to maximise that accuracy more. Imagine you had a toddler, and you were ...


1

Summary: the loss needs to be differentiable, with some caveats. I will introduce some notation, which I hope is clear: if not I am happy to clarify. Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_\mathrm{min}$ which ...


1

RNNs are known to be superior to MLP in case of sequential data, like yours. But complex models like LSTM and GRU require a lot of data to achieve their potential. I don't know about your data but you can try to validate your architecture, approach and overall setting using a different, known time-series benchmark data. Maybe something is wrong with ...


1

I often check one of these: https://www.paperswithcode.com/ http://www.arxiv-sanity.com/ https://www.youtube.com/c/YannicKilcher/videos https://www.reddit.com/r/MachineLearning/ And, of course, Twitter :)


1

I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. If we’re in a hurry, one rule of thumb is to use the fourth root of the total number of unique categorical elements while another is that the embedding dimension should be approximately 1.6 times the square root of ...


Only top voted, non community-wiki answers of a minimum length are eligible