New answers tagged

1

In machine learning, a tensor is a multi-dimensional array (i.e. a generalization of a matrix to more than 2 dimensions), which has some properties, such as the number of dimensions or the shape, and to which you can apply operations (for example, you can take the mean of all elements across all dimensions). So, a scalar is a 0-d tensor (no dimensions), a ...


1

Your interpretation is definitely correct. As you correctly pointed out, the derivative of softplus is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for ReLU. What is quite interesting here is why softplus can be called an approximation to ReLU. If we break down the definition of softplus, we note that the ...


0

Channels can be thought of as alternate numbers in the same space. As an example, the three colour channels of a typical image are often values for amount of red, green or blue light received from each position within the picture. Your 1D convolution example has one input channel and one output channel. Depending on what the input represents, you might have ...


6

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016): You can see that these functions are linear functions for $x > 0$, that's why they are called Linear Units. For example, the ELU is defined as $$ ELU(x) = \begin{cases} x &\text{if } x > 0\\ \alpha (\exp(x)-1) & \text{if } x \leq 0. \end{cases} $...


1

Imagine the tensor as a some generalized $n$-dimensional hyperrectangle sliced into $n$-dimensional hypercubes. Each element of the tensor is labeled by the position along the given axis, say $(x_1, x_2, \ldots)$. Axis is not a property of tensor, rather the tensor is embedded in a $n$-dimensional space, where the axes are chosen along the sides of the ...


2

With this link I could read the paper. Thanks. So there is this discipline called sensor fusion. It is very sounded in the field of Autonomous Vehicles where in order to take one decision (whether to break or not) you have to take into account information for multiple sources: car mounted cameras, LIDAR, ultrasound, radar... So the term "fusion" ...


0

Also, keep in mind that not just any augmentation of the loss function is a regularization. For example, you can add terms to a loss function that enforce constraints on the solution but do not prevent overfitting nor facilitate generalization.


3

Regularization is not limited to methods like L1/L2 regularization which are specific versions of what you showed. Regularization is any technique that would prevent network from overfitting and help network to be more generalizable to unseen data. Some other techniques are Dropout, Early Stopping, Data Augmentation, limiting the capacity of network by ...


1

Updating model for each training example means batch size of 1, aka stochastic gradient descent(SGD). 1 iteration is defined as forward propagate, calculate loss, backpropagate and finally update weights. Since batch size is 1, running 5 epochs on 100 training examples with SGD means you will do 500 iterations, yes.


4

XPU is a device abstraction for Intel heterogeneous computation architectures, which can be mapped to CPU, GPU, FPGA and other accelerators. The "X" from XPU is just like a variable, like in maths, so you can do X=C and you get CPU accceleration, or X=G and you get GPU acceleration... That's the intuition behind that abstract name. In order to ...


2

Yes, it is a bit misleading. What it really means is input channels, so it would be: nn.Conv2d: Applies a 2D convolution over an input signal composed of several input channels. So, why don't just use channels instead of input planes? Well, initially the major deep learning applications were used for computer vision or image processing approaches. In CV or ...


2

The fact is you can always express an affine transformation as a linear transformation (more convenient because it is just a matrix/dot product). For instance, given an input $\textbf{x}=[x_1, ..., x_n]$, some weights $\textbf{a} = [a_1, a_2, ..., a_n]$ and a bias $b \in \mathbb{R}$, you can express the affine operation $y = \textbf{a}\cdot \textbf{x} + b$ ...


2

In linear algebra, a linear transformation (aka linear map or linear transform) $f: \mathcal{V} \rightarrow \mathcal{W}$ is a function that satisfies the following two conditions $f(u + v)=f(u)+f(v)$ (additivity) $f(\alpha u) = \alpha f(u)$ (scalar multiplication), where $u$ and $v$ vectors (i.e. elements of a vector space, which can also be $\mathbb{R}$ [...


0

Without the specific context, I cannot give a definitive answer, but it's very likely that a "differentiable architecture" refers to a neural network that represents/computes a differentiable function (so you need to use differentiable activation functions, such as the sigmoid), i.e. you can take the partial derivatives of the loss function with ...


3

A smooth function is usually defined to be a function that is $n$-times continuously differentiable, which means that $f$, $f'$, $\dots$, $f^{(n - 1)}$ are all differentiable and $f^{(n)}$ is continuous. Such functions are also called $C^n$ functions. It can be a bit of a vague term; some people might even stretch the definition and say any continuous ...


0

We need to compute the gradients in-order to train the deep neural networks. Deep neural network consists of many layers. Weight parameters are present between the layers. Since we need to compute the gradients of loss function for each weight, we use an algorithm called backprop. It is an abbreviation for backpropagation, which is also called as error ...


3

Essentially, any data you use to train or develop the model shouldn't be used as test data. In principle, "unseen" data gives a good estimate for the generalisation performance of the model; but this is only valid if the data really is unseen and hasn't been used in the model development process. If you've been tuning a model to increase its ...


0

I would like to add "The Master Algorithm" by Pedro Domingos. I would say it's more philosophical but still provides high level discussions about differences between algorithms. He also has a sense of humor which makes it a lighter read.


0

The famous book Artificial Intelligence: A Modern Approach (by Stuart Russell and Peter Norvig) covers all or most of the theoretical aspects of artificial intelligence (such as deep learning) and it also dedicates one chapter to the common philosophical topics that you mention.


1

I think that these terms may be used inconsistently across sources. If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the ...


1

I know at least one example where the rank of the dataset (more specifically, the rank of a matrix that is computed from the design matrix, i.e. the matrix with your data, which I will describe more in detail below) can have an impact on the number of solutions that you can have or how you find those solutions. I am thinking of linear regression. So, in ...


3

First of all, I don't know of any textbook that clarifies these terms, but, although I am not a statistician, in addition to the other answer, one possible way to look at it is as follows. You use probability theory to model your problem. For example, if it's a classification problem, you could define the conditional probability distribution $p(y \mid x)$, ...


Top 50 recent answers are included