Questions tagged [math]
For questions about mathematics related to artificial intelligence.
215
questions
1
vote
0
answers
37
views
Does all GAN's in literature need to satisfy the properties of objective function of initial GAN? [closed]
Consider the following value function of the initial GAN
$V(D, G) = \mathbb{E}_{x \sim p_{data(x)}} [\log D(x)] + \mathbb{E}_{z \sim p_z(z)} [1- \log D(G(z))]$
The min-max game on the value function: $...
- 3,059
0
votes
1
answer
20
views
What is the right way to find the alphas in this equation?
In the Grad-CAM++ paper the following equation (7) is posed (written here without the relu function):
$$
Y^c =
\sum_k \Bigl( \Bigl\{ \sum_{a,b} \alpha_{ab}^{kc}
\cdot \frac{\partial Y^c}{\...
- 131
0
votes
1
answer
26
views
Usability of power series in AI analysis
In mathematics, power series is given by
$$f(x) = \sum\limits_{n=0}^{\infty} c_n (x-a)^n$$
where $c_n , a \in \mathbb{R}$
Although most of the courses in academics cover moment generating functions in ...
- 3,059
0
votes
1
answer
51
views
What is the correct partial derivative of $Y^c$ with respect to $A_{ij}^{kc}$?
I have a question about the Grad-CAM++ paper. I do not understand how the following equation (10) for the alphas is obtained:
$$
\alpha_{ij}^{kc} =
\frac{\frac{\partial^2 Y^c}{(\partial A_{ij}^k)^2}}
{...
- 131
0
votes
0
answers
12
views
How do I compute the convolution of two kernels of the same size in practice?
Suppose I have a 256-by-256 input matrix called $X$ and two 3-by-3 kernels called $K_1$ and $K_2$. By the associativity of convolution
\begin{equation}
(X \star K_1) \star K_2 = X \star (K_1 \star K_2)...
- 131
1
vote
1
answer
39
views
Should I need to interpret the word "metric" in "performance metric" rigorously?
Consider the following abstract from the research paper titled A Note on the Inception Score for instance
Deep generative models are powerful tools that have produced
impressive results in recent ...
- 3,059
0
votes
1
answer
77
views
What does it mean by "gradient flow" in the context of neural networks?
Several research papers and textbooks (e.g. this) contain the phrase "gradient flow" in the context of neural networks.
I am confused about whether it has any rigorous and formal way of ...
- 3,059
0
votes
1
answer
35
views
What does it mean by "dynamics of a sequence" mathematically?
Consider the following paragraph from the topic named sequential models from the textbook titled Dive into Deep Learning
Both cases raise the obvious question of how to generate training
data. One ...
- 3,059
1
vote
1
answer
75
views
What do the square brackets $[ ]$ and $\mid$ mean in $[G_t \mid S_t=s]$?
Here is the formula of state-value function in Reinforcement Learning.
What do the square brackets $[ ]$ and $\mid$ mean in $[G_t \mid S_t=s]$? Why use square brackets? Why use $\mid$?
Why do ...
- 113
0
votes
0
answers
29
views
Can we compute probability of sample in R1-Natural Evolution Strategy with linear time and space?
In R1-NES, the sample is drawn from a multivariate normal distribution, $\vec{\theta} \sim \mathcal{N}\left( \vec{\mu},\mathbf{\Sigma} \right)$, with the covariance matrix is parametrized by $s$ and a ...
- 143
0
votes
0
answers
53
views
How to prove that "w will converge to TD fixed point once A is positive definite"
In Reinforcement Learning: An Introduction 2nd edition section 9.4 (p. 206), it says that when we use TD(0) as target and use semi-gradient method to update :
In general, $w_t$ will be reduced ...
- 1
0
votes
0
answers
14
views
Nearest-Neighbour recommendation distance ranking where some recommendations occur more than once
A nearest neighbour recommendation engine may, given a query vector identify the ranked top-K nearest vectors within a dataset using a distance measure such as cosine similarity. In my problem domain, ...
0
votes
1
answer
72
views
How to derive the dual function step by step in relative entropy policy search (REPS)?
TL:DR, (Why) is one of the terms in the expectation not derived properly?
Relative entropy policy search or REPS is used to optimize a policy in an MDP. The update step is limited in the policy space (...
- 143
0
votes
1
answer
160
views
What is the definition of a trace of a tensor?
Tensor is a multi-dimensional ordered collection of elements, which is used in deep learning to store and process data as well as intermediate steps.
We are aware of the trace of a two-dimensional ...
- 3,059
2
votes
1
answer
25
views
Are the domains of objective functions in AI always equals to $\mathbb{R}^D$ or subset of it?
Consider the following paragraph from the chapter named Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.
Central to this chapter is the ...
- 3,059
1
vote
1
answer
52
views
What is the analytical formula for "Kaiming He" probability density function?
A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable.
For example, the probability density function of a ...
- 3,059
0
votes
1
answer
135
views
In this paper, if region $R_{2}$ moves in a sliding window manner, won't the saliency map have a smaller size than the original image?
In the paper Salient Region Detection and Segmentation, I have a question pertaining to section 3 on the convolution-like operation being performed. I had already asked a few questions about the paper ...
user48670
1
vote
1
answer
70
views
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.
The chain rule of ...
- 3,059
0
votes
0
answers
71
views
What's wrong with my understanding of how RNNs work?
Recently, I've been trying to derive the mathematics behind various Neural Network structures. I managed to derive the MLP and tested it to be on par with a Keras implementation (Using the MNIST ...
0
votes
0
answers
26
views
What are the different types of geometry in literature that may be used for deep learning?
Recently, I asked a question on the concept of a manifold and received an answer that points to a relatively new subfield of deep learning named geometric deep learning.
In the ...
- 3,059
0
votes
0
answers
39
views
What are mathematically the factors of variation in deep learning?
The following paragraph from an answer tells us about factors of variation
Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
- 3,059
2
votes
0
answers
71
views
REINFORCE differentiation on sum or single value?
I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
- 49
0
votes
0
answers
31
views
What should the value of $ρ$ in the $w(n+1) = w(n) + \rho*\text{error}(i)x(i)$ formula of Least Mean Squares be?
I am trying to better understand the Least Mean Squares algorithm, in order to implement it programmatically.
If we consider its weight updating formula $$w(n+1) = w(n) + \rho * \text{error}(i)x(i),$$ ...
- 1
0
votes
1
answer
35
views
Isssue in understanding the derivation regarding mean squared error
The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)
I am facing difficulty in understanding the zero derivative ...
- 3,059
0
votes
1
answer
37
views
How to understand slope of a (non-convex) function at a point in domain?
Consider the following paragraph from Numerical Computation of deep learning book that says derivative as a slope of the function curve at a point
Suppose we have a function $y= f(x)$, where both $x$ ...
- 3,059
0
votes
1
answer
101
views
What are the Calculus books recommended for beginner to advanced researchers in artificial intelligence?
Calculus is a branch of mathematics that primarily deals with the rate of change of outputs of a function w.r.t the inputs.
It contains several concepts including limits, first-order derivatives, ...
- 3,059
1
vote
1
answer
75
views
What does it mean "having Lipschitz continuous derivatives"?
We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in Numerical Computation of the deep learning book.
In the context of deep ...
- 3,059
1
vote
0
answers
37
views
Is it true that real world data is highly discontinuous?
A function $f$ is said to be continuous at a point $c$ if it satisfies three properties:
Should be defined at the point $c$
Left and right-hand limits at $c$ must be equal i.e., the limit must exist
...
- 3,059
0
votes
0
answers
16
views
Predict a part of the input based of the output
I'm working on a fun project where I have a dataset of input and output data, both having a fixed size of characters.
I would like to predict a part of the input based on a known output as follows:
$$...
- 1
0
votes
2
answers
36
views
Reason for relaxing limit in derivative in this context?
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..
Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function ...
- 3,059
3
votes
3
answers
88
views
What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?
In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by $e$, called as Euler's number or Euler's constant or Napier's constant. The ...
- 3,059
1
vote
2
answers
24
views
Why not undefined expression is different from numerical underflow?
Consider an architecture or programming language that uses $n$ bits for storing a floating point number in a particular format. Then each and every floating point number it can store should be in a ...
- 3,059
2
votes
3
answers
108
views
Is there any domain in machine learning that solves a problem by using only analytical algorithms?
Most of the algorithms in machine learning I am aware of use datasets and learning happens in an iterative manner given some examples. The examples can also be understood as experience in the case of ...
- 3,059
1
vote
0
answers
20
views
Is there any concept like 'applying affine transformation on multiple inputs'?
Affine transformation on $X$ is a transformation of the following form
$$Y = wX + b$$
In general, $w, X, Y$ and $b$ tensors.
We generally call tensor $X$ as an input to affine transformation or the ...
- 3,059
0
votes
1
answer
37
views
Is the range of inception score flexible or bounded based on number of classes?
Inception score is used to evaluate the generative models. It is a score given based on quality and diversity of images generated.
I have doubt about the range of inception score because of the reason ...
- 3,059
0
votes
0
answers
33
views
Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?
There are instances in literature where we need to change loss function in order to escape from gradient problems.
Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
- 3,059
0
votes
0
answers
15
views
A 2D distance measure that gives weightage to the angle between the two points?
n1 n8 n7
n2 c n6
n3 n4 n5
Assume that all neighbors ni are (Euclidean) ...
- 21
1
vote
1
answer
56
views
What are the iid random variables for a dataset in the GAN framework?
I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.
The answer tells that it is due to the law of large numbers which is based on the assumption ...
- 3,059
0
votes
1
answer
148
views
How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
The research paper titled Improved Training of Wasserstein GANs proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator.
We now propose an ...
- 3,059
3
votes
2
answers
1k
views
What is Lipschitz constraint and why it is enforced on discriminator?
The following is the abstract for the research paper titled Improved Training of Wasserstein GANs
Generative Adversarial Networks (GANs) are powerful generative models,
but suffer from training ...
- 3,059
0
votes
2
answers
317
views
What is meant by an axis of a tensor?
Tensor is an ordered collection of elements. The elements are generally real numbers. Tensors are used in deep learning for storing data.
There is a wide usage of the word "axis" related to ...
- 3,059
3
votes
1
answer
103
views
What are the necessary mathematical properties to be a loss function in gradient based optimizations?
Loss functions are used in training neural networks.
I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.
I ...
- 3,059
2
votes
0
answers
26
views
Is there any closed form analytical expression to represent fractional max pooling?
There are Nineteen types of pooling layers in PyTorch.
Almost all of the layers are provided with corresponding analytical formulae. But analytical formulae are not provided for the fractional max-...
- 3,059
1
vote
1
answer
86
views
Questions about a research paper on salient region detection and segmentation [closed]
I am reading this paper in an attempt to recreate the salient region detection and segmentation model employed. I have the following questions pertaining to section 3 of the paper and I would highly ...
user48670
1
vote
0
answers
78
views
How can I compute a mathematical formula for my CNN?
Let's say, for example, I have built the following CNN model using Keras:
...
user48670
1
vote
1
answer
51
views
How to interpret the policy gradient expression in reinforcement learning?
I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
2
votes
1
answer
328
views
Why is tanh a "smoothly" differentiable function?
The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
- 3,059
1
vote
1
answer
100
views
What math should I learn before and while using and applying deep learning?
I want to learn deep learning. After researching a little, I came to the conclusion that I need a lot of math. I've started a linear algebra course, and it takes a long time (2-3 weeks). I want to ...
- 11
1
vote
1
answer
214
views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
- 3,059
0
votes
0
answers
23
views
Derivation of an probability expansion used in Word2Vec classifier model
We are using the following notations, for this question, to calculate the probability values
\begin{array}{|c|c|} \hline
\text{$w$} & \text{target word embedding vector} \\ \hline
\text{$c$} &...
- 3,059