Questions tagged [math]
For questions about mathematics related to artificial intelligence.
228
questions
1
vote
1answer
17 views
Are the domains of objective functions in AI always equals to $\mathbb{R}^D$ or subset of it?
Consider the following paragraph from the chapter named Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.
Central to this chapter is the ...
2
votes
1answer
42 views
What is the analytical formula for "Kaiming He" probability density function?
A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable.
For example, the probability density function of a ...
1
vote
1answer
33 views
Educational Resources and Programming Languages for AI & ML [closed]
I am a Mathematics graduate who is interested in AI and Machine Learning. I would like to dig deep into the maths of them but I do not know where to start. It seems Linear Algebra is the most crucial ...
1
vote
1answer
100 views
In this paper, if region $R_{2}$ moves in a sliding window manner, won't the saliency map have a smaller size than the original image?
In the paper Salient Region Detection and Segmentation, I have a question pertaining to section 3 on the convolution-like operation being performed. I had already asked a few questions about the paper ...
1
vote
1answer
37 views
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.
The chain rule of ...
0
votes
0answers
69 views
What's wrong with my understanding of how RNNs work?
Recently, I've been trying to derive the mathematics behind various Neural Network structures. I managed to derive the MLP and tested it to be on par with a Keras implementation (Using the MNIST ...
0
votes
0answers
24 views
What are the different types of geometry in literature that may be used for deep learning?
Recently, I asked a question on the concept of a manifold and received an answer that points to a relatively new subfield of deep learning named geometric deep learning.
In the ...
0
votes
0answers
36 views
What are mathematically the factors of variation in deep learning?
The following paragraph from an answer tells us about factors of variation
Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
2
votes
0answers
68 views
REINFORCE differentiation on sum or single value?
I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
0
votes
0answers
30 views
What should the value of $ρ$ in the $w(n+1) = w(n) + \rho*\text{error}(i)x(i)$ formula of Least Mean Squares be?
I am trying to better understand the Least Mean Squares algorithm, in order to implement it programmatically.
If we consider its weight updating formula $$w(n+1) = w(n) + \rho * \text{error}(i)x(i),$$ ...
0
votes
1answer
26 views
Isssue in understanding the derivation regarding mean squared error
The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)
I am facing difficulty in understanding the zero derivative ...
0
votes
1answer
28 views
How to understand slope of a (non-convex) function at a point in domain?
Consider the following paragraph from Numerical Computation of deep learning book that says derivative as a slope of the function curve at a point
Suppose we have a function $y= f(x)$, where both $x$ ...
1
vote
1answer
54 views
What are the Calculus books recommended for beginner to advanced researchers in artificial intelligence?
Calculus is a branch of mathematics that primarily deals with the rate of change of outputs of a function w.r.t the inputs.
It contains several concepts including limits, first-order derivatives, ...
0
votes
1answer
43 views
What does it mean "having Lipschitz continuous derivatives"?
We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in Numerical Computation of the deep learning book.
In the context of deep ...
1
vote
0answers
33 views
Is it true that real world data is highly discontinuous?
A function $f$ is said to be continuous at a point $c$ if it satisfies three properties:
Should be defined at the point $c$
Left and right-hand limits at $c$ must be equal i.e., the limit must exist
...
0
votes
0answers
14 views
Predict a part of the input based of the output
I'm working on a fun project where I have a dataset of input and output data, both having a fixed size of characters.
I would like to predict a part of the input based on a known output as follows:
$$...
0
votes
2answers
33 views
Reason for relaxing limit in derivative in this context?
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..
Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function ...
1
vote
2answers
24 views
Why not undefined expression is different from numerical underflow?
Consider an architecture or programming language that uses $n$ bits for storing a floating point number in a particular format. Then each and every floating point number it can store should be in a ...
2
votes
3answers
102 views
Is there any domain in machine learning that solves a problem by using only analytical algorithms?
Most of the algorithms in machine learning I am aware of use datasets and learning happens in an iterative manner given some examples. The examples can also be understood as experience in the case of ...
1
vote
0answers
17 views
Is there any concept like 'applying affine transformation on multiple inputs'?
Affine transformation on $X$ is a transformation of the following form
$$Y = wX + b$$
In general, $w, X, Y$ and $b$ tensors.
We generally call tensor $X$ as an input to affine transformation or the ...
0
votes
1answer
25 views
Is the range of inception score flexible or bounded based on number of classes?
Inception score is used to evaluate the generative models. It is a score given based on quality and diversity of images generated.
I have doubt about the range of inception score because of the reason ...
0
votes
0answers
31 views
Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?
There are instances in literature where we need to change loss function in order to escape from gradient problems.
Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
0
votes
0answers
14 views
A 2D distance measure that gives weightage to the angle between the two points?
n1 n8 n7
n2 c n6
n3 n4 n5
Assume that all neighbors ni are (Euclidean) ...
1
vote
1answer
37 views
What are the iid random variables for a dataset in the GAN framework?
I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.
The answer tells that it is due to the law of large numbers which is based on the assumption ...
0
votes
1answer
33 views
How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
The research paper titled Improved Training of Wasserstein GANs proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator.
We now propose an ...
1
vote
2answers
248 views
What is Lipschitz constraint and why it is enforced on discriminator?
The following is the abstract for the research paper titled Improved Training of Wasserstein GANs
Generative Adversarial Networks (GANs) are powerful generative models,
but suffer from training ...
0
votes
2answers
103 views
What is meant by an axis of a tensor?
Tensor is an ordered collection of elements. The elements are generally real numbers. Tensors are used in deep learning for storing data.
There is a wide usage of the word "axis" related to ...
3
votes
1answer
57 views
What are the necessary mathematical properties to be a loss function in gradient based optimizations?
Loss functions are used in training neural networks.
I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.
I ...
2
votes
0answers
23 views
Is there any closed form analytical expression to represent fractional max pooling?
There are Nineteen types of pooling layers in PyTorch.
Almost all of the layers are provided with corresponding analytical formulae. But analytical formulae are not provided for the fractional max-...
1
vote
1answer
75 views
Questions about a research paper on salient region detection and segmentation
I am reading this paper in an attempt to recreate the salient region detection and segmentation model employed. I have the following questions pertaining to section 3 of the paper and I would highly ...
1
vote
0answers
40 views
How can I compute a mathematical formula for my CNN?
Let's say, for example, I have built the following CNN model using Keras:
...
1
vote
1answer
36 views
How to interpret the policy gradient expression in reinforcement learning?
I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
3
votes
1answer
146 views
Why is tanh a "smoothly" differentiable function?
The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
1
vote
1answer
91 views
What math should I learn before and while using and applying deep learning?
I want to learn deep learning. After researching a little, I came to the conclusion that I need a lot of math. I've started a linear algebra course, and it takes a long time (2-3 weeks). I want to ...
1
vote
1answer
115 views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
0
votes
0answers
22 views
Derivation of an probability expansion used in Word2Vec classifier model
We are using the following notations, for this question, to calculate the probability values
\begin{array}{|c|c|} \hline
\text{$w$} & \text{target word embedding vector} \\ \hline
\text{$c$} &...
0
votes
0answers
21 views
Are there any good references that describe the equations of the forward pass of Graph Neural Networks?
I am trying to program Graph Neural Network from scratch.
Can the community please suggest a good reference/s to read about the equations of the forward pass in Graph Neural Networks, especially in ...
0
votes
0answers
24 views
(Deep) feature engineering for lambda terms (mathematical expressions, higher order logic formulas) - is such thing?
Automated theorem proving with (deep) reinforcement learning (DRL) approach is hot topic in current AI research when domains of games are becoming saturated and completed research topics. For example, ...
0
votes
1answer
45 views
how to go from mathematical problem to neural network (and back)?
I am a little confused on how, you can find online papers that describe complex Machine Learning formulas in a mathematical/probabilistic way, and, in the other hands, easy tutorials that teach you ...
3
votes
1answer
164 views
How is the state-value function expressed as a product of sums?
The state-value function for a given policy $\pi$ is given by
$$\begin{align}
V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\
&=E_{\pi}\...
0
votes
1answer
60 views
Why don't integrated gradients explain samples correctly?
I have a linear tabular dataset made of floats. The dataset follows a simple rule like:
...
1
vote
1answer
24 views
How is the variational lower bound for hard attention derived in Show, Attend and Tell
How is the jump from line 1 to line 2 done in equation 10 of Show, Attend and Tell?
While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is ...
-2
votes
1answer
55 views
Assumptions of a Linear Regression [closed]
I was going through the concept of Linear Regression and ran into the concept of deciding whether a Linear Regression Model is the best fit for your data by 5 assumptions:
Linearity
Homoscedasticity
...
0
votes
0answers
17 views
How do we get from entropy to KL divergence in this paper?
I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the ...
0
votes
0answers
24 views
Do $V_\theta$ and $V_s$ represent partial or total derivatives in the paper "Learning Continuous Control Policies by Stochastic Value Gradients"?
I was reading up on the Stochastic Value Gradients paper by Heess et al. In the paper, they describe a recursive process to calculate path-wise derivatives via equations (3) and (4), at the bottom of ...
2
votes
0answers
36 views
Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?
In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
1
vote
0answers
47 views
What are the math theorems regarding the Multilayer Perceptron?
I've come across a theorem "Convergence theorem
Simple Perceptron" for the first time, here-> https://zaguan.unizar.es/record/69205/files/TAZ-TFG-2018-148.pdf, page 27, (is in Spanish)
...
0
votes
1answer
70 views
What's the difference between a 1d tensor and a 2d tensor with 1 dimension?
I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this:
...
0
votes
1answer
79 views
Why can a neural network use more than one activation function?
From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
0
votes
0answers
14 views
Is there an arrow missing in the derivation of front-door adjustment formula from do-calculus?
Here is Judea Pearl's derivation of the front-door adjustment formula:
Is there an arrow from Genotype to Cancer missing in the second diagram at the right?
just like this?
