Questions tagged [math]

For questions about mathematics related to artificial intelligence.

Filter by
Sorted by
Tagged with
1
vote
1answer
614 views

What is the intuition behind the entropy formula used in the ID3 algorithm?

What is the intuition behind the following entropy formula used in the ID3 algorithm? $$ \text{info}(D) = -\sum_{i=1}^m p_i \log_2(p_i) $$
2
votes
3answers
106 views

What does the formula $1-\sum_i(e_i-a_i)^2$ mean in this NEAT Python API?

I have looked at the documentation for the NEAT Python API found here, but it shows calculus like this: The error for each genome is $1-\sum_i(e_i-a_i)^2$ I haven't learned calculus at the moment....
5
votes
1answer
302 views

What is “conditioning” on a feature?

On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes; Redacted phase space: Studying the distribution of inputs ...
4
votes
1answer
58 views

What do the subscripts mean in $N_{t,n,\sigma,L}$?

A neural network can apparently be denoted as $N_{t,n,\sigma,L}$. What do these subscripts $t, n, \sigma$ and $L$ mean? Could you link me to a paper, article or webpage with an explanation for this?
5
votes
1answer
85 views

Why does a Lipschitz continuous discriminator in GANs assure statistical boundedness?

I have been reading the paper which introduced spectral normalization in GANs. At some point the paper mentions the following: The machine learning community has been pointing out recently that ...
4
votes
2answers
55 views

Which linear algebra book should I read to understand vectorized operations?

I am reading the Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining ...
0
votes
1answer
36 views

Backpropagation: Chain Rule to the Third Last Layer

I'm trying to solve dLoss/dW1. The network is as in picture below with identity activation at all neurons: Solving dLoss/dW7 is simple as there's only 1 way to output: $Delta = Out-Y$ $Loss = abs(...
3
votes
1answer
48 views

What are the differences between stability and convergence in reinforcement learning?

The terms are mentioned in the paper: “An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning.” (Sutton, Mahmood, White; 2016) and more, of course. In which paper, they ...
3
votes
2answers
69 views

Why is the max a non-expansive operator?

In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming ...
0
votes
2answers
39 views

Is it still called linear separation with a layer of more than 1 neuron

A single neuron will be able to do linear separation. For example, XOR simulator network: ...
2
votes
1answer
97 views

Is there a mathematical example for Conditional Random Fields?

I am learning about probabilistic graphical models and I was wondering if there is an example explaining the math behind conditional random fields. Looking solely on the formula, I have no idea what ...
10
votes
2answers
2k views

Is the mean-squared error always convex in the context of neural networks?

Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks. Let's say we have the following: $X$: training ...
7
votes
3answers
2k views

Why is the derivative of the activation functions in neural networks important?

I'm new to NN. I am trying to understand some of its foundations. One question that I have is: why the derivative of an activation function is important (not the function itself), and why it's the ...
4
votes
2answers
81 views

How would an AI work out this question?

I am trying to create an AI that makes reasonable guesses at truths of statements. However... Human: "Prove that no number exists which is one more than a billion." AI: "Is it true for the number 1? ...
1
vote
1answer
92 views

Is there a rigorous proof for finding Hopfield minima?

I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just let the network keep ...
3
votes
1answer
109 views

How can I derive the rotation matrix from the axis-angle rotation vector?

Given an axis-angle rotation vector $\Theta = (2,2,0)$, after finding the unit vector $k=(1/\sqrt{2}, 1/\sqrt{2}, 0)$ and angle $\theta = 2\sqrt{2}$ representing the same rotation, I need to derive ...
2
votes
0answers
62 views

What is the meaning of the words 'bias' and 'variance' in RL?

In algorithms like MC/TD (tabular value approximation) two of the metrics used to measure their performance are the bias and the variance. What do these terms mean? And which characteristic of the ...
2
votes
1answer
41 views

Why is the expectation calculated over finite number of points drawn from a probability distribution?

This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is $f(x)$ not being multiplied by $p(x)$?
3
votes
4answers
1k views

AI applications of the Fibonacci series

I have been looking at Fibonacci series, the golden ratio and its uses in nature, like how flowers and animals grow based on the series. I was wondering whether we could use the Fibonacci series and ...
4
votes
1answer
149 views

How is G(z) related to x in GAN proof?

In the proofs for the original GAN paper, it is written: $$∫_x p_{data}(x) \log D(x)dx+∫_zp(z)\log(1−D(G(z)))dz =∫_xp_{data}(x)\log D(x)+p_G(x) \log(1−D(x))dx$$ I've seen some explanations asserting ...
5
votes
1answer
90 views

Is there a limit of minimum error for a particular training dataset in artificial Neural Network?

In error-based learning using gradient descent, if I give you a training dataset, then can you find the minimum error after training? And the minimum error should be true for all architectures of a ...
1
vote
1answer
70 views

What does “probabilistically” mean?

I'm reading the A. E. Eiben and J. E. Smith book Introduction to Evolutionary Computing (Springer 2003). On section 3.5 Recombination, page 47, the second paragraph said: Recombination operators ...
11
votes
3answers
2k views

What sort of mathematical problems are there in AI that people are working on?

I recently got a 18-month postdoc position in a math department. It's a position with relative light teaching duty and a lot of freedom about what type of research that I want to do. Previously I was ...
15
votes
3answers
907 views

What are the mathematical prerequisites for an AI researcher?

What are the mathematical prerequisites for understanding the core part of the algorithms in artificial intelligence and developing own algorithm? Please, refer me the specific books.
9
votes
2answers
2k views

Which areas of applied math are relevant to AI? [duplicate]

My background is in electrical engineering. I have a good grasp of CS foundations (e.g. data structures, algorithms, operating systems, discrete math and software engineering). I have option of ...
2
votes
1answer
50 views

Is the Markov property assumed in the forward algorithm?

I'm majoring in pure linguistics (not computational), and I don't have any basic knowledge regarding computational science or mathematics. But I happen to take "Automatic Speech Recognition" course in ...
3
votes
2answers
401 views

What does the argmax of the expectation of the log likelihood mean?

What does the following equation mean? What does each part of the formula represent or mean? $$\theta^* = \underset {\theta}{\arg \max} \Bbb E_{x \sim p_{data}} \log {p_{model}(x|\theta) }$$
3
votes
2answers
86 views

Are there any discount-factors based on branching factors?

I recently came across this function: $$\sum_{t = 0}^{\infty} \gamma^t R_t.$$ It's elegant and looks to be useful in the type of deterministic, perfect-information, finite models I'm working with. ...
3
votes
3answers
414 views

Which neural network should I use to approximate a specific function?

We have convolutional neural networks and recurrent neural networks for analysing respectively images and sequential data. How do I determine which neural network architecture is more appropriate to ...
4
votes
3answers
196 views

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my ...
4
votes
1answer
157 views

How are filters weights updated for a CNN?

I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the ...
3
votes
1answer
81 views

Standard deviation of the total input to a neuron

Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates the standard deviation of the output of a hidden neuron. From: $$ \sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2] $$ When I ...
4
votes
2answers
453 views

Viola Jones Algorithm

Can Viola Jones algorithm be used to detect the facial emotion. Actually it was used in creating harr-cascade file for object and facial detection, but what confused me is whether it can be used to ...
13
votes
2answers
566 views

Is there any scientific/mathematical argument that prevents deep learning from ever producing strong AI?

I read Judea Pearl's The Book of Why, in which he mentions that deep learning is just a glorified curve fitting technology, and will not be able to produce human-like intelligence. From his book ...
4
votes
1answer
155 views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
3
votes
0answers
88 views

What characteristics make it difficult for a Neural Network to approximate a function?

What are the characteristics which make a function difficult for the Neural Network to approximate? Intuitively, one might think uneven functions might be difficult to approximate, but uneven ...
2
votes
0answers
42 views

Should I use the hyperbolic distance loss in the case of Poincarè Disk Model?

I trained a neural network which makes a regression to a Poincarè Disk Model with radius $r = 1$. I want to optimize using the hyperbolic distance $$ \operatorname{arcosh} \left( 1 + \frac{2|pq|^2|...
6
votes
2answers
654 views

How is local minima possible in gradient descent?

Gradient descent works on the equation of mean squared error, which is an equation of a parabola $y=x^2$. We often say that weight adjustment in a neural network by gradient descent algorithm can hit ...
5
votes
1answer
275 views

Why is the denominator ignored in the Bayes' rule?

The naive Bayes' generative algorithm is often represented by the following formula $$\text{argmax}_{y} p(y \mid x) = \text{argmax}_y \frac{p(x|y)p(y)}{p(x)} \approx \text{argmax}_y p(x|y)p(y)$$ Why ...
5
votes
2answers
111 views

Are on-line backpropagation iterations perpendicular to the constraint?

Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2 relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the ...
4
votes
1answer
94 views

Why does the cost function contain a 2 at the denominator?

A cost function used in machine is often the following $$C = \frac{1}{2} \| y - \hat{y} \| ^2$$ Why is there $\frac{1}{2}$ in front of the squared distance?
1
vote
3answers
90 views

Is it possible to compute $P( F \mid S )$ given $P(F \mid S,A)$, $P(F \mid S, \lnot A)$?

I have a bayesian network, which has the following data: $P(S) = 0.07$ $P(A) = 0.01$ $P(F \mid S,A) = 1.0$ $P(F \mid S, \lnot A) = 0.7$ $P(F \mid \lnot S, A) = 0.9$ $P(F \mid \lnot S, \lnot A) =...
1
vote
1answer
63 views

Is the next state drawn from the joint distribution of the previous state and action?

Suppose $G_t$, the discounted return at time $t$ is defined as: $$ G_t \triangleq R_t+\gamma R_{t+1}+\gamma^{2}R_{t+2} + \cdots = \sum_{j=1}^{\infty} \gamma^{k}R_{t+k}$$ where $R_t$ is the reward at ...
2
votes
1answer
95 views

Which matrix represents the similarity between words when using SVD?

Two words can be similar if they co-occur "a lot" together. They can also be similar if they have similar vectors. This similarity can be captured using cosine similarity. Let $A$ be a $n \times n$ ...
0
votes
2answers
1k views

What are examples of applications of the Fourier transform to AI?

The (discrete and continuous) Fourier transform (FT) is used in signal processing in order to convert a signal (or function) in a certain domain (e.g. the time domain) to another domain (e.g., ...
1
vote
0answers
21 views

Does a mechanical system repeats itself?

The inverted pendulum problem is a famous control task. It can be solved with a technique called system identification. System identification means to formalize the state-action space of a system in a ...
2
votes
0answers
28 views

Calculating tangent vector of curve s(P,$\alpha$) at given point $\alpha$ = 0

I am reading the paper "Transformation Invariance in Pattern Recognition – Tangent Distance and Tangent Propagation", where the tangent vector is calculated for the given curve $s(P,\alpha)$ at $\...
7
votes
2answers
2k views

Why exactly do neural networks require i.i.d. data?

In reinforcement learning, in general, successive states (actions and rewards) are highly correlated. An "experience replay" buffer was used, in the DQN architecture, to avoid training the neural ...
4
votes
0answers
93 views

What are the main benefits of using Bayesian networks?

I have some trouble understanding the benefits of Bayesian networks. Am I correct that the key benefit of the network is that one does not need to use chain rule of probability in order to calculate ...
6
votes
4answers
4k views

What are the mathematical prerequisites to be able to study general artificial intelligence?

What are the mathematical prerequisites to be able to study general artificial intelligence (AI) or strong AI?