A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

# Tag Info

0

So, after playing a bit with this, I suspect that the reason this doesn't work is the fact that the np.linalg.inv (or pinv) function is not powerful/precise enough to deal with an inverse of a (784,784) matrix. I noticed that if I ask that inverse I get garbage results (even though no errors are thrown), and $W W^{-1} \neq I$. But if I ask for the other ...

0

Maybe you are looking for a combination of a version control system (like git and Github) and a tool like comet.ml. In the past, I used comet.ml to keep track of different experiments performed with different hyper-parameters or different versions of the code. There are other alternatives to comet.ml, such as sacred, but they may also have different features ...

0

Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, ...

0

Setting aside the dubious nature of subliminal messages. You have to clearly define what subliminal is. In all your cases presented, a manual algorithmic approach would be better. An ANN would tend to average out the subliminal messages as noise, just as your brain would. In the case of #2, #3, it's a matter of contrast, in which case computer vision methods ...

1

To emphasize (and this is not emphasized in this answer), in the case of neural networks, the biases or, more precisely, the connections (or weights) between biases and other neurons are also learnable parameters, so the back-propagation algorithm calculates a gradient of the loss function that contains the partial derivatives with respect to the connections ...

1

In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a ...

2

Mathematical Exploration let $\Theta^+$ be the pseudo-inverse of $\Theta$. Recall, that if a vector $\boldsymbol v \in R(\Theta)$ (ie in the row space) then $\boldsymbol v = \Theta^+\Theta\boldsymbol v$. That is, so long as we select a vector that is in the rowspace of $\Theta$ then we can reconstruct it with full fidelity using the pseudo inverse. Thus, ...

3

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail. Using backpropagation to compute the gradient is an ...

2

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers) The better question is ...

2

It depends. It could give you a boost or it could not. Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches). Ideal World: We optimize the network as a whole to gain better course grained features over the ...

0

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct. Hence, $p_i \log(q_i) = 0 \log(q_i) = 0$ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i)$ for the correct prediction. Therefore, your loss reduces to:  H(p,q) = - \sum p_i \...

3

I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS. It's usually called the loss function (and, in general, objective function) and often denoted as $\mathcal{L}$ or $L$ (or something like that, i.e. it is not really important how you denote it). The specific function used as a ...

3

Welcome to AI Stack exchange! You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing. So what do we ...

0

Ideally, yes. Ideally, because the network should be fed with the words of an entire book (wich vary around 100k words). With an hypotetical amount of processing power, you should be able to just train the NN with like thousands of books. It might be possible to be trained with quantum computers.... who knows... For smaller stories, I think that the major ...

0

Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model. U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one ...

1

If is a truly a random number, and you could guess each of the next successive five in sequence, then you could win the lottery consistently. This is one of the first tasks many people try to do when first learning machine learning. If the lottery is truly a random physical process with fair, i.e., balanced ping pong balls, then you cannot predict which ...

1

There are two weight-initializing methods for neural networks: 1-Zero initializing 2-Random initializing https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78 If you choose zero initalizing method in every train loop, you may get same results OR you can use transfer learning according to your problem, it allows ...

1

I don't think you can. Say a NN with 3 layers gives an accuracy of 95.3% and another NN with 4 layers gives an accuracy of 95.4%. Then there is no guarantee that the 4 layer NN is better than the 3 layered NN. Since with different initial values the 3 layer NN might perform better. You could run multiple times and probabilistically say that this is better, ...

4

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases) If you are using gradient descent as a training method, then differentiability ...

1

It depends on the data. If it is structured like form data, then you might not need AI at all — simple regular expression patterns might be fine. This would apply for example to address data. If you have the word street followed by a colon, followed by some text, it seems fairly obvious that this is the name of a street, and possibly also a house ...

0

But because the inputs have to have a fixed length Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph. This has the advantage of being able to take into account word order and constellations, as ...

0

I ran a lot of randomly created networks to solve this problem, but none of the structures where able to reliably "solve" this problem. Of course, some of them where able to solve it one time, some of them even twice but there where only one which solved it 3 times: LearningRate: 0,510141694690167 Momentum: 0,962972165068133; Layer/Neuron-Count: 2 (14, 9) ...

2

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value. For ...

4

One of the things you may have missed out in your design is some arrow going from the last layer in the layer side back to the first layer. e.g. If you're thinking some thoughts you'll want those to keep going round and round in your head. At the moment while your design would have some way to learn and react to the environment, it wouldn't have any short ...

1

Simply said, predicting pseudo random number is just not possible for now. Pseudo random numbers generated now have a high enough "randomness" so that it cannot be predicted. Pseudo random numbers is the basis of modern cryptography which is widely used in the world wide web and more. It may be possible in the future through faster computers and stronger AI, ...

1

Getting the intent of the sentence is not an easy task. To get you started on what to do, have a look on word vectors. You can also download pre-trained word2vec models. They help in getting similarity of words and reasoning with words. To get the intent of a sentence, you can use LSTM. Fun fact most NLP algorithms strip away punctuation with is sufficient ...

1

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you only need one hidden layer to approximate the multiplication on a compact, note that you need to apply a non linear activation on the hidden layer to do this.

2

A couple of points: Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high. Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea. Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations. Model ...

0

A fairly recent paper posits an answer to this: Reconciling modern machine learning practice and the bias-variance trade-off. Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal https://arxiv.org/abs/1812.11118 https://www.pnas.org/content/116/32/15849 I'm probably not qualified to summarize, but it sounds like their conjectured mechanism is: by having ...

2

A neural network is composed of continuous functions. Neural networks are regularized by adding an l2 penalty on the weights to the loss function. This means the neural network will try to make the weights as small as possible. The weights are also initiallized with a N(0, 1) distribution so the initial weights will also tend to be small. All of this means ...

1

There are probably multiple different explanations and reasonings, but I can offer you one. If your output vector contains negative values, to get something that's related to probabilities (all components positive, summing to $1$) you cannot do what you suggested because you can possibly get a negative probability which doesn't make sense. Good property of ...

0

There is normally more to generalisation than just increasing training data. It helps to make the task noisy, through various means. One common and popular method is to use dropout, which encourages the network to utilise every node, and avoids dependencies on small clusters of nodes. So how does making the task more noisy help with generalisation? Well it'...

1

Here's a link to my answer on CV Stack Exchange, where I have mentioned about latent spaces and some deep learning models that learn these representations: https://stats.stackexchange.com/questions/442352/what-is-a-latent-space/442360#442360 In short, deep learning models for Domain Adaptation, Computer Vision, Natural Language Processing, Recommendation ...

2

From here: Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.

3

The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use) generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.

0

My suggestion is to go with 1st option. reason is you will get to know much about data and initial stage will find some challenges in developing the model, over a period of time you will get to better results after hypertunning. Please go through article , ignore you have already read this article

2

Look at Google's Open Image Dataset @ https://storage.googleapis.com/openimages/web/index.html They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships. Here is the link for the traffic signs dataset.

0

You can do custom POS Tagging and use it as a multi featured sequence2sequence.

2

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life. If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the ...

2

TL;DR Here is a beautiful explanation with diagrams: source To address: the cell state is essentially long term memory embedding (correct me if I'm wrong) The embedding can be long or short term and it is a vector. To answer: Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special ...

Top 50 recent answers are included