Tag Info

0

You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words. For each important ...

0

What is the time complexity? The time complexity of an algorithm is the number of basic operations, such as multiplications and summations, that the algorithm performs. The time complexity is usually expressed as a function of the input's size $n$ (but this does not always have to be the case: for instance, you can express the time complexity as a function ...

0

You could just do this; concatenate your input_vector with zero's vector that has the size of your output. Then in the first pass you concatenate with the output instaid of the zero's vector. After that repeat.. At the end just compare (compute the loss) your entire output from t0 to t1 to your target and backprop. You might want to look into recurrent ...

1

You want to look at recurrent neural networks.

1

I'll answer in a couple of stages. I feel somewhat lost as to what the input for the NN should look like. Your choices boil down to two options, each with their own multitude of variants: Vector Representation: Your input is a vector of the same size as your vocabulary where the elements represent the tokens in the input example. The most basic version of ...

0

You could use scikit-learn's MultiLabelBinarizer. It's essentially the multi-label equivalent of one-hot encoding. For each movie, create a vector of zeros, where each zero is associated with a particular actor. If an actor is in that movie, change their zero to a one. In the context of a neural network, think of it as each actor having their own input ...

0

Hard to say in general. Speaking from my own experience and by looking at which models win Kaggle competitions (see here and here), I would say tree-based models e.g. Random Forests, Decision Trees, Gradient Boosting are favorable over neural networks when working with low-dimensional data and easy interpretable features (usually simple tabular data with ...

0

GANs are usually trained in a self-supervised fashion, i.e. they use the unlabelled data as the supervisory signal. Note that some self-supervised learning methods are unsupervised learning techniques, given that no human-annotated data is needed. However, not all SSL techniques are used for solving an unsupervised learning task. In fact, there are SSL ...

2

What is eager learning or lazy learning? Eager learning is when a model does all its computation before needing to make a prediction for unseen data. For example, Neural Networks are eager models. Lazy learning is when a model doesn't require any training, but all of its computation during inference. An example of such a model is k-NN. Eager learning is ...

0

There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and people who have attempted to try your method of having reduced connections have seen a performance hit and so no change was made. The first reason is that if you ...

0

If you adopt a slightly different point-of-view, then a neural network of this static kind is just a big function with parameters, $y=F(x,P)$, and the task of training the network is a non-linear fit of this function to the data set. That is, training the network is to reduce all of the residuals $y_k-F(x_k,P)$ simultaneously. This is a balancing act, just ...

0

The answer above makes some great comparisons/trade-offs. To help address the non-linearity issue with eLU units that the previous answer brings up, you can also use Leaky-ReLU units, which are linear in both the positive and negative range, and piecewise-linear across the whole real domain. Please see the link here for more details: https://qr.ae/pNsatH.

0

It doesn't. Whether or not this is useful is another story, but it is totally fine to do that neural net you have with just one input value. Perhaps you choose one pixel of the photo and make your classification based on the intensity in that one pixel (I guess I'm assuming a black-and-white photo), or you have some method to condense an entire photograph ...

1

I'm currently working with Temporal Convolution Networks (TCNs) for making predictions with time series data (link to article here: https://medium.com/@raushan2807/temporal-convolutional-networks-bfea16e6d7d2). These types of networks, like other types of convolutional networks for time series, use a dilated convolution operation, which, unlike the standard ...

1

A few years ago, deep learning was a buzz word, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!). From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, ...

1

In general it's better to not use sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use sigmoid-like function, then go with Tanh function, at least it has ~0 mean.

4

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization. Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares ...

2

ELU does not suffer from from dying neurons issue, unlike ReLU. While ELU can help you to acheive a better accuracy, it is a slower than ReLU because of its non linearity in its negative range. Choosing a right activation function totally depends on situations but you need to also consider other similar types of activation function such as leaky ReLU. Check ...

2

It depends, as mentioned in comments, on your model and labels. For example how would you use standardisation on multi classification problem? Generally, standardisation is more flavourable for input data as its mean is around 0. I assume you have a regression model and in that case using standardisation could be better than normalisation.

1

The idea is simple, but it requires some time to develop. Assumption: I am assuming in your problem the final model will have seen all possible shapes. What your algorithm needs is a convolutional NN to understand each shape by extracting features, but you just need to be very careful with pooling. Then what you need is a recurrent NN. In the example you ...

2

Currently both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models. Also these AFs, more specifically ReLU, are very fast learning AF which make them even ...

1

How to fix the network above to auto-classify XOR data, in unsupervised manner? This cannot be done, except accidentally. Unsupervised learning cannot replace or emulate supervised learning. As a thought experiment, consider why you would expect the network to discover XOR, when simply considering outputs rounded to binary, you could equally find AND, OR, ...

1

The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as saturating activation functions). For example, the sigmoid function has codomain $[0, 1]$, as you can see from the illustration below. This property/behaviour can lead to the vanishing gradient problem ...

2

I'm not sure any intelligent mechanism can be entirely free of symbolic logic. Even where a decision is statistically based, a machine that takes actions must include some form of: IF {some condition} THEN {some action} As to the popularity of newly proven statistical AI methods (ANN and genetic algorithms), this derives from the greater utility they ...

4

They don't. In fact, if we look to similar models like convolutional neural nets, we see that it's popular to have some form of a rectified linear unit (ReLU) activation in earlier layers and the output layer is often a softmax activation, which provides an output that can be viewed as a probability distribution. Generally, it depends on what you're trying ...

2

You might also ask if there's any particular reason why we would use a neural net. If we're to train a neural net to play chess, we need to be able to: 1. Feed it positions as input vectors (easy enough), 2. Decide on an output format. Perhaps a distribution over possible moves (but then, how to represent that such that the meaning of a specific output cell ...

1

Data management and bandwidth are key issues for interconnecting multiple GPUs. These are such big issues that it is hard to think about other challenges like neural network architecture, metrics, etc. The key to success for interconnecting multiple GPUs on a single computer is NVIDIA's NVLink: NVLink is a wire-based communications protocol for near-range ...

2

In reinforcement learning (RL), an immediate reward value must be returned after each action, alomng with the next state. This value can be zero though, which will have no direct impact on optimality or setting goals. Unless you are modifying the reward scheme to try and make an environment easier to learn (sometimes called reward shaping), then you should ...

2

ANNs as used today need 1. a lot of data 2. a lot of computational power. Before we had any of the above two, we didn't really know how to properly build ANNs since we didn't quite have the means to train the network, and thus couldn't evaluate it. "Symbolic AI" on the other hand, is very much just a bunch of if-else/logical conditions, much like ...

0

No, you can't. In CNN, if you want to detect landmark, you need to prepare data with region box, it's coordinates, width, height, than number of points that should be detected and points coordinates. Then your target vector should be, This is your target vector. Optionally you can use YOLO algorithm.

0

You can increase no of hidden layers. Following is an example (But not very efficient)

2

It's not a specific web for sell or share neural network models, but actually you can easily find other people models in Github! Just search it! For example, this is a random repo I've found for Cat Classification. But.. the problem is everyone have different problems. So you can't easily use other people neural network models and then use it for your ...

2

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large. I have primarily used kernel regularizers. First I try to control over fitting using dropout layers. If that does not do the job or leads to poor training accuracy I try the Kernel regularizer. I ...

0

Since after a number of iterations the cost function is not reducing, this may be able to be diagnosed as a vanishing gradient problem. A solution to this is the use of a Residual neural network. Another solution is that you carefully initialise your weights as throughout your neural network your gradient may exponentially explode or exponentially vanish. ...

0

I have not used fastai library but this also happens on tensorboard when you have more than one training being recorded on the same plot. Looking at the picture, I think this is a very special type of graph because for a single LR value you have 2 loss values associated. Put in other words, you have the same LR value for different loss values. My guess it ...

3

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability ...

0

You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value. See if the results has changed after adjusting the learning rate.

0

Just wanted to add that the new text Deep Learning Architectures A Mathematical Approach mentions this result, but I'm not sure if it gives a proof. It does mention an improved result by Hanin (http://arxiv.org/abs/1708.02691) for which I think it does give at least a partial proof. The original paper by Hanin seems to omit some proofs as well, but the ...

0

This would be more suitable as a comment but I don't have enough points; but here's my opinion. Optimisation algorithms like gradient descent are iterative algorithms. So it is rarely possible that they arrive at the minima in 1 epoch. A single epoch means that all data points have been visited once or a certain number of data samples have been taken from a ...

4

Let's quickly get out our copies of Deep Learning by Goodfellow et al. (2016). More specifically, I'm referring to page 276. On this page, the authors argue for a relatively small minibatch size, since there are less than linear returns for estimating the gradient when increasing the minibatch size. Returns here refer to the reduction of the standard error ...

0

If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information. When it comes to correlation between data samples ...

2

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs). All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without overfitting). The more nodes (or neurons) you include into the network, the more complex a ...

0

I think the choice of technique strongly depends on how fine-grained your forecast-predictions need to be. When it comes to forecasting by Reinforcement Learning (RL), one prominent example is the stock-trading RL agent. The agent must decide which stock to buy or sell, thereby drawing upon predictions concerting the expected future development of some stock....

Top 50 recent answers are included