New answers tagged

2

Transformer models have limited sequence length at inference time because of positional embeddings. But there are workarounds. Self-attention in transformer does not distinguish the order of keys/values, it works as if the sequence is a bag of words. So to expose the sequence order to the model, one typically adds an extra "positional embedding" ...


1

I would say that in general situation more estimators are better. RandomForest fits a lot of estimators - decision trees that take a subset of data (obtained sampling with replacement) and subset of features (by default sqrt(n_features) in sklearn). Each of these estimators is noisy and prone to overfitting, producing a complicated decision surface. But when ...


0

Generative models like latent variable models (e.g. VAE) use directed graphical models and these sort of factorizations as a foundation for learning. In VAEs, Neural nets are used to estimate posteriors/priors to generate samples. This sort of explicit factorization is helpful in other generative models as well like autoregressive models which are basically ...


0

The size of a concept representation is important because the time to write/process this representation is a lower bound on the running time of the learning algorithm. For example, you can represent the boolean parity function with a circuit composed of $\land$, $\lor$ and $\neg$ such that its size is polynomial in $n$, while, if you used a disjunctive ...


0

Let's brake down each component of the formula, on the left side we have ${\Omega(g)}$ The authors state that they use omega to refer to the set of models too difficult to be tractable (or explained), so g is just a model, or set of weights, omega g is a set of models. Next we have ${\infty \mathbb{1}}$ This is not a common notation, at least as far as I'm ...


1

Is this a valid implementation of second-order regression? No, but it is not far off. To perform a full second-order regression, you will need all terms for $x_{i,j}x_{i,k}$ where the first index is the example and the second index the feature. This includes every combination of two input variables. In your element-wise squaring you only produce terms $x_{i,...


1

For deep learning models, embedding vectors have become the standard way of encoding text features almost immediately after their introduction. The reason for this is that neural networks work with data encoded with continuous values ranging from 0 to 1 (or sometimes from -1 to 1). Bag of Words and TF-IDF can be modified to produce values in this range, but ...


1

Intuitively, I feel like if there are 30 foods, each with 2 states, then that is 60 states, no $2^{30}$. Let's try it with 3 pellets. If you are right there would be $2 \times 3 = 6$ states, if the authors are right there would be $2^3 = 8$ states. Using * for a pellet, and - for a space, we have the following states: * * * * * - * - * * - - - * * - * - - -...


2

What you refer to as logic AI is a subset of what is called symbolic AI, as you manipulate symbols, according to certain rules (which could be rules of logic). These rules are either authored by a human being, or they can be learned from examples. There are algorithms to derive decision trees (eg ID3) or other rule sets from training data. But the important ...


1

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction. For example: Let $L(\theta) = \theta^2 $ and $\theta_0= 3$ Let the subscript n in $\theta_n$ denote the iteration number. Then $\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$ For the loss to decrease in this case $\epsilon < 1$ needs to hold ...


0

The other answer gives the nice famous example of the sort of problem that game theory tackles and it partially describes what machine learning is. However, it does not emphasize that this type of game theory problem, where you have two or more agents competing with each other, also appears in the context of machine learning. More concretely, machine ...


0

K means tried to cluster data points into 0 and 1 rules for cluster assignment i.e. Data Point belongs to a class or it does not. But sometimes the data points comes from classes whose probability distribution functions have overlap with each other. Now Hard Clustering of K means leads to truncation on the data generating\original probability distribution ...


0

This is just a heuristic, but how about f(V) = cosine_similarity(A,V) * min(1, 1 - cosine_similarity(B,V))? I'm applying the min here so that the multiplier won't get larger than one. Up to you whether you want to include it or not. This could be tuned further as f(V) = max(0, cosine_similarity(A,V))^a * min(1, 1 - cosine_similarity(B,V))^b, but you'd need ...


0

As the objective is to find the most similar to A and disimilar Vector to B approach 2 would be the most appropriate. Why not Approach 1: It can lead to confusing results. If you look at the example below multiple scenarios lead to same final value. This may lead to few problems : For same output how do you know which vector should be preferred Subtracting ...


1

I will give you a few scenarios where matrix factorisation stills works pretty well. Topic Modelling : Given a matrix of Document as Rows and Terms/Words as column you can use Non Negative Matrix factorisation to identify Topics. Number of Topics is defined by user or can be treated as hyperparameter. Image Ref : https://towardsdatascience.com/nmf-a-visual-...


0

When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that ...


1

As DKDK said, Indeed one could fit both linear and exponential function and see which one has smaller residual, without using any complex AI. But OTOH this could be a great toy-problem for learning about neural networks. You could have a network with these parts: A network with a final sigmoid activation, which predicts whether the function is linear or not....


1

Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do interface and Interface mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called "keywords" of your configurations (such as vlan), or maybe ...


1

I have found the answer to this doubt I had. Here, 0.007709330413490534 = 1 / S, q = input, Z = 3. Basically, this is the formula to quantize the input value. If you pull out 1/S then it becomes clear. Here, is an article related to this topic.


2

How about dividing the problem? You can first train a classification model that predicts the type of function (linear or exponential). Then you can use your seperately trained nn depending on the classification output. P.S. I'm not sure why you would use a neural network for this problem. Fitting a linear/exponential function seems to be a relatively simple ...


0

Convolutional Neural Networks are mostly used for all kind of computer vision tasks. Here you can find a tutorial on how to train a CNN for image classification from scratch.


2

Since sport commentaries are a fairly restricted domain, and the language does not vary much, I would go for a canned text approach. Analyse what kind of events you get, and what variables you're dealing with. Then write some template sentences with placeholders for the variables. The more you write for the same data, the more varied your text will be. You ...


Top 50 recent answers are included