Questions tagged [deep-learning]

For questions related to deep learning, which refers to a subset of machine learning methods based on artificial neural networks (ANNs) with multiple hidden layers. The adjective deep thus refers to the number of layers of the ANNs. The expression deep learning was apparently introduced (although not in the context of machine learning or ANNs) in 1986 by Rina Dechter in the paper "Learning while searching in constraint-satisfaction-problems".

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

The MLP output of a neural network can be written as $||x||||w_l||cos(\theta_l)$, why is the norm easier to maximize?

The MLP output of a neural network is a dot product between the weights and the input and therefore can be written as $||x||||w_l||cos(\theta_l)$(https://en.wikipedia.org/wiki/Dot_product#...
1
vote
0answers
23 views

Why do you calculate the mean and standard deviation over the complete dataset before training rather than for every batch?

In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible. Mostly a min-max scaler is used. Alternatively, there is also a ...
0
votes
0answers
10 views

How to make CNN to recognize whole picture, not just the details?

In my current project I use CNNs to analize plots (CNN autoencoders for feature extraction and KMeans for clusterization) and I get the feeling that these networks, can extract only features that are ...
1
vote
1answer
19 views

What would be the state of the art image captioning deep learning model?

I saw a couple of architectures, like CNN-LSTM, with and without attention model, use of Glove vector, self-critical models, etc. I am overwhelmed looking at different notebooks and architectures, ...
-2
votes
0answers
18 views

extract desire keyword/text pair

I am looking for extract keyword pair from text files. They might not be next to each other and do not have same pattern for each occurrence. And I would not think regex will works because there is no ...
1
vote
1answer
31 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
0
votes
1answer
19 views

How does high entropy targets relate to less variance of the gradient between training cases?

I've been trying to understand the Distilling the Knowledge in a Neural Network paper by Hinton et al. But I cannot fully understand this: When the soft targets have high entropy, they provide much ...
2
votes
1answer
31 views

How to improve a trained model over time (i.e. with more predictions)?

I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
1
vote
1answer
27 views

Can residual connections be beneficial when we have a small training dataset?

I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is ...
1
vote
1answer
40 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
0
votes
0answers
20 views

How can AI algorithms be used in regards to cryptocurrency and token mining?

I am new to AI and lack the knowledge of it's capabilities. A question popped in my head, in regards to Blockchains and the mining of cryptocurrencies and tokens, how can machine learning algorithms ...
0
votes
0answers
15 views

How do the trainable projection layer used in PRADO and pQRNN work?

Trainable projection layers are said to be a very powerful thing but after reading: https://www.aclweb.org/anthology/D19-1506.pdf https://arxiv.org/pdf/2101.08890.pdf I don't understand how it works....
1
vote
2answers
34 views

How can “any process you can imagine” be thought of as function computation?

I stumbled upon this passage when reading this guide. Universality theorems are a commonplace in computer science, so much so that we sometimes forget how astonishing they are. But it's worth ...
1
vote
0answers
26 views

What are some use cases of discrete optimization in Deep Learning?

When we talk of optimization, it usually boils down to gradient descent and its variants in the context of deep learning. However, I wonder if there are some works that use discrete optimization in ...
0
votes
0answers
23 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
2
votes
2answers
180 views

Is the target assumed to be a noisy version of the output of the model in machine learning?

I wonder if the following equation (you can find it in almost every ML book) refers to a general assumption that we make when using machine learning: $$y = f(x)+\epsilon,$$ where $y$ is our output, $f$...
0
votes
1answer
34 views

How to transfer declarative knowledge into neural networks

Humans learn facts about the world like "most A are B" by own experience and by being told so (by other people or texts). The systems and mechanisms of storage and usage of such facts (by an ...
0
votes
0answers
15 views

Dealing with huge peak in data distribution

I am trying to predict a continuous value using a deep neural network. I have about 100,000 samples, where input is a sequence of RNA, and output is a continuous metric determining the quality of the ...
1
vote
0answers
11 views

Use case for dilated pooling operator used in a Machine Learning model?

I have seen that most of the deep-learning frameworks have the ability to do dilated pooling. Many frameworks have recently been updated to add the dilated property to the pooling. However, I have not ...
0
votes
2answers
29 views

Does anybody know what would happen if I changed input shape of pytorch models?

In this https://pytorch.org/vision/stable/models.html tutorial it clearly states: All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of ...
0
votes
1answer
42 views

Is the final model scaling done on the full training set?

We have our training set and our test set. When we scale our data we "fit" the scaler transform to the training set and then we scale both the training set and test set using this scaler ...
0
votes
0answers
16 views

Multi dimensional LSTM modeling in KERAS

I have a database of time series signals with multiple features and Im trying to build a model to predict whether or not two samples are related to each other. For example : a database of 1000 sample ...
0
votes
0answers
11 views

Controlling mutual information in latent variables

Recently, I read some interesting papers on mutual information (MI) estimation in high dimensional variables using neural networks [Belghazi et al., 2018][Poole et al., 2019]. These methods besides ...
0
votes
1answer
40 views

Is it possible that the model is overfitting when the training and validation accuracy increase?

I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are. I am working with a large dataset (around ...
0
votes
0answers
8 views

Is it possible to get this loss with spikes, when training an LSTM with the cross-entropy on a multi-class classification problem of a time series?

The main question here will be "should I look for a bug?" My setup is a time series multiclass classification task, labeled per frame. I am using an LSTM, feeding inputs and using ...
1
vote
1answer
55 views

How to construct Transformers to predict multidimensional time series?

There is plenty of information describing Transformers in a lot of detail how to use them for NLP tasks. Transformers can be applied for time series forecasting. See for example "Adversarial ...
0
votes
2answers
74 views

Could I just choose the other (non-predicted) class when the accuracy is low?

I have a binary classification problem. My neural network is getting between 10% and 45% accuracy on the validation set and 80% on the training set. Now, if I have a 10% accuracy and I just take the ...
0
votes
0answers
15 views

How to have a DNN output a classification for each user at once?

I have a Reinforcement Learning environment with an agent that allocates power values to different users. To do so, I have thought of implementing a deep neural network like the one shown in the ...
0
votes
0answers
32 views

Is vectorizing backpropagation feasible?

Does it make sense to have the backpropagation of a neural network layer happen all at once if the learning rate is lowered? This would mean the new weights of that layer would be independent of each ...
0
votes
0answers
12 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
1
vote
1answer
20 views

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
1
vote
1answer
66 views

In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/ I am getting stuck on the last step of updating the ...
0
votes
0answers
33 views

What are the best optimizations I can add to my neural network?

I am making an artificial neural network from scratch (without nn libraries) in python. So, as you can guess, its extremely unoptimized and slow. For this neural ...
0
votes
0answers
15 views

Mixed precision training - why we're fine with doing point wise operations in FP32

I'm starting to learn more about mixed-precision training, and I'm in particular confused about point-wise operations. In the original article (link), the authors mention, citing: Point-wise ...
0
votes
1answer
21 views

Can Facebook's LASER be used like BERT?

Can Facebook's LASER be fine-tuned like BERT for Question Answering tasks or Sentiment Analysis? From my understanding, they created an embedding that allows for similar words in different languages ...
0
votes
1answer
23 views

How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?

I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the ...
1
vote
0answers
12 views

How to deal with losses on different scales in multi-task learning?

Say I'm training a model for multiple tasks by trying to minimize sum of losses $L_1 + L_2$ via gradient descent. If these losses are on a different scale, the one whose range is greater will dominate ...
0
votes
0answers
29 views

Are there regularisation methods related only to architecture of the CNNs?

Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the ...
0
votes
1answer
23 views

Validation Accuracy remains constant while training VGG?

I posted this question on stackoverflow and got downvoted for unmentioned reason, so I'll repost it here, hoping to get some insights This is the plot This is the code: ...
1
vote
1answer
20 views

Does use of GPU reduces time of execution of .predict() method in Keras/TF model? [closed]

Does using GPU instead of CPU with tensorflow and keras model reduces the time of .predict() method on a trained network or maybe it only reduces time of training of the network?
0
votes
0answers
23 views

Why is my siamese network learning very well in e.g. 1 out of every 5 runs?

Why is my siamese network learning very well in e.g. 1 out of every 5 runs? The rest of the time it's not learning and maintains an accuracy of 0.5. Any explanations? Is the contrastive loss taken in ...
1
vote
1answer
30 views

How to define a “don't care” class in time series classification in Pytorch?

This is a theoretical question. Setup I have a time series classification task in which I should output a classification of 3 classes for every time stamp t. All ...
4
votes
0answers
39 views

It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems?

It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems? If we take, for example, the travelling salesman problem (or the dominating set problem). Let's say ...
0
votes
0answers
17 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
2
votes
2answers
60 views

Is feature engineer an important step for a deep learning approach?

I'd like to ask you if feature engineering is an important step for a deep learning approach. By feature engineering I mean some advanced preprocessing steps, such as looking at histogram ...
1
vote
2answers
43 views

What model structure I should use to train on low res and blurry images?

I am looking for advice or suggestion. I have photos like these: photo_1 and photo_2 and many more similar to that. The average shape of these photos is about 160 x 100. What we are doing is we are ...
0
votes
0answers
11 views

Handling multi scale data input to neural networks

This is a theoretical question. Let's assume the simplest setup of binary classification of a time series. The input every frame is a multi channel sample of many sensors. Each sensor is completely ...
4
votes
2answers
138 views

When is it necessary to manually extract features to feed into the neural network rather than providing raw data?

Usually, Neural Networks uses raw data. You do not need to extract features manually. NN's can find & extract good features which is a pattern of an image, signal or any kind of data. When we ...
1
vote
1answer
45 views

Is there a full and precise formulation of Theorem 1 in the Integrated Gradients paper?

Theorem 1 (page 5) in the paper about Integrated Gradients states that Integrated gradients is the unique path method that is symmetry-preserving. What I miss is A precise formulation of the ...
0
votes
0answers
32 views

How to implement neural network using FPGA?

I want to design neural networks using FPGAs and make neuromorphic chip. Is there a tutorial or course to learn how to do that and to implement neural network using FPGA especially for evaluating the ...

1
2 3 4 5
30