Questions tagged [architecture]
For questions related to the architecture of AI models, e.g. the architecture of neural networks.
86
questions
1
vote
1
answer
50
views
What is the channel dimension other than color representation in Conv2D? Shall I use Conv3D instead?
I have grayscaled image, specifically medical data ultrasonography.
In the context of medical domain, there are techniques to capture data that called "View". It's like point of views of ...
0
votes
0
answers
19
views
Deep Learning: Architecture vs. Features for Performance?
In deep learning, when aiming for peak metric performance, is a well-designed architecture with imperfect features/dataset generally preferable to a poorly designed architecture with high-quality ...
3
votes
0
answers
63
views
How random should an untrained generative AI output really be?
I am developing a particular implementation of VAE, and, how usually one does while implementing any architecture, I passed a random input to the model to test if everything worked fine (e.g. check ...
- 31
0
votes
0
answers
24
views
When to know if I am "on the right track" for a CNN architecture
Context
Very new to CNNs and ML in general. I am building a simple binary image segmentation network for generating black and white image masks (white pixels = desired object; black pixels = all else)....
0
votes
0
answers
24
views
What kind of learning architecture can I use to solve a set of nonlinear thermodynamic equations?
I am in the business of solving equations based in thermodynamics. Technically speaking, I am trying to solve for the binodals for a ternary system. The binodal is a curve in a phase diagram.
...
- 101
1
vote
2
answers
593
views
Tips and tricks when training a very large language model?
Have never trained a (very) large language model, so I am wondering if the process is the same as training a (regular) language model, i.e. you prepare the data, set up the architecture, ...
1
vote
1
answer
33
views
Practical differences between the different RL architectures
I have tried many different RL architectures: DQN, PPO, Policy optimization, and for my specific problem they all failed in their basic setup. Eventually I discovered that my problem had too sparse/...
- 71
0
votes
0
answers
24
views
How to change network that works on a subset of the input?
To clarify my question, I'll take an example: let's say you're trying to classify pictures, by determining whether there's a dog or not in the picture. Let's assume the pictures to be 1000x1000 pixels....
0
votes
1
answer
93
views
Which models can be applied recursively?
I come from a math background, so I am not up-to-date with machine learning literature.
For the purpose of learning dynamics, I would like to train a model to minimize the following loss:
$$\mathcal{L}...
- 101
0
votes
1
answer
38
views
Patterns binary classification - model doesn't overfit
I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
I have written a model with 3 dense ...
- 123
0
votes
2
answers
247
views
BYOL: Why is there a prediction network in the online network but not in the target network?
In the BYOL paper, the following architecture is presented:
Why is a prediction network added to the online network, which is not present in the target network?
How are the online and target compared ...
- 2,620
0
votes
1
answer
24
views
What architecture is used for deep quadruplet network for person re-identification
I am trying to implement the paper Beyond triplet loss: a deep quadruplet network for person re-identification. In the paper, they provide a figure (attached below) containing the network architecture,...
1
vote
1
answer
73
views
Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?
Considering the architecture of encoder and decoder in transformer as shown below:
Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific separate Feed forward ...
- 158
1
vote
1
answer
85
views
Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models
I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help ...
0
votes
1
answer
163
views
Do different architectures really make a difference or is it just a matter of the training process?
I was wondering which influence different architectures for deep learning truly have on the performance. Of course, substantial changes in the paradigms we use when building neural networks (such as ...
- 121
2
votes
0
answers
31
views
How can I learn about NN architecture?
I have a pretty good understanding of individual neural net layers (fully connected, convolution, pooling, activation, etc) but struggle to construct combinations of them to solve a given problem. I ...
- 21
0
votes
0
answers
53
views
Is ANN architecture mesh topology exist?
I'm just wondering if there's ANN architecture that looks like mesh topology at context of computer networking.
If exist or possible, is layer notion still applied?
2
votes
1
answer
63
views
Propagating gradients through an "Item Selector" network
Consider the following problem:
There are $N$ items and $S$ slots. Each item is a vector of length $D$.
The goal is to train a neural network to select one item per slot in order to minimize the loss ...
- 201
1
vote
1
answer
254
views
What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?
I was reading this link , and saw some creative architectures for PPO.
I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
- 362
1
vote
0
answers
27
views
How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?
Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
- 11
3
votes
1
answer
888
views
How do I design the network for Deep Q-Network?
I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.
I have a very simple environment, basically a 18x18 matrix, where 3 ...
- 133
2
votes
0
answers
1k
views
How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?
Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2.
Formally, my question is about their computational graph:...
- 39
0
votes
0
answers
61
views
How to learn a neural network with equivalent constraints on the weights
Let $f(x)$ be an output of a neural network with input $x$.
My data is a pair $(x,y)$ and my loss function is a function of $f(x)$ and $f(y)$, i.e., $g(f(x),f(y))$.
What kind of architecture enables ...
0
votes
0
answers
22
views
Is it possible that a deep neural network, with some variations, can be used for multiple tasks?
I am asking this question on deep neural network architectures only. If you want to restrict the domain of tasks then you can choose computer vision for this question.
Suppose there is an architecture ...
- 3,960
2
votes
0
answers
34
views
Is there any way to force one input have more effect on model?
Now I am working on building a deep learning model for a regression problem. I used 50 inputs and try to add one new categorical input. The problem is that this one input is much more important than ...
- 21
1
vote
1
answer
378
views
Are the output dimensions of the first and second convolutional layer in YOLO paper correct?
I was reading the last version of the YOLO paper available in Arxiv, and I don't fully understand the output dimensions (I understand width and height, but not depth) of the first and second ...
- 31
6
votes
3
answers
3k
views
Why do Transformers have a sequence limit at inference time?
As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
- 3,000
2
votes
1
answer
367
views
CNN Architectures for local features vs global context
Kaparthy in his blog post said
[this] hints at the kinds of architectures we’ll eventually explore. As an example - are very local features enough or do we need global context?
I'd like to gain ...
- 356
0
votes
1
answer
224
views
What is the difference between a vision transformer and image-based relational learning?
I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
- 670
0
votes
0
answers
36
views
Do deep learning researchers generally visualize intermediate steps?
Many researchers in deep learning research come up with new CNN architectures.
The architectures are (just) combinations of a few existing layers.
Along with their mathematical intuition, in general, ...
- 3,960
1
vote
1
answer
270
views
Where does batch normalization layers present in a neural network?
Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training.
Since we train a neural network by dividing training data ...
- 3,960
1
vote
0
answers
35
views
How to learn transition type in a 1-hour extended DJ Mix?
How would you design a model which learns the transitions in a given 1-hour DJ Mix? To be specific, the model should be able to learn transitions, specify the occurring time and the type (Crossfade, ...
- 11
1
vote
0
answers
46
views
Under what circumstances is a fully connected layer similar to PCA?
I am reading this paper on image retrieval where the goal is to train a network that produces highly discriminative descriptors (aka embeddings) for input images. If you are familiar with facial ...
- 1,369
0
votes
1
answer
126
views
What type of ANN architecture to choose?
I have $N$ number of teachers each of which has an input feature vector ($25$ dimensional) consisting of positive numerical values for different quality of aspects (for example: lecturing ability, ...
- 199
2
votes
1
answer
146
views
How to properly use Flatten layer?
Context
I'm trying to create net that will be able to recognize printed-like digits. Something like MNIST, but only for standard printing font.
Images are of the size 40x40 and I'd like to put them ...
- 242
4
votes
2
answers
248
views
Which neural network can I use to solve this constrained optimisation problem?
Let $\mathcal{S}$ be the training data set, where each input $u^i \in \mathcal{S}$ has $d$ features.
I want to design an ANN so that the cost function below is minimized (the sum of the square of ...
- 199
2
votes
1
answer
967
views
How can we get a differentiable neural network to count things?
Imagine I have images with apples in them. I want to train a neural network which can count the number of apples in each image.
BUT, I don't want to use a detector, then count the number of bounding ...
- 1,369
1
vote
0
answers
60
views
Are there regularisation methods related only to architecture of the CNNs?
Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the ...
- 1,280
3
votes
2
answers
3k
views
In classification, how does the number of classes affect the model size and amount of data needed to train?
When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?
Are there any soft or hard ...
4
votes
1
answer
547
views
What is a unified neural network model?
In many articles (for example, in the YOLO paper, this paper or this one), I see the term "unified" being used. I was wondering what the meaning of "unified" in this case is.
- 63
5
votes
2
answers
1k
views
What's the difference between architectures and backbones?
In the paper "ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery", the authors talk about using:
Feature Pyramid Networks (as the ...
2
votes
0
answers
989
views
What exactly are deep learning primitives?
I came across the concept of "deep learning primitives" from the Nvidia talk Jetson AGX Xavier New Era Autonomous Machines (on slide 44).
There doesn't seem to be a lot of articles in the ...
- 21
3
votes
1
answer
75
views
Are there deep neural networks that have inputs connected with deeper hidden layers?
Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)?
If so could you give some ...
- 1,280
0
votes
1
answer
88
views
How to use a NN for seq2seq tasks?
I am trying to make a NN(probably with dense layers) to map a specific input to a specific output (or basically sequence2sequence). I want the model to learn the relation between the sequences and ...
- 156
2
votes
3
answers
237
views
Why does a neuron in a multi-layer network need several input connections?
For example, if I have the following architecture:
Each neuron in the hidden layer has a connection from each one in the
input layer.
3 x 1 Input Matrix and a 4 x 3 weight matrix (for the ...
- 121
1
vote
0
answers
45
views
How can one be sure that a particular neural network architecture would work?
Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
1
vote
1
answer
46
views
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?
I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters).
I'm comparing my solution with some ...
- 171
0
votes
1
answer
93
views
Is the size of a neural network directly linked with an increase in its inteligence?
Just came across this article on GPT-3, and that lead me to the question:
In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?
Also, if that ...
- 191
2
votes
0
answers
475
views
Merge two different CNN models into one
I have 2 different models with each model doing a separate function and have been trained with different weights. Is there any way I can merge these two models to get a single model.
If it can be ...
2
votes
1
answer
188
views
Can you explain me this CNN architecture?
I am starting to get my head around convolutional neural networks, and I have been working with the CIFAR-10 dataset and some research papers that used it. In one of these papers, they mention a ...
- 141