Questions tagged [architecture]

For questions related to the architecture of AI models, e.g. the architecture of neural networks.

Filter by
Sorted by
Tagged with
0 votes
1 answer
39 views

Do different architectures really make a difference or is it just a matter of the training process?

I was wondering which influence different architectures for deep learning truly have on the performance. Of course, substantial changes in the paradigms we use when building neural networks (such as ...
  • 121
2 votes
0 answers
24 views

How can I learn about NN architecture?

I have a pretty good understanding of individual neural net layers (fully connected, convolution, pooling, activation, etc) but struggle to construct combinations of them to solve a given problem. I ...
0 votes
0 answers
19 views

Why would one still use a traditional GAN architecture or WGAN architecture instead of a WGAN-GP architecture?

I've been diving into the literature of GANs, and quite early on, I was pretty convinced that WGAN-GPs were the way to go. The WGAN-GP architecture is, as far as I know, theoretically and empirically ...
0 votes
0 answers
21 views

Is ANN architecture mesh topology exist?

I'm just wondering if there's ANN architecture that looks like mesh topology at context of computer networking. If exist or possible, is layer notion still applied?
2 votes
1 answer
49 views

Propagating gradients through an "Item Selector" network

Consider the following problem: There are $N$ items and $S$ slots. Each item is a vector of length $D$. The goal is to train a neural network to select one item per slot in order to minimize the loss ...
  • 191
0 votes
0 answers
47 views

How does backpropagation work with multi-branch models?

How does backpropagation work when the input layer feeds into two or more separate branches of layers before merging back to produce a single output, such as can be implemented in the Keras Functional ...
0 votes
0 answers
20 views

How state is combined with action in crtitic networks?

Actor-critic networks are present in deep reinforcement learning algorithms. Actor-network takes a state as input and gives action as output. Critic-network takes state and action as input and gives a ...
  • 3,451
1 vote
0 answers
27 views

What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?

I was reading this link , and saw some creative architectures for PPO. I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
0 votes
0 answers
11 views

Approach for predictive model being trained while running at the same time

I have only surface level knowledge in all things AI but am thinking about tackling a specific use case with it in the future. I would like to predict user input, specifically a resident doing stuff ...
  • 1
1 vote
0 answers
23 views

How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?

Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
3 votes
1 answer
137 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
  • 133
0 votes
0 answers
28 views

Where Can I Find Resources on Extracting Meaningful Content From Web Pages?

I am in the process of conducting a literature review for my thesis. Currently, I am struggling when it comes to developing a theoretical framework/methodology or to even correctly outline an approach ...
2 votes
0 answers
531 views

How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?

Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2. Formally, my question is about their computational graph:...
0 votes
0 answers
44 views

Transformers for regression on permutation of fixed size sequence?

Transformers have shown remarkable performance operating on sequences, but are equivariant to the order in the input sequence. Positional Encoding alleviates that problem, but how good is it? In my ...
  • 256
0 votes
0 answers
42 views

How to learn a neural network with equivalent constraints on the weights

Let $f(x)$ be an output of a neural network with input $x$. My data is a pair $(x,y)$ and my loss function is a function of $f(x)$ and $f(y)$, i.e., $g(f(x),f(y))$. What kind of architecture enables ...
0 votes
0 answers
32 views

Learning a quasi-convex function that passes the origin

I am trying to learn a function $y=f(x)$ through samples $(x_i,y_i)$, where $x$ is $n$-dimensional. We know that the function $f$ is quasi-convex and passes through the origin, i.e., $f(0)=0$. Is ...
0 votes
0 answers
19 views

Is it possible that a deep neural network, with some variations, can be used for multiple tasks?

I am asking this question on deep neural network architectures only. If you want to restrict the domain of tasks then you can choose computer vision for this question. Suppose there is an architecture ...
  • 3,451
2 votes
0 answers
28 views

Is there any way to force one input have more effect on model?

Now I am working on building a deep learning model for a regression problem. I used 50 inputs and try to add one new categorical input. The problem is that this one input is much more important than ...
0 votes
1 answer
185 views

Are the output dimensions of the first and second convolutional layer in YOLO paper correct?

I was reading the last version of the YOLO paper available in Arxiv, and I don't fully understand the output dimensions (I understand width and height, but not depth) of the first and second ...
5 votes
2 answers
889 views

Why do Transformers have a sequence limit at inference time?

As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
1 vote
1 answer
181 views

CNN Architectures for local features vs global context

Kaparthy in his blog post said [this] hints at the kinds of architectures we’ll eventually explore. As an example - are very local features enough or do we need global context? I'd like to gain ...
0 votes
1 answer
141 views

What is the difference between a vision transformer and image-based relational learning?

I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
0 votes
0 answers
33 views

Do deep learning researchers generally visualize intermediate steps?

Many researchers in deep learning research come up with new CNN architectures. The architectures are (just) combinations of a few existing layers. Along with their mathematical intuition, in general, ...
  • 3,451
1 vote
1 answer
120 views

Where does batch normalization layers present in a neural network?

Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training. Since we train a neural network by dividing training data ...
  • 3,451
1 vote
0 answers
30 views

How to learn transition type in a 1-hour extended DJ Mix?

How would you design a model which learns the transitions in a given 1-hour DJ Mix? To be specific, the model should be able to learn transitions, specify the occurring time and the type (Crossfade, ...
1 vote
0 answers
35 views

Under what circumstances is a fully connected layer similar to PCA?

I am reading this paper on image retrieval where the goal is to train a network that produces highly discriminative descriptors (aka embeddings) for input images. If you are familiar with facial ...
0 votes
1 answer
113 views

What type of ANN architecture to choose?

I have $N$ number of teachers each of which has an input feature vector ($25$ dimensional) consisting of positive numerical values for different quality of aspects (for example: lecturing ability, ...
2 votes
1 answer
64 views

How to properly use Flatten layer?

Context I'm trying to create net that will be able to recognize printed-like digits. Something like MNIST, but only for standard printing font. Images are of the size 40x40 and I'd like to put them ...
4 votes
1 answer
164 views

Which neural network can I use to solve this constrained optimisation problem?

Let $\mathcal{S}$ be the training data set, where each input $u^i \in \mathcal{S}$ has $d$ features. I want to design an ANN so that the cost function below is minimized (the sum of the square of ...
2 votes
1 answer
461 views

How can we get a differentiable neural network to count things?

Imagine I have images with apples in them. I want to train a neural network which can count the number of apples in each image. BUT, I don't want to use a detector, then count the number of bounding ...
1 vote
0 answers
56 views

Are there regularisation methods related only to architecture of the CNNs?

Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the ...
  • 1,220
2 votes
2 answers
1k views

In classification, how does the number of classes affect the model size and amount of data needed to train?

When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train? Are there any soft or hard ...
4 votes
1 answer
290 views

What is a unified neural network model?

In many articles (for example, in the YOLO paper, this paper or this one), I see the term "unified" being used. I was wondering what the meaning of "unified" in this case is.
5 votes
2 answers
547 views

What's the difference between architectures and backbones?

In the paper "ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery", the authors talk about using: Feature Pyramid Networks (as the ...
1 vote
0 answers
407 views

What exactly are deep learning primitives?

I came across the concept of "deep learning primitives" from the Nvidia talk Jetson AGX Xavier New Era Autonomous Machines (on slide 44). There doesn't seem to be a lot of articles in the ...
  • 11
3 votes
1 answer
58 views

Are there deep neural networks that have inputs connected with deeper hidden layers?

Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)? If so could you give some ...
  • 1,220
0 votes
1 answer
69 views

How to use a NN for seq2seq tasks?

I am trying to make a NN(probably with dense layers) to map a specific input to a specific output (or basically sequence2sequence). I want the model to learn the relation between the sequences and ...
  • 126
2 votes
3 answers
159 views

Why does a neuron in a multi-layer network need several input connections?

For example, if I have the following architecture: Each neuron in the hidden layer has a connection from each one in the input layer. 3 x 1 Input Matrix and a 4 x 3 weight matrix (for the ...
  • 21
1 vote
0 answers
42 views

How can one be sure that a particular neural network architecture would work?

Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
1 vote
1 answer
41 views

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters). I'm comparing my solution with some ...
  • 151
0 votes
1 answer
79 views

Is the size of a neural network directly linked with an increase in its inteligence?

Just came across this article on GPT-3, and that lead me to the question: In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger? Also, if that ...
2 votes
0 answers
359 views

Merge two different CNN models into one

I have 2 different models with each model doing a separate function and have been trained with different weights. Is there any way I can merge these two models to get a single model. If it can be ...
2 votes
1 answer
141 views

Can you explain me this CNN architecture?

I am starting to get my head around convolutional neural networks, and I have been working with the CIFAR-10 dataset and some research papers that used it. In one of these papers, they mention a ...
  • 141
9 votes
4 answers
3k views

Should neural nets be deeper the more complex the learning problem is?

I know it's not an exact science. But would you say that generally for more complicated tasks, deeper nets are required?
1 vote
4 answers
237 views

Should batch-normalization/dropout/activation-function layers be used after the last fully connected layer?

I am using the following architechture: 3*(fully connected -> batch normalization -> relu -> dropout) -> fully connected Should I add the ...
1 vote
1 answer
90 views

Why are denser layers needed in computer vision neural nets?

Many neural net architectures for computer vision tasks use several convolutional layers and then several fully-connected (or dense) layers. While the reasons for using convolutional layers are clear ...
0 votes
2 answers
181 views

RL: What should be the output of the NN for an agent trying to learn how to play a game?

Say the game is tic tac toe. I found two possible output layers: Vector of length 9: each float of the vector represents 1 action (one of the 9 boxes in Tic Tac Toe). The agent will play the ...
  • 275
2 votes
2 answers
66 views

Heavily mixing signal differentiation from Open Set of backgrounds via CNN

I am currently attempting to detect a signal from background noise. The signal is pretty well known but the background has a lot of variability. I've since come to know this problem as Open Set ...
4 votes
2 answers
124 views

Is a basic neural network architecture better with small datasets?

I'm currently trying to predict 1 output value with 52 input values. The problem is that I only have around 100 rows of data that I can use. Will I get more accurate results when I use a small ...
1 vote
0 answers
54 views

Using U-NET for image semantic segmentation

I'm getting literally crazy trying to understand how U-NET works. Maybe it is very easy, but I'm stuck (and I have a terrible headache). So, I need your help. I'm going to segment MRI to find white ...