Questions tagged [pytorch]

For conceptual questions that somehow involve the PyTorch library, but note that programming questions are off-topic here.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
26 views

Diffusion model for image to image translation

I am interested in using the diffusion model for image-to-image translation (pair images). I used this repository for semantic image synthesis via diffusion models Code. How can I use it for image-to-...
yun dan's user avatar
1 vote
1 answer
49 views

Is there any advantage to providing multi-dimensional input to torch modules?

Most layer types in torch.nn such as torch.nn.Linear accept input with more than one dimension. Is there any advantage in doing so if you can shape your data to represent a certain arrangement in ...
kot's user avatar
  • 13
0 votes
0 answers
14 views

Keeping the grad of the tensor when using inverse_transform

To train a network, I scaled both the input and outputs of my data like the following: ...
jasw's user avatar
  • 1
1 vote
0 answers
14 views

Integrated gradients on text to text models

I am trying to apply integrated gradients (using library captum) on a text generation model. Specifically, it is a model that generates patches for input buggy code. I want to know if applying the ...
Nimantha's user avatar
  • 111
0 votes
0 answers
48 views

ST-GCN: graph convolution operator in Geometry-Aware Interaction Network (GAIN)

I need help implementing the model in this paper: They have adopted spatio-temporal graph convolution operator in ST-GCN [section 3.1.2]. I've found there is popular libraries available for GCN: ...
Kholdarbekov's user avatar
0 votes
1 answer
23 views

Multiple Loss Functions For Proper Parameter Updates [closed]

I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, ic_loss, res_loss, ...
Burak Karaosmanoğlu's user avatar
1 vote
0 answers
244 views

Training/Fine Tuning LLaVa

We wanted to fine-tune LLaVa model on some custom set of images. I wanted to know the Dataset format required for training and then finetuning.
Anonymous Atom's user avatar
1 vote
0 answers
73 views

Diffusion Model Failing to Learn

I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
Karl's user avatar
  • 206
0 votes
1 answer
70 views

How to force Transformer to give more weight to certain tokens

I'm developing an encoder-decoder based transformer model and I would like to ask if there are ways to incentivize or penalize certain tokens during training. I'm working on a translation task where ...
jasperagrante's user avatar
-1 votes
1 answer
102 views

What is GNN Cheatsheet in PyG Docs [closed]

I am going through the Pytorch Geometric documentation: https://pytorch-geometric.readthedocs.io/en/latest/index.html which is built on Pytorch .Here they mentioned about GNN Cheatsheet: https://...
sripathi akhil's user avatar
0 votes
2 answers
317 views

I'm trying to understand the use model for different Python libraries

I'm new to ML/AI field, and after completing several free university courses from MIT OpenCourseWare and Harvard CS50, I've gained some familiarity with the theoretical foundations of Artificial ...
Boris L.'s user avatar
  • 109
0 votes
0 answers
23 views

Machine learning network design for 2D point prediction labeling

I'm looking to develop a basic neural network for labeling a detected set of 2D points. I've generated a random set of 3D points within a certain radius, all of which lie on a flat plane (Y value is 0)...
colyton's user avatar
0 votes
1 answer
52 views

Why Does the Model not Improve in PyTorch?

I have a simple curve fitting problem in hand. I wrote some code in PyTorch as follows: ...
Burak Karaosmanoğlu's user avatar
0 votes
0 answers
28 views

Understand Memory Usage of Pytorch Tensors for Inference

I plan on translating large text corpora from various languages to english with Large Language Models. Therefore, I tried messing around a bit to see the computational limits of my machine. ...
Max's user avatar
  • 1
0 votes
0 answers
16 views

Does splitting a model across multiple GPUs (using model parallelism) reduce model accuracy?

When training a model in PyTorch on multiple GPUs, it should ideally give the same results as using just one GPU. However, are there any specific things to be careful about during implementation?
willtryagain's user avatar
2 votes
2 answers
57 views

Is it possible to reconstruct convolutional layers' input using transposed convolution?

I've been trying to visualize internal activations in CNN and came across this paper: "Visualizing and Understanding Convolutional Networks" by Zeiler & Fergus. In the paper they ...
Shawn Li's user avatar
  • 143
2 votes
1 answer
78 views

Filling replay buffer with expert trajectories for PPO/DQN

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
Erik Storm's user avatar
1 vote
1 answer
69 views

How to transform a loss function into a score function?

Loss_Function/Maximize_Function/Score_Function, CustomLoss, pytorch. Using Custom Loss for Maximizing Score in PyTorch I'm using a PyTorch model with an LSTM input layer, a linear hidden layer, and 3 ...
IAQuestions's user avatar
0 votes
0 answers
60 views

Why does each row of data have the same bottleneck features in the Autoencoder after training?

I was training an autoencoder for anomaly detection and I wish to extract the bottleneck features of the encoder for K-NN. The model architecture is as such: ...
Aengus's user avatar
  • 1
0 votes
0 answers
38 views

Backpropagation in a transformer

I have a transformer for timeseries forcasting based on this article https://arxiv.org/abs/2001.08317 Given a source containing $src=(x_{t-5},x_{t-4},x_{t-3},x_{t-2},x_{t-1})$ and a target of $tgt=(x_{...
Michał Kuczynski's user avatar
1 vote
2 answers
47 views

How to Represent Boardless Board Game as Input to RL Model?

I am currently doing my thesis project by creating an Imitation Learning (IL) agent that learns to play the board game Hive, which lacks a traditional 2D board. Pieces are placed relative to one ...
Johnny McKenzie's user avatar
2 votes
2 answers
2k views

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use? Many papers use Left Padding, but is right padding wrong since transformers gives the following ...
basujindal's user avatar
1 vote
1 answer
80 views

How to overcome symmetry in the solution space when learning a simple neural network?

What are good solution recipes to overcome the problem that in neural network learning, when the solution space has symmetries, learning may eventually stall due to the sum of the gradients over the ...
DCTLib's user avatar
  • 61
0 votes
0 answers
17 views

Analysis of the output samples from an autoencoder

I am conducting some experiments on an autoencoder as part of my research project. For our first experiment, we have a feedforward neural network (using pytorch), which is being given an input of ...
Darth_Vader's user avatar
1 vote
1 answer
145 views

Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?

I have been trying to implement a Transformer architecture using PyTorch by following the Attention Is All You Need paper as well as the The Annotated Transformer blog post to compare my code with ...
Daviiid's user avatar
  • 573
0 votes
1 answer
193 views

How to do backpropagation with argmax?

I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
Subrat Prasad's user avatar
1 vote
0 answers
506 views

Relation between Batch Size and Micro Batch Size

In distributed training of large models (pipeline parallelism), a mini batch of training samples is divided into n-micro batches. Each device performs forward and backward passes for a micro batch. ...
Rituraj Singh's user avatar
0 votes
0 answers
34 views

How to calculate CIoU or DIoU loss only for certain unmasked boxes in tensor and ignore the masked values?

...
Amish Agrawal's user avatar
0 votes
1 answer
92 views

Why is the output of my graph neural network not permutation equivariant?

I am using Pytorch to train a graph neural network on a 4x4 graph. Each node has one feature, and the output has one feature. Essentially, the architecture of my GNN looks like this (I'm training the ...
Acad's user avatar
  • 111
0 votes
1 answer
63 views

What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file [closed]

I'm building a feature for my application that requires reading the properties of a saved ML model file (after it's trained). However, as I am pretty new to this field, I don't really understand the ...
Ryan Wang's user avatar
  • 113
1 vote
1 answer
96 views

Is it possible to write/build an AI model without using Frameworks? [closed]

I'm a relatively newbie in this world of Artificial Intelligence, although I am able to use frameworks such as Tensorflow and also understand the general concepts behind training weights and ...
Ryan Wang's user avatar
  • 113
0 votes
0 answers
10 views

Early binary classification of timeseries

I'm trying to figure out how to solve this problem that I'll try to explain in the next few lines. I have a timeseries of length ~200k values and every 700 points I have a label that indicates the ...
irazza's user avatar
  • 1
0 votes
1 answer
34 views

How to handle BatchNorm in the last layers of Neural Networks?

I am creating a neural network using batchnorm as a regularization method to enable deep models and prevent overfitting. I understand that batchnorming supresses the internal covariance shift ...
Quantum's user avatar
  • 121
0 votes
0 answers
56 views

How to Change the Length of a Sequence Using Linear Projection In PyTorch?

I was trying to reproduce a voice conversion model using PyTorch and I had difficulty implementing a module called PBTC (Parallel Bank of Transposed Convlutions), used in https://arxiv.org/pdf/2303....
Sophiefy's user avatar
0 votes
0 answers
39 views

Not pre-trained binary transformer model

I stacked with a problem. My default Transformer model totally does not learn how to evaluate python logical expressions, like: '(False and not True) xor False or (not False and False)'. Model should ...
Oleksandr Ovcharenko's user avatar
0 votes
0 answers
260 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
0 votes
1 answer
72 views

Do batches need to be sequential in Transformer traning?

When training a transformer model (I'm using nn.TransformerEncoder from pytorch) is it better to use sequential batches (for example, three sequences ...
Evgenii's user avatar
  • 103
0 votes
1 answer
21 views

Loss function not able to capture the maxima of probability distribution

I am trying to predict noise (random gaussian) with the help of a neural network. I am implementing a L2 loss (torch.nn.function.mse_loss) for computing the loss function between the prediction ...
Formal_this's user avatar
0 votes
0 answers
33 views

NUMA effects on single-process inference

I've been reading this article, which explains some of the effects of a multi-cpu, hyperthreaded system and suggests ways to get around the issues. Namely: put each problem domain on its own socket; ...
aphid's user avatar
  • 101
0 votes
0 answers
38 views

Reproducing Knowledge Distillation on MNIST data

I'm trying to implement Knowledge Distillation, specifically to reproduce the MNIST example given in the paper. My (PyTorch) implementation can be found here. I would expect it is very self-evident ...
Maverick Meerkat's user avatar
1 vote
0 answers
108 views

How can an MLP be implemented with convolutional layers?

I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here): It is strange to find ...
Jacob Morales Gonzalez's user avatar
1 vote
0 answers
52 views

Why would increasing layers in PyTorch Transformer significantly increase loss?

I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
Matt Harrison's user avatar
1 vote
1 answer
79 views

Fluctuations in loss during in epoch evaluation of GRU

I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and ...
PatrickSVM's user avatar
1 vote
1 answer
166 views

Playing around with Transformer - accuracy not improving [closed]

I am playing around with a decoder only transformer model, The Colab is here if you find that easier https://colab.research.google.com/drive/1SHyJ9Oa3E4j1x8YFlXQbd1mjUjWhHGOV#scrollTo=60e13119 or see ...
JB1's user avatar
  • 11
1 vote
0 answers
31 views

dropout as the final layer or in every layer to avoid underfitting/overfitting

I am training a Dense neural network where I am having input as a 3x3 matrix, and predicting the eigenvalues of that matrix. Initially, I was having num_samples = 2000, so my model was not able to ...
Rohit Singh's user avatar
0 votes
1 answer
108 views

Can I implement a sklearn model inside a Pytorch nn.Module? [closed]

I am making a custom Pytorch model that at some point, clusters a latent space that was created by another, previous routine of the model (Autoencoder). In a bit more detail, my model is a regular ...
puradrogasincortar's user avatar
0 votes
1 answer
86 views

Does it make sense to store information in a variable defined inside a Pytorch nn.Module?

I have a pytorch model (custom model inherited from nn.Module). I'm developing some architecture, for which makes sense for my task to have a list defined in the model as: ...
puradrogasincortar's user avatar
2 votes
2 answers
496 views

How do we determine the slope for leakyrelu activation function?

I am using LeakyReLU activation function in my architecture. We know that the default slope value is 1e-2. I want to understand ...
Rohit Singh's user avatar
1 vote
0 answers
71 views

Periodical fluctuations in loss curves

I am training a neural network (specifically a GRU based architecture but I think this is not too relevant for the question). My loss curves, especially the training loss but also the validation loss, ...
PatrickSVM's user avatar
0 votes
3 answers
207 views

2D convolution with channels versus 3D convolution for layers of a map?

Introduction I am considering to use a convolutional neural network in implementing Monte Carlo control with function approximation. I am using a Monte Carlo estimate as it is unbiased and has nice ...
Dylan Solms's user avatar

1
2 3 4 5