Questions tagged [gpu]
Graphics processing units or GPUs are specialized hardware for the manipulation of images and calculation of local image properties.
43 questions
0
votes
0
answers
34
views
How to solve the issue with getting free ports in Pytorch DDP?
I am facing issues with getting a free port in the DDP setup block of PyTorch for parallelizing my deep learning training job across multiple GPUs on a Linux HPC cluster.
I am trying to submit a deep ...
-1
votes
2
answers
44
views
Why gpu works faster for float number rather then zeros?
It seems no difference for cpu but not for gpu
- 39
0
votes
0
answers
19
views
Efficient Net V2 M ONNX model infers significantly slower on small input
When I convert an Efficient net v2 m model from Pytorch to Onnx on differently sized inputs, I notice a strange and unexplained behavior. I was hoping to find an explanation to my observations from ...
0
votes
0
answers
23
views
Moving tensor to cuda reduces precision
I am trying to move a tensor to cuda, but unfortunately, I am loosing precision. The first entry is 5.02e-5 which is converted to 0. Is there a work around?
...
- 1
0
votes
1
answer
45
views
Do any Machine Learning Models Use Hardware Differently? [closed]
My understanding is that most if not all of the current models are running on top of basically the same hardware operations.
That is to say, typically library calls to pre defined gpu routines that ...
0
votes
0
answers
106
views
Llama 3.2 Vision-Instruct Inference Speed on A100 or H100 GPU
Can anyone provide an estimated time of how long does it take for Llama-3.2 Vision-Instruct 11-B model to:
process an image size of 1-MB and prompt size of 1000 words and
generate a response of 500 ...
- 1
1
vote
0
answers
53
views
Memory consumption issues during the validation phase/loop [closed]
Context:
I am trying to fine-tune codet5-base model for a use-case on AWS's g5.2xlarge instance, and the following were my ...
0
votes
0
answers
60
views
How is the complexity of the chunked attention computation in "Self Attention Does Not Need O(n2) Memory" independent from the query chunks size?
In Self-attention Does Not Need O(n{2}) Memory the authors present a say to have a constant memory complexity attention algorithm that is sequential in nature and also present an implementation that ...
- 575
1
vote
1
answer
1k
views
What is accelerated years in describing the amount of the training time?
As described in this article, it was written that GPT-3 took 405 V100 years to train in 2020.
I'm a bit confused about this definition, does that mean the process was accelerated like using a V100 GPU ...
-2
votes
1
answer
110
views
How to run our python scripts utilizing our device's GPU? [closed]
My laptop has NVIDIA GeForce GTX1650 GPU. I want to utilize this GPU to run my Python script. Any help in the form of code would be really helpful. I mean tried researching this so much but I couldn't ...
- 1
2
votes
1
answer
2k
views
Has anyone tried to use llama.cpp with NVLink?
Apparantly its possible to pool the memory of two 3090 using NVLink (although not with 4090). This would make it possible to run large LLM's on consumer hardware.
https://huggingface.co/transformers/...
- 135
1
vote
0
answers
95
views
Inference process and flow, and role of GPU, CPU, and RAM [closed]
This is a noob question.
I load a HuggingFace transformer model into GPU and create a HuggingFace pipeline using that model. Then I run inference on the model using the pipeline.
I would be glad to ...
- 191
1
vote
0
answers
1k
views
Relation between Batch Size and Micro Batch Size
In distributed training of large models (pipeline parallelism), a mini batch of training samples is divided into n-micro batches. Each device performs forward and backward passes for a micro batch.
...
- 111
1
vote
1
answer
710
views
How are gaming GPUs different from GPUs for ML? [closed]
I have a seemingly innocent question. I was looking to buy a computer for deep learning development using large language models. When I searched online on consumer e-commerce websites, I could not ...
- 304
5
votes
3
answers
6k
views
For an LLM model, how can I estimate its memory requirements based on storage usage?
It is easy to see the amount of disk space consumed by an LLM model (downloaded from huggingface, for instance). Just go in the relevant directory and check the file sizes.
How can I estimate the ...
- 191
0
votes
0
answers
3k
views
How important is to fix this common Tensorflow warning?
I have seen some people with the same problem, this is the warning:
...
user57690
2
votes
0
answers
99
views
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
I am interested in detecting small structures on images which are themselves many GB in size. ...
- 121
0
votes
0
answers
676
views
Do tensor cores provide advantages for running Stable Diffusion or only for training?
If I am only interested in running Stable Diffusion, using pre-trained weights, to generate images, are there any advantages to using a GPU with more Tensor cores? Or will any CUDA-compatible GPU ...
- 101
0
votes
1
answer
115
views
How to speed up my neural network?
I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am ...
0
votes
2
answers
6k
views
How "exactly" are AI-accelerator chip ASICs built differently than GPUs as GPU seem to lead for many AI workloads on performance
There is a lot of discussion on google search about AI-custom-accelerators (like Intel's Gaudi) and GPUs.
Almost all of them say generic things like, a) AI Accelerator chip is for specialized AI ...
- 181
0
votes
1
answer
98
views
Pretrain large model on single GPU [closed]
i want to pretrain some model on P100 which is provided by kaggle. Pretraining on 3 A100 is about 1.5 day. I have 2 questions:
Can I put the same seed everywhere so that the results match, train the ...
1
vote
1
answer
89
views
Why training the same model on the same data can be slower on better card?
Can someone explain why training CNN model (in my case DenseNet201) on the same data, and the same data processing pipeline can be slower on better GPU (RTX3090) than worse one (RTX3060), with the ...
- 1,280
0
votes
1
answer
37
views
What kind of neural network and GPU should I use to classify images into > 10 000 classes?
I am trying to developp an image classifier that would have more than 10 000 classes but I don't know what kind of neural network I should use ?
Some Other questions arise from this one :
How big ...
-1
votes
1
answer
2k
views
Choosing proper graphic card for deep learning AND gaming [closed]
Though the combination between gaming and deep learning might not sound "serious" enough still this is the case - on the one hand, I need a great GPU for my son to play games, and it will be ...
- 303
0
votes
0
answers
2k
views
Tensorflow-gpu and multiprocessing
I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds ...
-1
votes
1
answer
1k
views
Should I spend money on a machine-learning capable PC or just use Google CoLab? [closed]
Assuming I have internet access, should I spend money on a PC or just use Google Colab?
I'll be doing deep-learning training.
Google CoLab: https://colab.research.google.com/
0
votes
2
answers
459
views
Is it true that batch size of form $2^k$ gives better results?
I am confused among the following in selecting the batch size for my model.
#1: powers of 2
I generally see that batch sizes are in powers of two: 32, 64, 128, 256.
#2: maximum GPU
Suppose my GPU ...
- 3,990
2
votes
0
answers
433
views
How to train neural networks with multiprocessing?
I am trying to figure out how multiprocessing works in neural networks.
In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is ...
- 121
1
vote
0
answers
59
views
How can traditional edge detection algorithms be implemented on a GPU?
How can edge detection algorithms, which are not based on deep learning, such as the canny edge detector, be implemented on a GPU? For example, how are non-edge pixels removed from an image once it ...
- 111
1
vote
2
answers
16k
views
How much computing power does it cost to run GPT-3? [closed]
I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, ...
- 121
2
votes
1
answer
157
views
How do GPUs faciliate the training of a Deep Learning Architecture?
I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.
To my understanding, GPUs help in performing independent tasks simultaneously to ...
8
votes
2
answers
10k
views
Effect of batch size and number of GPUs on model accuracy
I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation.
Here are my GPU and batch size configurations
use ...
- 239
-1
votes
0
answers
30
views
What are the properties of a model that is well suited for for high performance real-time inference
What are general best practices or considerations in designing a model that is optimized for real-time inference?
3
votes
0
answers
29
views
Good model and training algorithm to store texture data for fast gpu inference
Now, the following may sound silly, but I want to do it for my better understanding of performance and implementation of GPU inference for a set of deep learning problems.
What I want to do is to ...
4
votes
2
answers
886
views
How does a transformer leverage the GPU to be trained faster than RNNs?
How does a transformer leverage the GPU to be trained faster than RNNs?
I understand the parameter space of the transformer might be significantly larger than that of the RNN. But why does the ...
- 151
3
votes
1
answer
219
views
Training network with 4 GPUs performance is not exactly 4 times over one GPU why? [closed]
Training neural network with 4 GPUs using pyTorch, performance is not even 2 times (btw 1 & 2 times) compare to using one GPU. From Nvidia-smi we see GPU usage is for few milliseconds and next 5-...
- 83
3
votes
1
answer
3k
views
How can I reduce the GPU memory usage with large images?
I am trying to train a CNN-LSTM model. The size of my images is 640x640. I have a GTX 1080 ti 11GB. I am using Keras with the TensorFlow backend.
Here is the model.
...
- 33
9
votes
3
answers
7k
views
Is a GPU always faster than a CPU for training neural networks?
Currently, I am working on a few projects that use feedforward neural networks for regression and classification of simple tabular data. I have noticed that training a neural network using TensorFlow-...
- 1,280
2
votes
0
answers
28
views
In addition to matrix algebra, can GPU's also handle the various Kernel functions for Neural Networks?
I've read a number of articles on how GPUs can speed up matrix algebra calculations, but I'm wondering how calculations are performed when one uses various kernel functions in a neural network.
If ...
- 131
1
vote
0
answers
50
views
Any good resources for learning programming GPU level operations? [closed]
I want to be able to improve my lower level device specific programming abilities to assist in future endeavors. Examples would be learning to write custom tensorflow operations in C++ optimized to ...
- 2,387
1
vote
1
answer
616
views
Can I calculate the training performance of GPUs by comparing their specification? [closed]
I am currently using Nvidia GTX1050 with 640 CUDA cores and 2GB GDDR5 for Deep Neural Network training. I want to buy a new GPU for training, but I am not sure how much performance improvement I can ...
- 53
1
vote
2
answers
6k
views
Does fp32 & fp64 performance of GPU affect deep learning model training? [closed]
I am purchasing Titan RTX GPU. Everything seems fine with that except float32 & float64 performance which seems lower vis-a-vis some of its counter parts. I wanted to understand if single ...
- 325
8
votes
2
answers
22k
views
Can LSTM neural networks be sped up by a GPU?
I am training LSTM neural networks with Keras on a small mobile GPU. The speed on the GPU is slower than on the CPU. I found some articles that say that it is hard to train LSTMs (and, in general, ...
- 289