I have access to a GPU server with four gpus. Now I would like to train multiple models or folds of one model, one on each gpu. How can I schedule multiple trainings, so that a new training instance will be started, once a gpu becomes available?

I've seen that there is the Nvidia container toolkit, where I can assign a specific GPU at a time. With docker compose I would be able to start another container with the depends option- However, I envision a more automated solution. Something like a single node cluster, which automatically picks a scheduled training container, once another container that was blocking a gpu is finished. But so far, I only found (data) distributed training approaches.

Does anyone know about a way to accomplish this? Thanks in advance! Best, Eric



You must log in to answer this question.

Browse other questions tagged .