0
$\begingroup$

I recently trained Kaggles "Advanced Housing Prices"-Competition using Catboost. For training i used a compute-instance on Google Cloud Platform (GCP) (CPU: Xeon Quad-Core, RAM: 15GB, GPU: Tesla V100) and trained the model using the CPU first, then on the GPU.

reg = CatBoostRegressor(iterations=4000, learning_rate=0.06)
reg.fit(X, y, cat_features=categorical, verbose=400)
0:  learn: 76301.1754682    total: 85.8ms   remaining: 5m 43s
400:    learn: 12187.9081510    total: 6.74s    remaining: 1m
800:    learn: 8249.9456584 total: 13.5s    remaining: 53.8s
1200:   learn: 5816.9481262 total: 20.2s    remaining: 47.1s
1600:   learn: 4371.4781446 total: 26.8s    remaining: 40.2s
2000:   learn: 3331.5702252 total: 33.4s    remaining: 33.4s
2400:   learn: 2600.9734000 total: 40s  remaining: 26.6s
2800:   learn: 2066.7154652 total: 46.5s    remaining: 19.9s
3200:   learn: 1680.6404763 total: 53.2s    remaining: 13.3s
3600:   learn: 1373.1095875 total: 59.9s    remaining: 6.64s
3999:   learn: 1146.4845213 total: 1m 6s    remaining: 0us
reg_gpu = CatBoostRegressor(iterations=4000, learning_rate=0.04, loss_function='RMSE', task_type='GPU')
reg_gpu.fit(X, y, cat_features=categorical, verbose=400)
0:  learn: 78013.3893895    total: 39.9ms   remaining: 2m 39s
400:    learn: 24004.7793662    total: 15.1s    remaining: 2m 15s
800:    learn: 21142.6899404    total: 29.9s    remaining: 1m 59s
1200:   learn: 19752.2511301    total: 44s  remaining: 1m 42s
1600:   learn: 18967.4797535    total: 58.3s    remaining: 1m 27s
2000:   learn: 18201.5697225    total: 1m 12s   remaining: 1m 12s
2400:   learn: 17599.8105474    total: 1m 26s   remaining: 57.6s
2800:   learn: 17162.3720952    total: 1m 40s   remaining: 43s
3200:   learn: 16706.0578195    total: 1m 55s   remaining: 28.7s
3600:   learn: 16351.7230276    total: 2m 9s    remaining: 14.3s
3999:   learn: 16073.2521291    total: 2m 23s   remaining: 0us

I experienced a learning slowdown of about 5x compared to training on the CPU. I gave the same notebook to a colleague to train on his machine (CPU: Ryzen 1700x, RAM: 16GB, GPU: GTX1080) and he experienced a learning-speedup of about 30%.

I know, that according to this issue GPU-training on small datasets should always be relatively slow and since the mentioned dataset is a table of 1460x82 values it is in fact a relatively small dataset. However, this doesn't explain why my colleague experienced such a speedup.

Is the GPU shared when using GCP? How can i accelerate GPU-training on GCP?

$\endgroup$
  • $\begingroup$ I can't know for sure, but my guess would be that the gpu is being shared or there might be a bottleneck when using a Tesla, as I believe Teslas are a little different to a consumer GPU (such as a GTX1080). It might also be useful to time the google compute engine CPU and GPU vs your friends CPU and GPU. It might be given the very powerful CPU it's able to surpass the GPU due to the time taken for the data to be split among many cores in the GPU. $\endgroup$ – Recessive Jul 16 at 3:42
  • $\begingroup$ Ok, so in this case the overhead of the Tesla-GPU would be bigger than its advantages? This seams counter intuitive to me, since Tesla-GPUs are labeled as "Deep Learning Accelerators". $\endgroup$ – Tim Hilt Jul 16 at 7:14
  • $\begingroup$ I believe that AI StackExchange is not the right place for such questions. Tho, Moderators may move it to some other SE, which will be more helpful to you. $\endgroup$ – Azmuth Jul 16 at 16:42
  • $\begingroup$ Can you recommend a better community? $\endgroup$ – Tim Hilt Jul 16 at 16:44
  • $\begingroup$ This question seems to be off-topic here. Please, see ai.stackexchange.com/help/on-topic. A more appropriate site to ask hardware questions might be Data Science SE. $\endgroup$ – nbro Jul 17 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.