I recently trained Kaggles "Advanced Housing Prices"-Competition using Catboost. For training i used a compute-instance on Google Cloud Platform (GCP) (CPU: Xeon Quad-Core, RAM: 15GB, GPU: Tesla V100) and trained the model using the CPU first, then on the GPU.
reg = CatBoostRegressor(iterations=4000, learning_rate=0.06) reg.fit(X, y, cat_features=categorical, verbose=400)
0: learn: 76301.1754682 total: 85.8ms remaining: 5m 43s 400: learn: 12187.9081510 total: 6.74s remaining: 1m 800: learn: 8249.9456584 total: 13.5s remaining: 53.8s 1200: learn: 5816.9481262 total: 20.2s remaining: 47.1s 1600: learn: 4371.4781446 total: 26.8s remaining: 40.2s 2000: learn: 3331.5702252 total: 33.4s remaining: 33.4s 2400: learn: 2600.9734000 total: 40s remaining: 26.6s 2800: learn: 2066.7154652 total: 46.5s remaining: 19.9s 3200: learn: 1680.6404763 total: 53.2s remaining: 13.3s 3600: learn: 1373.1095875 total: 59.9s remaining: 6.64s 3999: learn: 1146.4845213 total: 1m 6s remaining: 0us
reg_gpu = CatBoostRegressor(iterations=4000, learning_rate=0.04, loss_function='RMSE', task_type='GPU') reg_gpu.fit(X, y, cat_features=categorical, verbose=400)
0: learn: 78013.3893895 total: 39.9ms remaining: 2m 39s 400: learn: 24004.7793662 total: 15.1s remaining: 2m 15s 800: learn: 21142.6899404 total: 29.9s remaining: 1m 59s 1200: learn: 19752.2511301 total: 44s remaining: 1m 42s 1600: learn: 18967.4797535 total: 58.3s remaining: 1m 27s 2000: learn: 18201.5697225 total: 1m 12s remaining: 1m 12s 2400: learn: 17599.8105474 total: 1m 26s remaining: 57.6s 2800: learn: 17162.3720952 total: 1m 40s remaining: 43s 3200: learn: 16706.0578195 total: 1m 55s remaining: 28.7s 3600: learn: 16351.7230276 total: 2m 9s remaining: 14.3s 3999: learn: 16073.2521291 total: 2m 23s remaining: 0us
I experienced a learning slowdown of about 5x compared to training on the CPU. I gave the same notebook to a colleague to train on his machine (CPU: Ryzen 1700x, RAM: 16GB, GPU: GTX1080) and he experienced a learning-speedup of about 30%.
I know, that according to this issue GPU-training on small datasets should always be relatively slow and since the mentioned dataset is a table of 1460x82 values it is in fact a relatively small dataset. However, this doesn't explain why my colleague experienced such a speedup.
Is the GPU shared when using GCP? How can i accelerate GPU-training on GCP?