4
$\begingroup$

I have a dataset which contains 4000k rows and 6 columns. The goal is to predict travel time demand of a taxi. I have read many articles regarding how to approach the problem. So, every writer tell his own way. The thing which I have concluded from all my readings is that I have to use multiple algorithms and check the accuracy of each one. Then I can ensemble them by averaging or any other approach.

Which algorithms will be best for my problem accuracy-wise? Some links to code will be helpful for me.

I currently only have training set of data. After I work on it, it will be evaluated on any testing set by my professor. So, what should I do now? Either split data I have into my own testing and training set or separately generate dummy data as a testing set?

$\endgroup$
1
$\begingroup$

In general, this type of problem is called a regression problem since the target variable (i.e. travel time) can take any value in a continuous domain. In theory, you can use any regression algorithms (a subset supervised learning techniques) to solve this problem. Some of the most popular ones are linear regression, K-nearest neighbor (regressor), and neural networks.

As you observed already, different algorithms result in (sometimes significantly) different results. Also, the parameter configurations (e.g., number of hidden layers in Neural Networks) can make a big difference. Sometimes, ensembling different models can be helpful, but in general, you should try to avoid overfitting (when your model is more complex than your data such that it memorizes the training set instead of learning it!). That may result in a very good performance on your training set but perform very poorly on your professor's testing set.

What I would do is:

  • exploring the dataset to see what are the contributing factor in travel time (any correlation between the columns).
  • cleaning and preprocessing my dataset (duplicates, null values, outliers)
  • reshaping my dataset if needed (normalizing some columns, merging or splitting columns)
  • dividing my dataset to training and evaluation subsets (so I train on one part and test on the other part to avoid overfitting)
  • choosing a simple baseline, applying and measuring the accuracy metrics.
  • trying to fine tune parameters of my baseline or trying other more advanced techniques.
  • comparing the results and improving any part of the pipeline when necessary (more/less cleaning, parameter tuning, ensembling).
$\endgroup$
  • 2
    $\begingroup$ Yes. LR and ANNs do work on approximation functions and there are some other ML algorithms which use different approach to solve a problem, so, if someone doesn't want to get biased to a single approach then he can use others and then average the effects of all algorithms to get an unbiased result. Although, ML is not accurate enough still it is the only way to solve such problems because such problems don't follow a common pattern plus they don't purely follow some function. So, we have to go with both things to get an unbiased result. The approach is purely a data scientist's choice. $\endgroup$ – Saeed Ahmad Jun 14 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.