2
$\begingroup$

I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing.

In all of those Udemy's lessons, people always use a perfect dataset and ready to input in a model. So all you have to do is run it.

How do I know if my dataset is ready for a model? What do I have to do to make it ready? Which evaluations?

I had a few statistics classes in college already and I learned a lot about correlations matrices, autocorrelation functions, and its lags, etc. and I didn't see yet in anywhere someone explaining how can I evaluate my data and then proceed to implement a model to solve my problem.

If anyone could point me a direction, give me some material, show me where I can learn this, anything, it would be really helpful!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Before jumping to modeling, there are a few tasks a data scientist (or ML/AI practitioner) must do:

  1. Ideation (or hypothesizing): Before applying any modeling approach, we need to ask the right questions. We must clearly mention our assumptions and declare how we want to measure the effectiveness of the pipeline. Note that, some tools/algorithms might not fit to the made assumptions or may not lead to the best values in the defined metrics. So, the pipeline must be designed in a way it serves the purpose of answering the defined questions.
  2. Data Cleansing: Real-world data sets are usually not clean. They have all sorts of data issues such as missing value, duplicates, outliers, wrong measurements, fragments, inconsistency, etc. Most of the ML techniques are sensitive (to different extents, of course) to such issues. Therefore, the data should be cleaned before any modeling can be done.
  3. Data Wrangling (or Feature Engineering): In many cases, the gathered data (even cleaned) is not immediately suitable for any modeling/analysis. For example, we may need to convert the documents of a text corpus to vectors of numbers (via TF-IDF or Embedding techniques) before being able to apply a text classifier simply because our classifier only takes numeric data. Converting measurements to other units, breaking addresses to their components, converting times and dates to different formats or timezones are just a few examples of data wrangling tasks (in a broader context, feature engineering may also refer to dimension reduction or feature selection/projection).
  4. Exploratory Data Analysis (EDA): To do the cleansing, understanding the data set features, and ideation, we often need to explore the given data set using visual (e.g., dashboards and diagrams) or summary statistics tools.

Disclaimer: I have no business interest with Udemy. The links are just shared because @pedro-de-sá mentioned they take some courses from Udemy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I would add that data cleansing and feature engineering are often part of the data preprocessing phase. $\endgroup$
    – nbro
    Commented Jun 27, 2019 at 11:28
  • $\begingroup$ Thank you so much, @Borhan Kazimipour! Also if you have any books, articles or anything else to indicate me, works just fine too! It doesn't necessarily need to be Udemy's courses. Thanks once again! $\endgroup$
    – Pedro de Sá
    Commented Jun 28, 2019 at 11:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .