I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who only will buy, in second - only who won't buy.

The 1st dataset contains 500 rows and 2nd 10000 rows. What will be the right approach to merge it? How can I normalize them? And to point for an algorithm that the last column is the main sequence on what it should learn?


id    income date will_buy

23123 200    10.5 Yes

and second dataset:

id    income date will_buy

2323  100    10.5 No
  • 2
    $\begingroup$ Welcome to AI.SE! This question is not on topic here, because it concerns a task that is part of data science, not part of AI (in contrast, many other kinds of topics actually overlap). You will generally have better luck on DataScience.SE with this kind of question. That said, if the IDs are unique, you can just append one dataset to the other to merge them. You can normalize them with any number of standard tools, like Scikit Learn's Scaler, and then write the normalized data back out as a CSV file. $\endgroup$ – John Doucette Nov 15 '19 at 14:04
  • $\begingroup$ @JohnDoucette Thanks for you suggestion! $\endgroup$ – 干猕猴桃 Nov 15 '19 at 15:12
  • $\begingroup$ Apologies. Closed as off-topic (but I'm glad you got the information you were looking for!) $\endgroup$ – DukeZhou Nov 15 '19 at 20:37

You can use append function:

final = df1.append(df2, ignore_index=True)

To set the last column as labels, you set them as so by:

labels = np.array(final["will_buy"])

So, when calling the fit method on the model you build, you set labels = labels.

| improve this answer | |
  • $\begingroup$ Thanks for suggestion! And do you think it's a good idea to take only 500 rows from the 2nd dataset so merged will have normalized distribution of 2 labels? $\endgroup$ – 干猕猴桃 Nov 15 '19 at 12:05
  • $\begingroup$ Oh you mean to have equal number of rows each type? Yes but if the data frames are ordered in some manner, I would shuffle first. After that you can select top 500 rows by, for example df1_500 = df1.head(500) $\endgroup$ – serali Nov 15 '19 at 12:11
  • $\begingroup$ @干猕猴桃 Maybe. Whether that is a good idea will depend on what sort of task you plan to use this dataset for later on. $\endgroup$ – John Doucette Nov 15 '19 at 15:34
  • $\begingroup$ @JohnDoucette basically sort of binary classification, I need to algorithm to predict whether user buy product or not, just 2 states $\endgroup$ – 干猕猴桃 Nov 15 '19 at 17:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.