How to encode both sentences and categorical data?

Question

I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual sentence with several pieces of information that outlines prices of something, delivery month, product name and sometimes some other info. That column is basically a text message and its format can vary.

Example DataFrame looks like this:

Name	User	Text	Target
sarah	noro	@-23.50 july 380/crx	CRX Laptop
john	simons	(atlas5) sep nc8 npc 131.5 @ 132.5	NPC Playstation
...	...	...	...

The columns Name, User, Text are features, and column Target is the target column that I want to predict. I would like to use a classifier (Random Forest, or Neural Net, or GBDT) to classify the Target based on other three columns.

I can category encode the columns Name, User, Target as I know how many of unique names, users, and targets there are.

1. How do I encode the Text column as it is a sentence?
2. Would it be better to regex split the Text column and simplify it (remove numbers etc) or just feed in the whole thing?

Answer 1

From the two example and the explanation that you provide I wouldn't say that the column text contains sentences, it looks more like gibberish that couldn't fit in any other column, and hence stuck for convenience in a unique generic one.

I think that here the best way is to perform a bit more data exploration, and try to split that feature into several categorical and numerical features. You mentioned already delivery month and price, those could be extract and become features themselves. And for all the extra stuff that remain I would just create a binary feature similar to "contains_extra_info" yes/no.