4
$\begingroup$

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.

For example, the text "hello" would be represented like

{h: 1, e: 0, l: 0, o: 0}
{h: 0, e: 1, l: 0, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 0, o: 1}

I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like

0110100001100101011011000110110001101111

Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

My understanding is that the ASCII encoding would not get the best performance or results from the RNN because the ASCII codes for each character are not meaningful; they are arbitrary. If the number of each ASCII code represented something meaningful about the letter, it would work better. But they don't.

The same principles apply as when deciding how to encode any categorical data. If your categories are ordinal (eg. 'First', 'Second' .. or 'Age group 18-24', Age group 25-35' .. or even 'Social Class E', 'Social Class D' ..), then assigning a single numerical value to each class might work well. But in categorical data where there is no meaningful order, one hot encoding will work better.

This is an example of the principle of giving neural networks the most expressive data that we can. In the case of non-ordinal, arbitrary categories, one-hot is more expressive to the next layer of neurons (will stimulate them more distinctly) than using a numerical encoding.

Improve this answer
$\endgroup$
0
$\begingroup$

The request for input encoding is that mathematical difference between encoded inputs (numerical subtraction in absolute value of vector norm, etc) must be proportional to logical (real world) dis-similitude between the objects they represent.

Lets say by example the 3 first uppercase letters. In one-hot encoding they will be:

'A' : (1,0,0)
'B' : (0,1,0)
'C' : (0,0,1)

Note that the distance between any two pairs is 0 if the letters are the same and 2 if they differ.

Compare with ASCII in binary as encoding:

'A' : 0100 0001  
'B' : 0100 0010  
'C' : 0100 0011

Note distance between 'A' and 'B' is 2, but distance between 'A' and 'C' is 1. Thus, we are saying that an 'A' is more similar to a 'C' than a 'B'. Something artificial that will cause network errors.

Equivalent problem appears when using ASCII as integer:

'A' : 65
'B' : 66  
'C' : 67

we are saying that distance between 'A' and 'C' is 2, twice the distance between 'A' and 'B'. Again, something that will disturb network performance.

Improve this answer
$\endgroup$

You must log in to answer this question.