4
$\begingroup$

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.

For example, the text "hello" would be represented like

{h: 1, e: 0, l: 0, o: 0}
{h: 0, e: 1, l: 0, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 0, o: 1}

I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like

0110100001100101011011000110110001101111 

Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?

$\endgroup$
0
$\begingroup$

My understanding is that the ASCII encoding would not get the best performance or results from the RNN because the ASCII codes for each character are not meaningful; they are arbitrary. If the number of each ASCII code represented something meaningful about the letter, it would work better. But they don't.

The same principles apply as when deciding how to encode any categorical data. If your categories are ordinal (eg. 'First', 'Second' .. or 'Age group 18-24', Age group 25-35' .. or even 'Social Class E', 'Social Class D' ..), then assigning a single numerical value to each class might work well. But in categorical data where there is no meaningful order, one hot encoding will work better.

This is an example of the principle of giving neural networks the most expressive data that we can. In the case of non-ordinal, arbitrary categories, one-hot is more expressive to the next layer of neurons (will stimulate them more distinctly) than using a numerical encoding.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.