3
$\begingroup$

I've mostly seen (e.g. in http://karpathy.github.io/2015/05/21/rnn-effectiveness/) that when training RNN's on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.

For example, the text "hello" would be represented like

{h: 1, e: 0, l: 0, o: 0}
{h: 0, e: 1, l: 0, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 1, o: 0}
{h: 0, e: 0, l: 0, o: 1}

I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like

0110100001100101011011000110110001101111 

Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.