1
$\begingroup$

I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal vectors (orthogonal matrix) in another. While geometrically these encode the same vector space, the one-hot embedding outperformed the random orthogonal one consistently. I suppose this has to do with the clarity of the signal: A zero vector with a single 1-valued cell is an easier to learn signal than just some vector with lots of different values in each cell.

I wondered if you know of any papers on this kind of effect. I did not find any but would like to back up this finding and check if my reasoning for why this is the case makes sense/ find a better or more in-depth explanation.

$\endgroup$
  • $\begingroup$ Why were you using a CNN in this case? It would not seem to offer any advantage unless you had repeated vectors (i.e. classifying a sequence of vectors in one go) where patterns would be meaningful. $\endgroup$ – Neil Slater Sep 8 at 10:25
  • $\begingroup$ @NeilSlater I use the CNN for text classification. Texts are encoded as sequences of vectors (stacked to form a 2d matrix, like an image). Each vector corresponds to one character. Does that answer your question? $\endgroup$ – lo tolmencre Sep 8 at 16:38
  • $\begingroup$ I guess you thought I was talking about optical character recognition or something? $\endgroup$ – lo tolmencre Sep 8 at 16:49
  • $\begingroup$ Yes that adds useful detail. It may be worth adding it to the question in case it helps someone, because it explains why you are using a CNN (you have multiple encoded items in the input layer). The current question leaves that open and makes it look like you might be trying to apply a CNN against a single one-hot-encoded vector, which would be odd $\endgroup$ – Neil Slater Sep 8 at 17:03
  • $\begingroup$ @NeilSlater Ok. I take it then you do not know of any studies on this type of effect of ne-hot encoded data performing better than "orthogonally encoded" data, yes? This is not necessarily specific to CNNs, I'd assume. $\endgroup$ – lo tolmencre Sep 8 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.