1
$\begingroup$

I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal vectors (orthogonal matrix) in another. While geometrically these encode the same vector space, the one-hot embedding outperformed the random orthogonal one consistently. I suppose this has to do with the clarity of the signal: A zero vector with a single 1-valued cell is an easier to learn signal than just some vector with lots of different values in each cell.

I wondered if you know of any papers on this kind of effect. I did not find any but would like to back up this finding and check if my reasoning for why this is the case makes sense/ find a better or more in-depth explanation.

Improve this question
$\endgroup$
6
  • $\begingroup$ Why were you using a CNN in this case? It would not seem to offer any advantage unless you had repeated vectors (i.e. classifying a sequence of vectors in one go) where patterns would be meaningful. $\endgroup$
    – Neil Slater
    Commented Sep 8, 2019 at 10:25
  • $\begingroup$ @NeilSlater I use the CNN for text classification. Texts are encoded as sequences of vectors (stacked to form a 2d matrix, like an image). Each vector corresponds to one character. Does that answer your question? $\endgroup$
    – lo tolmencre
    Commented Sep 8, 2019 at 16:38
  • $\begingroup$ I guess you thought I was talking about optical character recognition or something? $\endgroup$
    – lo tolmencre
    Commented Sep 8, 2019 at 16:49
  • $\begingroup$ Yes that adds useful detail. It may be worth adding it to the question in case it helps someone, because it explains why you are using a CNN (you have multiple encoded items in the input layer). The current question leaves that open and makes it look like you might be trying to apply a CNN against a single one-hot-encoded vector, which would be odd $\endgroup$
    – Neil Slater
    Commented Sep 8, 2019 at 17:03
  • $\begingroup$ @NeilSlater Ok. I take it then you do not know of any studies on this type of effect of ne-hot encoded data performing better than "orthogonally encoded" data, yes? This is not necessarily specific to CNNs, I'd assume. $\endgroup$
    – lo tolmencre
    Commented Sep 8, 2019 at 17:23

1 Answer 1

Reset to default
0
$\begingroup$

There is a recent paper "Convolutional embedding for edit distance" (Dai et al., 2020) that shows that a convolutional net using one-hot character-based input encoding computes vector embeddings whose Euclidean distances correspond to edit distance. So it is not surprising that, for problems where edit distance makes sense as a similarity metric among texts, this approach will perform quite well out of the box. This is a very useful inductive prior, especially if there is limited training data.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .