How does a multidimensional vector get fed into a single node in a neural network?

Question

I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm not really sure how this works, because in all the books I've ever read, and all the projects that I've developed, only a single number has ever gone into each node. So, how does this work?

Answer 1

Fine thing to develop your algorithms from scratch, fine thing.

In deep learning, particularly in natural language processing (NLP) tasks, it is common to represent words as vectors and feed these vectors into neural network nodes. This technique is known as word embeddings or word vector representations. I can explain the concept to you based on the principles outlined in the book Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

In traditional neural networks, each node receives a single number as input, typically representing a feature or an activation value. However, when working with text data, a single number is not sufficient to capture the rich semantic meaning of words. Instead, we need a way to represent words in a continuous vector space, where similar words are close together and dissimilar words are far apart.

Word embeddings provide a solution to this problem. They are dense vector representations that capture semantic relationships between words. These vectors are learned from data and can be considered as a projection of words into a lower-dimensional space, where each dimension represents a different aspect of meaning.

One popular word embedding algorithm is Word2Vec, which has two main variants: Continuous Bag-of-Words (CBOW) and Skip-gram. Both algorithms aim to learn word embeddings by predicting the context of a word within a given sentence or a window of surrounding words.

The CBOW model, for instance, takes the context words as input and predicts the target word. In this case, the context words are encoded as vectors and summed together before being fed into a single node in the neural network. The network learns to associate the context vector with the corresponding target word. The training process adjusts the word embeddings in such a way that words with similar contexts have similar vector representations.

By using word embeddings, neural networks can leverage the semantic information captured in the vector space to perform various NLP tasks. These embeddings can capture syntactic and semantic relationships, analogies, and contextual information between words.

For example, Let's consider a simple example using the Word2Vec CBOW model to learn word embeddings from a small corpus.

Suppose we have the following corpus consisting of three sentences:

• "I love deep learning."
• "Deep learning is fascinating."
• "Neural networks are powerful."

To create word embeddings, we need to define a window size, which determines the number of context words we consider on each side of the target word. Let's assume a window size of 1 for this example.

First, we need to construct a vocabulary from our corpus. In this case, our vocabulary would consist of the following unique words:

["I", "love", "deep", "learning", "is", "fascinating", "neural", "networks", "are", "powerful"]

Next, we assign a one-hot encoding to each word in the vocabulary. A one-hot encoding represents each word as a binary vector where only the position corresponding to the word's index is 1, and all other positions are 0. For example, the one-hot encoding for the word "learning" would be [0, 0, 0, 1, 0, 0, 0, 0, 0, 0].

Now, we can create training examples for our CBOW model. Let's take the second sentence, "Deep learning is fascinating," as an example.

For each target word in the sentence, we consider the window of context words. With a window size of 1, we have the following training examples:

• Input (context) words: ["deep", "is"]
• Target (center) word: "learning"

To encode the context words as vectors, we use their respective one-hot encodings:

• Context word "deep": [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
• Context word "is": [0, 0, 0, 0, 1, 0, 0, 0, 0, 0]

To feed these context word vectors into a single node in the neural network, we can simply sum them together:

• Sum of context word vectors: [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]

The target word, "learning," will be used as the output label for training.

We repeat this process for all target words in the corpus, considering the context words within the defined window size. The neural network then learns to associate the summed context vectors with the corresponding target words by adjusting the word embeddings.

After training, we obtain word embeddings where words with similar meanings or contexts are closer together in the vector space. These embeddings can be used for various NLP tasks or as input to other downstream models.