Combine two feature vectors for a correct input of a neural network

Question

Let's consider this scenario. I have two conceptually different video datasets, for example a dataset A composed of videos about cats and a dataset B composed of videos about houses. Now, I'm able to extract a feature vectors from both the samples of the datasets A and B, and I know that, each sample in the dataset A is related to one and only one sample in the dataset B and they belong to a specific class (there are only 2 classes).

For example:

Sample x1 AND sample y1 ---> Class 1
Sample x2 AND sample y2 ---> Class 2
Sample x3 AND sample y3 ---> Class 1
and so on...

If I extract the feature vectors from samples in both datasets , which is the best way to combine them in order to give a correct input to the classifier (for example a neural network) ?

feature vector v1 extracted from x1 + feature vector v1' extracted from y1 ---> input for classifier

I ask this because I suspect that neural networks only take one vector as input, while I have to combine two vectors

Answer 1

The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.

Assume the first sample is made of the pair $X_1$ and $Y_1$. Let the corresponding feature vectors for $X_1$ and $Y_1$ be $\textbf{v}_1$ and $\textbf{v}_2$, respectively.

$$ \textbf{v}_1 = [f_1, f_2, \ldots , f_n],\\ \textbf{v}_2 = [g_1, g_2, \ldots , g_m]. $$ Then, the first sample's feature can be defined as $$ \textbf{v} = [f_1, f_2, \ldots , f_n, g_1, g_2, \ldots , g_m]. $$ Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).

Answer 2

$^*$Note - Question is bit unclear, in case the answer doesn't addresses the question, please ask for edit/delete Request.

---

GENERALIZATION

Suppose there are multiple datasets denoted by $A_i$. Datasets contain a set of Vectors $x_{j} $. Mathematically $A_i = \{ x_j\}_{j=0}^n$. We've to find an estimator function $\hat f$, such that $\hat f( \vec r) = y, \, \vec r \in X$ where $X $ is a special dataset created by combining all $A_i$ which helps in classification into $y \in Y$ which is the set of classes.

.

As @Amir Mentioned out, linearly separable feature can be easily separated by straight combination of vectors i.e. if $x_u \in A_i, w_v \in A_j \dots$, then $r = [x_1 \,x_2 \, \dots \, x_u \, w_1 \, \dots w_v \dots]$. Where, $r \in X$ which is the required dataset.

There are cases where the features are not linearly separable, We use basis expansion methods[1] to make required shape of hyperplane to separate the features. We create a new dataset combining $A_i \, \forall i \in C \subset \mathbb N$. Suppose that the new dataset is $X$, then $r \in X$ and $r = [r_0, r_1, \dots r_n].$

Then,

$$r_1 = u_1^2v_1^2 \\ r_2 = \sin(u_2)\sin(v_2) \\ r_3 = ae^{u_3 + v_3} \\ r_4 = a v_4 v_4 + a_2 u_4^2 v_4^2 + \dots \\ \dots$$

Here $u_p \in A_i; \, v_q \in A_j$

Here you can use all the creativity to set $r = [r_1, r_2, \dots , r_n]$ and make a new dataset. What equations and what functions you chose fully depends on the kind of hyperplane shape you want to obtain. Basis expansion is just one of the methods for feature extraction is certainly one of the most flexible too.

Now, you feed the newly created vectors into your trained estimator functions (which is Neural Net) which can classify things much easily now.

In case of Regression/Classification without Neural Net needs some extra treatment to train the model[2].

---

[2]Note: There is also a big role of encoding. For example, if you encode colors by numbers $1, 2, 3$ for RGB or $10,01, 11$ fully changes everything and your features too. In such cases, You may even need different equations to make your required dataset $X$ and vectors $r$.

---

REFERENCES:

1. Oleszak, Michal. https://towardsdatascience.com. Non-linear regression: basis expansion, polynomials & splines. Sep 30, 2019. Web. 6 May 2020.
2. Sangarshanan. https://medium.com. Improve your classification models using Mean /Target Encoding. Jun 23, 2018. Web. 6 May 2020.