# Combine two feature vectors for a correct input of a neural network

Let's consider this scenario. I have two conceptually different video datasets, for example a dataset A composed of videos about cats and a dataset B composed of videos about houses. Now, I'm able to extract a feature vectors from both the samples of the datasets A and B, and I know that, each sample in the dataset A is related to one and only one sample in the dataset B and they belong to a specific class (there are only 2 classes).

For example:

Sample x1 AND sample y1 ---> Class 1
Sample x2 AND sample y2 ---> Class 2
Sample x3 AND sample y3 ---> Class 1
and so on...


If I extract the feature vectors from samples in both datasets , which is the best way to combine them in order to give a correct input to the classifier (for example a neural network) ?

feature vector v1 extracted from x1 + feature vector v1' extracted from y1 ---> input for classifier

I ask this because I suspect that neural networks only take one vector as input, while I have to combine two vectors

The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.

Assume the first sample is made of the pair $$X_1$$ and $$Y_1$$. Let the corresponding feature vectors for $$X_1$$ and $$Y_1$$ be $$\textbf{v}_1$$ and $$\textbf{v}_2$$, respectively.

$$\textbf{v}_1 = [f_1, f_2, \ldots , f_n],\\ \textbf{v}_2 = [g_1, g_2, \ldots , g_m].$$ Then, the first sample's feature can be defined as $$\textbf{v} = [f_1, f_2, \ldots , f_n, g_1, g_2, \ldots , g_m].$$ Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).

• Is it a problem if the dimension of v1 and v2 is quite big ? Like 8000 ? – AleWolf May 9 '20 at 14:15
• It also depends on the number of samples that you have in your dataset. In other words, you need enough samples to be able to solve your problem in that high dimensional space. If you want to learn more please refer to subsection 5.11.1 The curse of dimensionality on the Deep Learning book (available at deeplearningbook.org) [1]. As a quick solution, you may use dimensionality reduction techniques to reduce the number of features that you have. [1] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. – Amir May 10 '20 at 17:27

$$^*$$Note - Question is bit unclear, in case the answer doesn't addresses the question, please ask for edit/delete Request.

## GENERALIZATION

Suppose there are multiple datasets denoted by $$A_i$$. Datasets contain a set of Vectors $$x_{j}$$. Mathematically $$A_i = \{ x_j\}_{j=0}^n$$. We've to find an estimator function $$\hat f$$, such that $$\hat f( \vec r) = y, \, \vec r \in X$$ where $$X$$ is a special dataset created by combining all $$A_i$$ which helps in classification into $$y \in Y$$ which is the set of classes.

.

As @Amir Mentioned out, linearly separable feature can be easily separated by straight combination of vectors i.e. if $$x_u \in A_i, w_v \in A_j \dots$$, then $$r = [x_1 \,x_2 \, \dots \, x_u \, w_1 \, \dots w_v \dots]$$. Where, $$r \in X$$ which is the required dataset.

There are cases where the features are not linearly separable, We use basis expansion methods[1] to make required shape of hyperplane to separate the features. We create a new dataset combining $$A_i \, \forall i \in C \subset \mathbb N$$. Suppose that the new dataset is $$X$$, then $$r \in X$$ and $$r = [r_0, r_1, \dots r_n].$$

Then,

$$r_1 = u_1^2v_1^2 \\ r_2 = \sin(u_2)\sin(v_2) \\ r_3 = ae^{u_3 + v_3} \\ r_4 = a v_4 v_4 + a_2 u_4^2 v_4^2 + \dots \\ \dots$$

Here $$u_p \in A_i; \, v_q \in A_j$$

Here you can use all the creativity to set $$r = [r_1, r_2, \dots , r_n]$$ and make a new dataset. What equations and what functions you chose fully depends on the kind of hyperplane shape you want to obtain. Basis expansion is just one of the methods for feature extraction is certainly one of the most flexible too.

Now, you feed the newly created vectors into your trained estimator functions (which is Neural Net) which can classify things much easily now.

In case of Regression/Classification without Neural Net needs some extra treatment to train the model[2].

[2]Note: There is also a big role of encoding. For example, if you encode colors by numbers $$1, 2, 3$$ for RGB or $$10,01, 11$$ fully changes everything and your features too. In such cases, You may even need different equations to make your required dataset $$X$$ and vectors $$r$$.

## REFERENCES:

1. Oleszak, Michal. https://towardsdatascience.com. Non-linear regression: basis expansion, polynomials & splines. Sep 30, 2019. Web. 6 May 2020.
2. Sangarshanan. https://medium.com. Improve your classification models using Mean /Target Encoding. Jun 23, 2018. Web. 6 May 2020.
• I really appreciate your help, your answer is fantastic. How can I know if the two input datasets are linearly separable, so that I can use the @Amir method ? – AleWolf May 9 '20 at 6:32
• @AleWolf I'll just put some short points. 1. Visualize Data to see that the features which you've combined is linearly separable or not. 2. In case you are not sure or don't have any idea, there's quick and dirty way is to use Deep Networks instead of Neural Networks with number of layers depending on intricacy of Features. Beware - Deep Nets require usually 2-3x more datasets than Probability based classifiers (Usually, but that depends). 3. The kind of function you need depends on features just like $\sin x , e^x$ are different. Prefer going to links, I've attached to see more. – RewCie May 9 '20 at 8:15
• Thats great, last question. Why intuitively, if the two datasets are linearly separable I can use the combination of vectors method (exposed by @Amir), yet if they are not it's better to use the basis expansion methods ? – AleWolf May 9 '20 at 9:13
• @AleWolf Yes, if you are using deep nets, then you can do that – RewCie May 9 '20 at 14:23
• @AleWolf Yep, try :) – RewCie May 9 '20 at 14:57