How to combine input from different types of data sources?

Question

I've to train a neural network using microphone data (wav files), accelerometer sensor data and light sensor data.

Right now the approach I thought was to convert all data into images and combine them into a single image and train my neural network.

Another approach was to convert wav files into arrays and combine them along with sensor data and train my neural net.

Are my approaches correct or is there a better way to do this?

Any suggestions/ideas are welcome.

Answer 1

Handling Multiple Input Types

Multiple input types are common as learning technology moves from academic labs and open source examples into the real world. The design process includes the below four steps, which related directly to this question. There are other steps that don't centrally related not mentioned here.

First Step — List Inputs in Most Usable Form

In the case of this question, there are three in the list.

• Monophonic or multichannel audio data as an array of samples or sample vectors acquired at a fixed sampling rate
• Acceleration data as an array of samples in $\mathbb{R}^n$ where $n \in \{1, 2, 3\}$ acquired at a fixed sample rate.
• Visible light or IR detection data as an array of samples of $c$ channels at a fixed sample rate.

Fixed sample rate is listed with each type. Variable sample rates pose additional burden on learning. If any of the three are acquired with a variable sample rate, they can be converted to fixed sample rate arrays provided there is a time stamp or a time-since-last-sample channel available with each sample vector.

Variable to fixed sample rate conversion can be accomplished easily via ffmpeg, programmatically via its library, or through any equivalent software designed for this purpose. The acceleration and visible light signals can also be adapted for ffmpeg's input.

When a specific dynamic characteristic must be preserved, finer control over the interpolation can be obtained using cubic spline algorithms. Such algorithms were originally written in FORTRAN or C and have been ported to Java and Python.

Second Step — When Spectral Data is Practically Required

With time series data, the phenomena being learned is often largely a function of vibration frequencies and amplitudes. This is almost always the case with audio data. The exception is when transient detection is the primary teller of the conditions being measured rather than more continuous sounds.

A note is spectral in description. A knock is a transient and both spectra and actual rise characteristics are indicative of source.

Presenting digital samples from the transducer circuit (in this case the microphone diaphragm's position in response to pressure waves) to the learning system directly is not a best practice. Unconverted vibration information forces the learning system to learn transformations for which optimized algorithms already exist. This slows learning and generally pushes complexity beyond the learning system's capacity.

When the spectral features best characterize the phenomena being learned, as in hums, screeches, vowel sounds, music, horns, or beeps, the best practice is to convert the samples in the time domain to frequency domain data.

This is usually done using one of the forms of the FFT (fast Fourier transform) algorithm in combination with a RMS (root mean square) calculation of amplitude and windowing. Windowing attenuation forms, such as the Hamming window or the cosine window, are available in most packages that have FFT capabilities.

The sample rate is also important. Nyquist's sample rate criteria must be met by the A-to-D converter (in the sound card or instrumentation circuitry) to ensure that the spectral range includes the best indicators of the phenomena to be learned. In machine learning, such indicators are called features.

In most cases, audio or vibration information is presented to a learning system or sub-system as spectral frequencies and amplitudes. Vibration recognition systems and speech to text systems generally execute FFTs (with RMS and windowing) on the raw input. In human ears, the conversion to spectral information is mechanical, using the tapering geometry of the ear's cochlea.

There also exists algorithms that can directly develop spectral information from variable sample rate data to amplitudes and frequencies, combining two transformations into one for efficiency of computing resources and additional input accuracy.

In summary of this section, whether light, IR, acceleration, or other data is treated as audio and converted to spectral information is based on whether

• The signals are vibrations that have rises and falls within specific frequency ranges like hums, notes, and speech or
• They are transients (as with tapping the mechanical package on which the accelerometer is mounted) or
• They are binary in nature (as in the case of linear encoder signals).

Third Step — When Topology of Types

The topological orientation of the input source types to one another is key to feeding the information into any learning system. These are the two pure extremes.

1. The signals from the source types are describing the same physical or data mining property.
2. The signals are measuring independent properties that together describe the state of the system being measured.

In the case of (1), a normalized sample from one type of input channel is, from a learning point of view, indistinguishable from a normalized sample from another. This would be the case if the accelerometer is measuring the same vibration that the microphone is measuring. This is the less usual case, but is more common when the data is for study of vibration to project mechanical or materials failures in advance.

With Background Clarified, the Q&A Becomes Clearer

The question asked this.

How [does one best] combine input from different types of data sources?

Whether the source data is case (1) or case (2) or some midpoint between them is key to answering this question. Samples from different source types that measure single phenomena can be phase adjusted, spectrally adjusted, and otherwise normalized to present a single data type at the input. Merging into a single serial data stream would be the optimal pre-processing for training in case (1).

More often, the topological orientation of the different input source types conforms to case (2) above. In that case, different input types should be presented in parallel to the learning system inputs. The merging would be counterproductive in this case, for a reason that will be clarified in the next section below.

To avoid sparsity in input data, which creates additional training burden, the sample rates should be equalized for all channels of all data source types. This may require conversion of the sample rate, which can be done using the same techniques (above) for converting from variable to fixed sample rates.

Images of Data

The question stated this.

Right now, the approach I [was considering is] to convert all data into images and combine them into a single image and train my neural network.

The attempt to normalize and combine into one input stream is the correct approach if (1) above is the case. However, presenting data as images requires the learning system restore the data to a less redundant, more feature encoded form, so conversion to images is not recommended.

If (2) above is the case, the provision of unique and property-independent data through the same input tensors requires the artificial network sort the input samples by type, which can be done with recurrent or attention based artificial networks, but it is unnecessary. Here's the rule.

If the signals from different sources are measuring independent properties to which learning will be applied, do not merge the input streams. Instead, present them in parallel to the learning system so that the learning system does not need to also learn how to discriminate sample types.

The reason topology is mentioned in the subtitle of this section is that there is a third case, (3), which is a hybrid of (1) and (2) as hinted above.

Many excellent detection and control systems process the input channels in relation to one another prior to the actual learning system components. This pre-processing takes advantage of known relationships in the data and the topology of signals in phase space.

The phase space of two or three signals can be visualized in 2D or 3D where each channel is assigned to a Cartesian coordinate system's axis and the time domain is not represented by any axis. Mechanical fault detection can be optimized using phase space analysis.

An example of this more complex case (3) is in the human recognition of both consonant and vowel sounds and various other non-speech audio phenomena. After the cochlear conversion to spectra, the transient information (as from a 'k' sound or a click) and tone information (as from an 'e' or a whistle) is presented to speech recognition nets in the brain in an integrated form.

This decreases the learning time newborns take to learn voice and household sounds and probably improved evolutionary viability in times of global stress or local predatory activity.

In automated vehicles and smart robotics, the same is true. Some sounds, accelerations, and visible light changes are transient and some are vibrations. Studying the topology of the surface created by the input signal types and their spectra in phase space provides much information. Armed with the results of such studies allows excellence in the preparation of input to a learning system.

Fourth Step — Normalizing and Standardizing

Once the above is complete, understanding how to present the data to the input layer of an artificial network becomes clearer. The goal of this design step is to have relatively equal ranges for the components of each input tensor, whether presented serially or in parallel depending on the above criteria.

Doing so reduces the burden on the learning system. All signals are presented at full strength but within the clipping range of the signal. This is a general principle and is important for both data center (batch) learning and real time learning and adaptive systems.