I am trying to train a CNN to learn 5D (kind of) data. The data is structured as follows. It has three spatial dimensions
[x, y, z], but it also has two "internal dimensions"
[theta, phi] at each
[x, y, z]. What I am trying to do is upsample the internal space from fewer
[theta, phi] data points.
When I train a 2d residual network with random
[x, y, z] points in just the internal space it learns -- but there is some noise in the
x, y, z space, there should be a correlation with neighbouring points.
What I wanted was some way to also include convolutions over the 3D
[x, y, z] space to try and remedy this.
A possible but maybe naive approach is to do the following: Stack the images as
[theta * phi, x, y, z] (so, many input channels) and then have some 3d convolution layers, then after that stack as
[x * y * z, theta, phi] and take 2d convolutions in the internal space.
Another approach is to use 5d filters that span over all dimensions. This might be hard to implement for me and probably very memory hungry.
Are there any other ways?