Can autoencoders be used for supervised learning without adding an output layer? Can we simply feed it with a concatenated input-output vector for training, and reconstruct the output part from the input part when doing inference? The output part would be treated as missing values during inference and some imputation would be applied.
One such paper I know of and which I implemented is Semi-Supervised Learning using Ladder Networks . I quote here their description of the model:
Our approach follows Valpola (2015), who proposed a Ladder network where the auxiliary task is to denoise representations at every level of the model. The model structure is an autoencoder with skip connections from the encoder to decoder and the learning task is similar to that in denoising autoencoders but applied to every layer, not just the inputs. The skip connections relieve the pressure to represent details in the higher layers of the model because, through the skip connections, the decoder can recover any details discarded by the encoder.
For further explanations on the architecture check Deconstructing the Ladder Network Architecture by Yoshua Bengio.
I recall reading papers about such systems, if I understand you correctly, but can't recall the titles at the moment.
The idea was to use character-based generative RNNs, train them on sequences encoded like "datadatadatadata|answer", and then when feeding in "otherdatadata|" then it would continue to generate some kind of expected answer.
But, as far as I recall, that was just a neat illustration since if you have the data to do something supervised, then you'd get better results by conventional methods.