My idea is to model and train a neural network that receives a text version of a PDF file as the input and gives the content text as output.
Take the scenario:
One prints a PDF file to a text file (the text file does not have images, but has the main text, headings, page numbers, some other footer text, and so on, and keeps the same number of columns - two for instance - of text);
This text file is submitted to a tool that strips everything that is not the main content of the text in one single text column (one text stream), keeping the section titles, paragraphs, and the text in a readable form (does not mix columns);
The tool generates a new version of the original text file containing only the main text portion, ready to be used for other purposes where the striped parts would be considered noise.
How to model this problem in a way a neural network can handle it?
Update 1Here are some clarifications on the problem.
The picture below shows two pages of a pdf version of a scientific paper. This is just to set the context, the PDF file is not the input for this problem, it is just to understand where the actual input data comes from.
The color boxes show some parts of interest for this discussion. Red boxes are headers and footers. We are not interested in them. Blue and green boxes are content text blocks. Different colors were used to emphasize the text is organized in columns and that is part of the problem. Those blue and green boxes are what we actually want.
Text fileIf a use the "save as text file" feature of my free PDF reader, I get a text file similar to the image below.
The text file is continuous, but I put the equivalent of the first two pages of the PDF file side-by-side just to make things easier to compare. We can see the very same colored boxes. In terms of words, those boxes contain the same text as in the PDF version.
Understanding the problemWhen we read a paper, we are usually not very interested in footers or headers. The main content is what we actually read and that will provide us with the knowledge we are looking for. In this case, the text is inside blue and green boxes.
So, what we want here is to generate a new version of the input (text) file organized in one single text stream (one column if you will), with the text laid-out in a form someone can read it, which means, alternating the blue and the green boxes.
However, if the original PDF has no footers, it should work in the same way, providing the main text content. If the text comes in three of four columns, the final product must be a text in good condition to be read without losing any information.
Any pictures will be simply stripped off the text version of the paper and we are fine with that.