How to extract the main text from a formatted text file?

Question

My idea is to model and train a neural network that receives a text version of a PDF file as the input and gives the content text as output.

Take the scenario:

1. One prints a PDF file to a text file (the text file does not have images, but has the main text, headings, page numbers, some other footer text, and so on, and keeps the same number of columns - two for instance - of text);

2. This text file is submitted to a tool that strips everything that is not the main content of the text in one single text column (one text stream), keeping the section titles, paragraphs, and the text in a readable form (does not mix columns);

3. The tool generates a new version of the original text file containing only the main text portion, ready to be used for other purposes where the striped parts would be considered noise.

How to model this problem in a way a neural network can handle it?

Update 1

Here are some clarifications on the problem.

PDF file

The picture below shows two pages of a pdf version of a scientific paper. This is just to set the context, the PDF file is not the input for this problem, it is just to understand where the actual input data comes from.

The color boxes show some parts of interest for this discussion. Red boxes are headers and footers. We are not interested in them. Blue and green boxes are content text blocks. Different colors were used to emphasize the text is organized in columns and that is part of the problem. Those blue and green boxes are what we actually want.

Text file

If a use the "save as text file" feature of my free PDF reader, I get a text file similar to the image below.

The text file is continuous, but I put the equivalent of the first two pages of the PDF file side-by-side just to make things easier to compare. We can see the very same colored boxes. In terms of words, those boxes contain the same text as in the PDF version.

Understanding the problem

When we read a paper, we are usually not very interested in footers or headers. The main content is what we actually read and that will provide us with the knowledge we are looking for. In this case, the text is inside blue and green boxes.

So, what we want here is to generate a new version of the input (text) file organized in one single text stream (one column if you will), with the text laid-out in a form someone can read it, which means, alternating the blue and the green boxes.

However, if the original PDF has no footers, it should work in the same way, providing the main text content. If the text comes in three of four columns, the final product must be a text in good condition to be read without losing any information.

Any pictures will be simply stripped off the text version of the paper and we are fine with that.

Answer 1

After a long wait and some digging, I accidently found what I was looking for. In 2015, polish researcher Dominika Traczyk publish an article presenting CERMINE, a solution for the posted problem.

His solution is SVM-based, but the article gives good insights for an alternate Neural Network version.

The article is open access and can be found on the Springer website, while all the source code is available on GitHub.

Answer 2

How to extract the main content text from a formated text file?

I am not sure that just a neural network is the best approach to your problem.

Traditional natural language processing software are using something else, and generally using a complex mix of several techniques. I am supposing you are processing written text available as some file (in a file format you are very familiar with, e.g. OOXML or PDF or HTML5).

Read the wikipage on natural-language understanding and the one on parse trees (or concrete syntax trees).

BTW, you might use LaTeX or the Lout formatter to produce some PDF file. Both are open-source software (easily available on most Linux distributions, including Debian or Ubuntu). I recommend you to try generating some PDF file using them, and experiment on the generated PDF file. And a lot of AI papers are available (as preprints) in PDF form.

You could also use, as a PDF input to experiment your software, this or that draft reports (you might enjoy reading them too...). If in 2021 your software is capable of "understanding" and "abstracting/summarizing" these PDF files, please send me an email to [email protected] explaining (in written English) how you did build your neural network and what is the output of your software.

There are several issues:

• extracting the non-textual things (e.g. HTML tags from HTML input, or strings from a PDF file, or some LaTeX one).

• detecting the human language used in your text (e.g. French or English or Russian or Chinese). N-gram based techniques come to mind.

• having a data structure or database representing a dictionnary of a thousand (at least) of significant words (in English or Russian or whatever human language you are interested in) related to the domain you want to handle (that dictionary would be different if you want to parse weather forecasts or documentation related to the automotive industry, since the word pressure or speed relates to different concepts. Notice also that "weather" and "time" are expressed in French by the same word: "temps" - as in "le temps qu'il fait" for ongoing weather and "le temps qui passe" for the flow of time). A "Queen" is not the same for a chess player and an historian. A program translating -or just analyzing- chess comments from English won't use the same word for translating / understanding "bishop" (in chess, "fou" in French, literally the crazy guy, unrelated to religion; in Russian chess books it would be "слон", literally an elephant) than another program translating / analyzing historical comments from English (e.g. about Mary Stuart).

• modeling inside your software some domain-specific knowledge related to your analyzed text, since you would handle differently weather forecast text to textual comments of chess competitions, or textual exercises in any computer science or programming book (like CLRS). You could use some frame-based representations, like in RefPerSys or in CyC.

• building a semantic network representing the input text. I believe you might need some prior one representing domain-specific knowledge in the area of the analyzed text (e.g. a program analyzing comments on chess games needs to know the rule of chess; another program analyzing StackOverflow answers probably needs to know something about operating systems in general). In think that in English "overflow" or "overheating" means very different concepts to software developers and to weather forecasters or climate experts.

Look also for inspiration into this blog of the late Jacques Pitrat. He did wrote an interesting book on your topic.

You might look inside the DECODER European project, and read more about expert systems and their inference engine and knowledge bases.

Your project could give you some PhD.

You certainly need several years of work to achieve your goals. I suggest contacting some academic in your area to be your advisor.

Notice that on Linux the pdf2text software is extracting text from PDF files. It is open-source, but I won't say it is an AI software. However, you could use it thru popen(3). See also regex(7).

BTW, the PDF specification is public as ISO 32000-2:2017 (and is related to PostScript). Get it and read it, and see also this youtube video or this 978 pages document. On Linux, most PDF files can usually be inspected with od(1) or less(1).

My HP Office Pro 8610 printer (connected to a Linux desktop) is capable of printing some PDF and of scanning into a PDF file. But if I print on paper some PDF file and scan it, the PDF file did change a lot, even if visually it looks the same.

Notice that some drawings -or photos- could be embedded in a PDF file, and appear to a non-blind human reader as letters.