0
$\begingroup$

I need to retrieve just the text from emails. The emails can be in HTML format, and can contain huge signatures, disclaimer legalese, and broken HTML from dozens of forwards and replies. But, I only want the actual email message and not any other cruft such as the whole quotation block, signatures, etc.

This isn't really a problem that could be solved with regex because HTML mail can get very, VERY messy.

Could a neural network perform this task? What kind of problem is this? Classification? Feature selection?

|improve this question
$\endgroup$
2
$\begingroup$

It's certainly possible to treat this as a natural language processing problem, basically you're looking to assign "salience" scores to the text.

Really, though, that's overkill for this kind of problem. Writing a regex or a CFG parser (or better: finding an existing parser) is likely to be easier and more reliable.

|improve this answer
$\endgroup$
  • $\begingroup$ Regex are out of the question. This problem cannot be solved by regex since the inputs are public and I can't write one for every mail software out there. The Talon library "sort of" does this, but the regex only "sort of" works for english. $\endgroup$ – hjf Aug 6 '18 at 13:16
  • $\begingroup$ I suspect you'd still find something like a CFG parser (e.g. made with YACC) more reliable and easier. While there are a lot of mail clients, you'd probably get 80%+ of traffic by covering the biggest 3 of them (outlook, gmail, thunderbird?). If you really can't use one, then "salience" scores are probably your best bet. There are some good python libraries for this I think: nltk.org $\endgroup$ – John Doucette Aug 6 '18 at 13:21
  • $\begingroup$ Talon actually does this. It claims that 90% of the time the "simple" approach works, and for the remaining 10% they used scikit-learn to train a model with the ENRON emails. I've tested the library. It works perfectly with some emails, but fails miserably with others. The "others" are malformed HTML emails (companies like to slap their stupid, useless legalese right at the server to every outgoing email, breaking HTML mail) $\endgroup$ – hjf Aug 6 '18 at 13:27
  • 1
    $\begingroup$ Ah! This clarifies things a lot. I think you should actually just use the raw text. Don't bother filtering out the unneeded HTML tags. If those turn out not to be useful, the network will learn to ignore them. If they turn out to be useful, the network will use them. In fact, this is the approach used by many email classification systems, not just neural nets (see, for instance, DMC and other compression-based approaches). $\endgroup$ – John Doucette Aug 6 '18 at 19:41
  • 1
    $\begingroup$ As a follow up: I spent the last 2 days writing regex to catch most of the signature and reply blocks. I found about 20 patterns in the sample of 1000 emails I took. while it's not perfect, it seems to be "good enough". the worst offender is, of course, Outlook, which seems to divide the replies with a div (because <hr> isn't good enough for microsoft). and this div depends on the user's language. Sometimes it specifies a height of 0cm, other times 0in... unbelievable. $\endgroup$ – hjf Aug 9 '18 at 22:13
1
$\begingroup$

It is a surmountable problem for someone experienced in software architecture and machine learning.

  1. Render the message to a virtual display such as xvfb, headless Chrome, or phantomjs.
  2. Capture the text with selenium, watir, or some other DOM controller, addressing your HTML and DHTML complexity concern.
  3. OCR the text in inline images and insert it appropriately.
  4. Once you have text with only word, line, list item, and paragraph breaks as structural separators, you have adequate separation of style and language content to then use naive Bayesian or one of the more recent forms of unsupervised categorization to find the separation point between the body and the signature block.

Extending your line of thinking, you may even be able to engineer a generative strategy for automated reply, but beware, this last feat is a dozen orders of magnitude more difficult than extracting text from HTML, DHTML, and typeset images and machine learning the separating signature blocks.

This last feat, if done poorly, would get you in trouble with many of your email reply recipients, and, if done well, would place you ahead of Amazon, Apple, and Google.

|improve this answer
$\endgroup$
  • $\begingroup$ Intriguing. I was actually wondering how to deal with the broken HTML usually found in emails. The second part you mention is irrelevant for us since the service we sell is actually humans to respond. But we need help categorizing the inputs and try and deliver to the right person. We're already doing this with tweets and facebook messages which are trivial being text-only. But email is a different beast. $\endgroup$ – hjf Aug 6 '18 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.