The underlying problem is combinatorial, as you note, but I'm not getting how you're ascribing value to words.
The key element of deciding line breaks, beyond the visual, is rhythmic. (There are other factors, as Bob Salita notes, but you've gotta start somewhere.)
It seems to me you need to teach the computer how to scan a phrase in the poetic sense, which relates to rhythm. This is obviously a very difficult task, but the number of syllables and stresses is fundamental numerical data of poetry.
In order to validate to human tastes, you'd then have to use a captcha crowdsourcing method, for both the rhythmic stresses as input, and getting human reactions to different line-break configurations. You would then reinforce the positive reactions, and the AI would tailor the line-break process to the audience.
However, instead of utilizing human tastes and aesthetic sensibilities, you could instead let the AI decide what is preferred, which would probably be comprised of some sort of symmetry considered optimal to an algorithm.
Following this logic, you wouldn't even need to have the AI learn the stresses, instead just focusing on raw syllables, or, numeric representation based on any factor. (With this method, the object is not to reformat poetry for humans, but for machines :)
This is more about the aesthetics, but Cameron Browne's Elegance in Game Design would seem to suggest there are engineering solutions to the type of aesthetic issues at the root of your problem.
I might start by teaching it to count the syllables of the poem, then having it look at the divisors. If it's roughly 10, it might be iambic pentameter. The AI doesn't care about the label, but it likes 10.
20 syllables might represent a couplet in that meter:
The time is out of joint, oh cursed spite
that ever I was born to set it right
I'd definitely start by feeding it older poetry, particularly poets that keep to strict meter. It's been a while since I've read Spencer and so forth, but I'd think poets of his time would be useful. Dr. Seuss, perhaps the greatest wielder of the rhyming couplet, would surely be extraordinarily useful.
The evaluation method would have to be fuzzy, because there would be increasing degrees of variance the more modern the poetry, ultimately resulting in free structures, except in the case of forms such as rap, which strongly utilize regularized rhythm. Machine learning is all about estimation and reinforcement, and is proving to be extremely useful dealing with fuzziness.
Dead mountain mouth of carious teeth that cannot spit
is a great example of modern line of poetry: the floor of 13 syllables / 2 makes a 6 beat line. Understanding that in context with the surrounding verse is much more difficult and illustrates the nature of the problem. Even scanning the poem correctly to that point to determine would be extremely difficult.
However, a different poem by the same author is extremely useful:
What is the late November doing / With the disturbance of the spring / And creatures of the summer heat, / And snowdrops writhing under feet / And hollyhocks that aim too high / Red into grey and tumble down / Late roses filled with early snow? / Thunder rolled by the rolling stars / Simulates triumphal cars / Deployed in constellated wars / Scorpion fights against the sun / Until the Sun and Moon go down / Comets weep and Leonids fly / Hunt the heavens and the plains / Whirled in a vortex that shall bring / The world to that destructive fire / Which burns before the ice-cap reigns
All lines of roughly 8 syllables, easy to pick out because of capitalization. But the real question is: ~136 13 lines of roughly 10 syllables, or 17 lines of roughly 8? It would want to calculate based on word blocks (words that cross syllabic thresholds and at least tell you where the break cannot be, and it should be possible to statistically divine the pattern, at least for regularized verse.)
The wounded surgeon plies the steel / That questions the distempered part; /
Beneath the bleeding hands we feel / The sharp compassion of the healer's art
/ Resolving the enigma of the fever chart.
This verse highlight the problem. 5 lines of 4 beats, but syllabically: 8, 8, 8, 10, 12.
Most likely:
- 46/5 = 9.2
- 46/4 = 11.5
- 46/6 = 7.66
Less likely:
- 46/3 = 15.3
- 46/2 = 23
- 46/7 = 6.57
2 lines has less perfect symmetry, but 5 lines is more likely, based on the overall number of syllables, and of the likely choices, has the least variance.
Ultimately it would be looking for the underlying structure, or lack of structure, and try to reorganize the unbroken text into something close to the original structure. While exactness is not always required because the process is ultimately subjective, and currently intractable, certain wrong choices would yield disastrous results.
In the prior example it may be able to discern the likelihood of a 5 line pattern, but it would have to figure out on which lines to place the extra syllables. Differentiating between particles and other parts of speech provides a clue, because the poet's language is very compact: there are 19 nouns, verbs, or prepositions.
More likely:
- 19/5 = 3.8
- 19/4 = 4.75
Less likely:
- 19/3 = 6.33
- 19/6 = 3.16
- 19/7 = 2.71
Further analysis might narrow it down. But extremely regularized verse remains the best place to start. 7 lines of roughly 10 syllables is "poetic":
‘The aged man that coffers-up his gold
Is plagu’d with cramps and gouts and painful fits;
And scarce hath eyes his treasure to behold,
But like still-pining Tantalus he sits,
And useless barns the harvest of his wits;
Having no other pleasure of his gain
But torment that it cannot cure his pain.
It cares about both X and Y values.
Initially you want to keep it to a single language, because syllables may be treated differently. That said, having the AI look for something like Dactylic hexameter would be extremely useful, because you could feed it Homer. You could also feed it Homer in English in many different forms of English meter, and in almost every other living language. By definition, the AI would value works such as these, because max number_of_translations provides the most robust data set. When it starts to value meaning, this will be especially important.
Understanding different ways of treating syllables(long/short vs. stressed/unstressed) will also be essential as it transitions into more modern poetry.
Here is a good link for basic English meter. Iambic and Trochaic meters will be easy, while meters that employ Anapests, Dactyls and Spondees will be more challenging.
In some cases, however, these will be mathematically interchangeable.
I went to the Garden of Love, / And saw what I never had seen: / A Chapel was built in the midst, / Where I used to play on the green.
It doesn't matter if the lines above are iambic/trochaic or dactylic/anapestic, it's still 4 lines of roughly 8 syllables. Thus "I went to the Garden of Love" is the same as "The wounded surgeon plies the steel", even though the beats for the lines are 3 and 4, respectively.
It should also have a stanza marker, (possibly 00?). Because it looks for patterns within patterns, stanzas are valued. Not all poetry has a stanza structure, but it arguably could. Deciding if stanzas are appropriate is partly a function of taking a syllabic divisor, breaking the poem down into number_of_lines, and looking at the divisors of that number.
It would need an added function to be able to recognize meaning patterns. For instance, repetition of proper nouns is the marker of plays. (From a meaning perspective, imo, plays is the ideal place to start because the marker is so easy to learn, and names all belong to a single set, and imply communication. It's no different functionally than any other identifier, and a concept all computers "understand".)
Eventually it would want to look for phonetic patterns, rhymes and near rhymes, which would also be indicators of potential good places for line breaks.
There is a very big data set that it can look at, and who knows what it might discern?
