Coherence Quiz answers

As promised, the results of yesterday's little experiment on "Coherence of sentence sequences" are here.

A tabular summary:

So the survey respondents (as a whole) guessed the original order of all twelve sentence-pairs correctly — though the margins varied from 2-to-1 to 99-to-1. The overall percent correct was 89%, though of course that percentage will depend on the particular mix of examples.

(The counts don't all sum to the same row-wise value because a couple of participants left some answers blank — there's probably a way to get Qualtrics to prevent that, but I didn't figure it out in time…)

One of the cases that people had trouble with, Pair 5, comes from a NYT story from 7/8/1998, where the second sentence is the first of three specific instances of the generalization stated in the first one::

Pay no attention to the soaring stock price of Rambus Inc., which has shot up 54 percent in the past two weeks.

Geoff Tate, the president and chief executive of Mountain View-based Rambus, shrugs off the rise as Wall Street's odd response to "old news."

"We try to ignore the stock price when it goes up or down," said Tate, 43. "When our stock price goes up, I am quick to remind everybody here, `This is great, but don't think this means we got smarter today. We're the same people we were before."'

Rambus is an anomaly in many ways. In a chip industry that's hitting a rocky spell, Rambus is on a roll. Rambus doesn't even make anything; it licenses its technology to chipmakers, and encourages computer makers to use chips made with Rambus technology.

In a stock market that rewards companies for short-term earnings, Rambus keeps its eye on the horizon. Yet unlike other companies whose stocks levitate on promises of future profits — firms like Yahoo and Amazon.com spring to mind — Rambus operates in the black, and has ever since its initial public offering last year.

The other troublesome case, Pair 8, comes from a passage in chapter 12 of Wuthering Heights, where Mrs. Linton is enacting insanity:

She could not bear the notion which I had put into her head of Mr. Linton’s philosophical resignation.  Tossing about, she increased her feverish bewilderment to madness, and tore the pillow with her teeth; then raising herself up all burning, desired that I would open the window.  We were in the middle of winter, the wind blew strong from the north-east, and I objected.  Both the expressions flitting over her face, and the changes of her moods, began to alarm me terribly; and brought to my recollection her former illness, and the doctor’s injunction that she should not be crossed.  A minute previously she was violent; now, supported on one arm, and not noticing my refusal to obey her, she seemed to find childish diversion in pulling the feathers from the rents she had just made, and ranging them on the sheet according to their different species: her mind had strayed to other associations.

‘That’s a turkey’s,’ she murmured to herself; ‘and this is a wild duck’s; and this is a pigeon’s.  Ah, they put pigeons’ feathers in the pillows—no wonder I couldn’t die!  Let me take care to throw it on the floor when I lie down.  And here is a moor-cock’s; and this—I should know it among a thousand—it’s a lapwing’s.  Bonny bird; wheeling over our heads in the middle of the moor.  It wanted to get to its nest, for the clouds had touched the swells, and it felt rain coming.  This feather was picked up from the heath, the bird was not shot: we saw its nest in the winter, full of little skeletons.  Heathcliff set a trap over it, and the old ones dared not come.  I made him promise he’d never shoot a lapwing after that, and he didn’t.  Yes, here are more!  Did he shoot my lapwings, Nelly?  Are they red, any of them?  Let me look.’

‘Give over with that baby-work!’ I interrupted, dragging the pillow away, and turning the holes towards the mattress, for she was removing its contents by handfuls.  ‘Lie down and shut your eyes: you’re wandering.  There’s a mess!  The down is flying about like snow.’

I went here and there collecting it. 

‘I see in you, Nelly,’ she continued dreamily, ‘an aged woman: you have grey hair and bent shoulders.  This bed is the fairy cave under Penistone crags, and you are gathering elf-bolts to hurt our heifers; pretending, while I am near, that they are only locks of wool.  That’s what you’ll come to fifty years hence: I know you are not so now.  I’m not wandering: you’re mistaken, or else I should believe you really were that withered hag, and I should think I was under Penistone Crags; and I’m conscious it’s night, and there are two candles on the table making the black press shine like jet.’

‘The black press? where is that?’ I asked.  ‘You are talking in your sleep!’

‘It’s against the wall, as it always is,’ she replied.  ‘It does appear odd—I see a face in it!’

‘There’s no press in the room, and never was,’ said I, resuming my seat, and looping up the curtain that I might watch her.

‘Don’t you see that face?’ she inquired, gazing earnestly at the mirror.

And say what I could, I was incapable of making her comprehend it to be her own; so I rose and covered it with a shawl.

‘It’s behind there still!’ she pursued, anxiously.  ‘And it stirred.  Who is it?  I hope it will not come out when you are gone!  Oh!  Nelly, the room is haunted!  I’m afraid of being alone!’

I took her hand in mine, and bid her be composed; for a succession of shudders convulsed her frame, and she would keep straining her gaze towards the glass.

‘There’s nobody here!’ I insisted.  ‘It was yourself, Mrs. Linton: you knew it a while since.’

‘Myself!’ she gasped, ‘and the clock is striking twelve!  It’s true, then! that’s dreadful!’

Her fingers clutched the clothes, and gathered them over her eyes.  I attempted to steal to the door with an intention of calling her husband; but I was summoned back by a piercing shriek—the shawl had dropped from the frame.

Overall, I find it surprising and interesting that random out-of-context sentence pairs are so often more coherently interpretable in the original order than in reversed order — even if we eliminate examples with obvious reference-chain issues.

As one of the commenters noted, this task seems likely to be a difficult one for automatic analysis — and that's what we've found in trying some of the techniques recommended in the literature, as well as various improvements.

I suspect that there are differences among authors and genres in how easy or difficult it is to infer the original order of random sequences from their works. And there are probably individual differences in how good people are at making such guesses.