Studies of ereading
Studies about how reading paper books compares to reading electronic devices like a Kobo have gotten some attention in the media. Which makes sense because it’s often a chance to have a great moral panic about how technology is now ruining reading, or maybe even cognition. I’ve posted about technology panics before but ereader stories deserve a special, dishonourable mention.
Here’s the story from August 2014 in the Guardian: Readers absorb less on Kindles than on paper, study finds. I’ll quote a large chunk to set the context.
A new study which found that readers using a Kindle were “significantly” worse than paperback readers at recalling when events occurred in a mystery story is part of major new Europe-wide research looking at the impact of digitisation on the reading experience.
The study, presented in Italy at a conference last month and set to be published as a paper, gave 50 readers the same short story by Elizabeth George to read. Half read the 28-page story on a Kindle, and half in a paperback, with readers then tested on aspects of the story including objects, characters and settings.
Anne Mangen of Norway’s Stavanger University, a lead researcher on the study, thought academics might “find differences in the immersion facilitated by the device, in emotional responses” to the story. Her predictions were based on an earlier study comparing reading an upsetting short story on paper and on iPad. “In this study, we found that paper readers did report higher on measures having to do with empathy and transportation and immersion, and narrative coherence, than iPad readers,” said Mangen.
But instead, the performance was largely similar, except when it came to the timing of events in the story. “The Kindle readers performed significantly worse on the plot reconstruction measure, ie, when they were asked to place 14 events in the correct order.”
See any red flags yet?
A guide to crap stats
If you haven’t seen the paper False-Positive Psychology : Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant [PDF link] before, do check it out. Or at least the abstract or this blog post about it. The paper beautifully outlines many of the ways in which science can go wrong with bad statistics.
The authors start with a hypothesis that we know is false: that listening to music about old age makes you younger (in the literal, chronological sense). They then “prove” it to a degree of certainty that would get the paper published in a psychology journal. What they did was deliberately manipulate the researcher degrees of freedom in running the experiment, to show that you can prove literally anything with crap stats.
Here are the 4 main ways that were identified that could be used to prove anything (note that these don’t have to be deliberate and often are not:
- Measure many variables, report on the ones that happen to be significant
- Combine many of the measures available for test subjects, report on the ones that happen to be significant
- Slice and dice the subject pool in many different ways (eg. by different demographic combinations) until you get a statistically significant result for some of these.
- End your experiment as soon as you get a statistically significant result.
What these all have in common is that the researcher is going on a fishing expedition. If enough conditions are measured, some will be statistically significant. The standard for a lot of social science is a 5% probability that your results are due to chance, or 1 in 20. So if you measure 20 different things in a scattershot matter, you’re very likely to strike gold just from chance alone.
So, how do you prove that ereaders are worse than paper readers?
Based on the paper above, all you have to do is to compare lots of different aspects of reading, without identifying in advance which one you’re zeroing on. For example, you could compare ereaders and paper texts with respect to:
- Comprehension
- Empathy
- Transportation
- Immersion
- Character names
- Place names
- Event order
- etc.
What will then happen? Even if there’s no genuine difference between ereaders and paper reading, at least some of these are likely to have ereaders perform worse, with statistical significance. But then if you rerun the test or run a similar one, you’re likely to get different results and maybe ereaders will do worse on some other measure.
Which is sorta what the story reported. The previous study showed that “paper readers did report higher on measures having to do with empathy and transportation and immersion, and narrative coherence, than iPad readers”. But this study was about Kindles where “the performance was largely similar, except when it came to the timing of events in the story”.
Colour me unimpressed. Not to mention that there were only two experienced Kindle users in the study and people who are first using an ereader are likely to be distracted by novelty. There is more at this blog post which also has a chart of the findings and as expected for a study based on noise, there was a measure where ereader users scored 8 points higher than paper readers.
The studies around this are inconsistent and very preliminary. We need a helluva lot more evidence to be able to actually say that ereaders influence our reading experience in a substantially different (to say nothing of negative) way. It is something I’m just inherently skeptical about. Sure, the medium can influence things, but imagine hearing similar arguments but against the revolution of written language. I’ve got a sample of this kind of bullshit:
The medium of the book makes people read in isolation which decreases the activation of those parts of the brain we use for social contexts and decreases understanding of the text. It lets us skim over parts of the text (instead of listening to a story at the required pace) meaning we absorb less detail. Finally, it decouples the reading experience from the auditory experience making it easier for awkward turns of phrase to creep in, and decreasing appreciation for the beauty of language, thereby degrading culture. We should not read and write. And with some research funds, I can prove this!
0 Comments