Does tutoring work? It depends on the test

Abstract tests based on principles, yes; stuff that takes years to learn, no

A reader sends me this question:

Hi Emil, I'm wondering what your opinion is on the question of whether reading comprehension skills can be improved through tutoring. I'm an American tutor who works primarily with students on the ACT and SAT. In my experience, students can meaningfully improve on the English and Math sections, since success there partially depends on prior knowledge of grammatical rules and mathematical subject matter. In contrast, tutoring them in Reading seems mostly pointless - the most I seem to be able to do is show them how critical it is to read questions and answer choices carefully. Other than giving them general tips, there doesn't seem to be a way to actually teach them "how" to read better, since the psychological process of reading is not really dependent on conscious operations. My company stresses the importance of annotation and all sorts of other "tricks", but in my view it's just marketing bullshit to make parents think that there's actually something that can be done in this area.

Are you familiar with any literature that specifically addresses this question?

Thank you,
X

It's hard to say. There's not that much proper research on tutoring specifically. Most of it focuses on the SAT and ACT. This kind of research suggests the typical tutoring experience yields about 20 points, so maybe 0.20 d or thereabouts. ACT has a review and a large-scale study here, but of course they might be partial to claiming not much can be done. Their study found 0.71 score points gain with a SD of 5.6, so 0.13 d. Pretty small, probably because most students don't receive significant tutoring (maybe a few hours reading a guide). They also looked into the effect of private tutoring specifically, by number of hours. Their table:

Look in the bottom where they examined 10 different tutoring activities, including some of their own products ("ACT Online Prep"), but the only thing that stood out was private tutor/consultant. They say:

Among the 10 test preparation activates investigated, only the number of hours working with a private tutor or consultant had a significant impact on retest scores. Using the reduced model, those who did not engage in this test preparation activity had an adjusted mean ACT Composite score of 24.31 while those who had 11 hours or more of this test preparation activity had an adjusted mean score of 24.91. Pair-wise comparisons of the four bins that measured the amount of time allocated to a tutor or consultant showed that students who participated in 11 hours or more with a tutor or consultant had a statistically higher ACT Composite re-test score than those who did not spend any hours on the activity. No other comparisons were found statistically different.

So, if you want to gain a lot of points on ACT-like tests, be prepared to spend a lot of time grinding them, Asian tier. This 2014 review of similar studies also found a gain of about 0.15 d.

As you may know, intelligence researchers have done a lot of research into how much one can gain on tests with coaching or training. As you say, the tests that are based on knowing a set of rules or principles to solve questions show the strongest effects. E.g., this recent study with figure matrices (similar to Raven's) found that students who watched a small video explaining the principles gained some 18 IQ (1.20 d).

But reading comprehensive, vocabulary etc. reflect knowledge that takes a long time to build up. Some short intensive tutoring probably cannot do so much about such scores aside from small tips to guess better. Neither can various kinds of 'brain training' which is mostly working memory (n-back style). None of these do anything at all for unrelated tests because they cannot increase general intelligence. False advertisement -- indeed, one such company had to pay a big fine for this.

I am not specifically familiar with attempts to teach reading comprehension as such, but I had a look on Google Scholar. First hit was this n=70 randomized study. Reading comprehension didn't really improve from their tutoring, but study too small to really say much. Usually though, there are lots of studies done on small children which usually find that one can boost any kind of test score, but in the short-term. However, these gains fade-away over time (fade-out effect):

When large-scale studies are done where no statistics cheating is done, these gains usually fade away after 2-3 years. The US Congress sponsored a such study with thousands of preschoolers (age 3-4) and found the same thing (Head Start Impact Study and Follow-Up).

These kinds of short-term boosts to reading comprehension would still be valuable for adults taking high-stakes tests, but it's probably better to spend time on the tests that are easier to improve (analogies, matrices, number series). Reading comprehension is probably best improved by taking a lot of such tests, possibly 100s if one wants to gain a significant improvement. I imagine most people wouldn't care enough to do that.