What “effective” means here
When people ask whether open educational resources are effective, they usually mean one of two things. The first is “do students learn at least as well as they would with a conventional text?” The second is “do they bring other benefits, such as lower costs or better access, without making learning worse?” These are different questions with different kinds of evidence, and a surprising amount of confusion comes from mixing them up.
This article describes how researchers approach the first question, what designs they use, and how to interpret the usual findings. It does not cite specific studies. Where an example helps, it is labeled as hypothetical.
What a good comparison needs
To test whether a material works, you need three things: a clear definition of the outcome, a comparison group, and some way to make the comparison fair.
An outcome. Common choices are course grades, scores on a common exam, pass rates, course completion and student retention. Each has pitfalls. Grades depend on the instructor’s marking, pass rates depend on who enrolls, and completion can be affected by many things unrelated to the textbook.
A comparison. Researchers compare students using open materials with students using conventional ones. The comparison can be between sections taught in the same term, between terms, or between institutions.
Fairness. The challenge is that groups may differ in ways other than the material. This is the heart of the matter, and it is why the design of controlled experiments is worth understanding in its own right.
Common designs
Randomized trials. Students or classes are assigned by chance to one set of materials or the other. This is the strongest design for showing cause, because chance tends to balance other factors. It is also rare in education, because it is hard to arrange and sometimes raises fairness concerns.
Matched or statistically adjusted comparisons. Researchers compare students who used open materials with similar students who did not, adjusting for things like prior grades. This can reduce, but not eliminate, differences between groups.
Before-and-after comparisons. The same course is compared before and after a switch. Problems arise if something else changed at the same time, such as the instructor, the exam or the student intake.
Simple comparisons of sections. Researchers compare sections that happened to use different materials. These are the easiest to do and the easiest to misread, because instructors or students may have chosen the section for reasons linked to the outcome.
Surveys of perceived learning. Students are asked whether the materials helped. This tells you about perception, not measured learning.
A hypothetical example
Imagine, purely as an illustration, a course taught in two sections. Section A uses an open text and Section B uses a commercial one. At the end, Section A’s average on a common exam is 72 and Section B’s is 70.
It is tempting to say that the open text did better. But several questions should come first:
- Were students allocated at random, or did they choose their section?
- Did the same instructor teach both?
- How variable were the scores? If individual scores range widely, a difference of two points between averages could easily be chance.
- How many students were in each section?
With these unanswered, the honest reading is “no clear difference”, and a difference of two points on its own proves nothing. If many studies of this kind were gathered together and the differences were small and went in both directions, the most reasonable summary would be “roughly similar”.
What the findings tend to look like
Without naming individual studies, the broad pattern described by many reviewers is that students using open materials usually do about as well as those using conventional ones, with some studies showing small advantages, some showing none and a few showing disadvantages. A common interpretation is that the material matters less than how it is used, and that the main case for open resources is therefore about cost and access rather than a boost in learning. This is a general summary, and it may not apply to every subject or context.
Where studies report positive effects on completion rates or on student retention, a frequent suggestion is that lower costs matter: students who cannot afford a book may delay buying it or skip it. This is plausible, though the research is mixed and cause is hard to isolate. Cost relief can matter most for textbook costs among lower-income students, but not every study separates out the effect by income.
What the findings do not show
- They do not show open is better. “No significant difference” is a common finding and, for a cost-saving intervention, can be a satisfactory one. It is not the same as “better”.
- They do not apply to all resources. Open covers an enormous range of quality. A result for one text says little about another.
- They do not capture long-term learning. Most studies measure end-of-term results.
- They do not separate the license from the format. A free digital text may be used differently from a printed one, so any effect may come from the format.
- They do not prove causes. Unless the design supports it, differences could come from many factors.
Measuring more than grades
Some researchers look at other things: how often students read the material, whether they have the book in the first week, how many credits they take, or how they describe their experience. Learning analytics from digital platforms can give detailed data on use, though it raises privacy concerns and tells you about activity rather than understanding. Measurement itself is imperfect; ideas around measurement uncertainty apply to educational data too.
How to judge a claim of effectiveness
When you read that open resources “work”, check:
- what was compared and how fairly;
- how many students were included;
- whether the outcome was learning or something else;
- whether the authors state limits;
- whether the result appeared in more than one study.
A fair summary
The best-supported statement is a modest one: in many studies, open educational resources appear to support learning about as well as conventional ones, while often reducing cost. Stronger claims need stronger evidence than most studies currently offer, and results vary with context.