What this article does and does not do
A literature review normally summarizes specific studies. This article does not, because it would be easy to get names, dates and findings wrong, and it is not necessary for the purpose. Instead, it describes the kinds of research that exist on open educational resources, what each tends to measure, and how a non-specialist can judge a particular paper or summary they come across.
If you want the specific studies, the best route is to search a library database and read the originals, or a recent review article, rather than rely on secondhand summaries, including this one.
The main types of research
Cost and savings studies
These estimate how much students or institutions spend on materials and what a switch to open resources might save. They are usually the clearest in design, because costs are countable. Their weaknesses are the assumptions: how many students would really have bought the book, whether used or rented copies are counted, and whether the savings are for one course or across a program.
Perception and attitude surveys
Many studies ask students or teachers what they think of open materials: whether they found them useful, of good quality, or easy to use. These are common because they are cheap to run. The findings tend to be positive or neutral, but the usual cautions apply. People who respond may be more interested in the topic, answers reflect opinion rather than learning, and a student who has never used the alternative may not have a basis for comparison.
Learning outcome comparisons
Here researchers compare grades, pass rates or test scores between students using open materials and those using conventional ones. These are the studies people most want, and also the hardest to do well. The challenge is separating the effect of the material from everything else that differs between groups.
Use and adoption studies
These look at who uses open resources, how often and why, and what stops them. They draw mostly on surveys and interviews, and they are the source of much of the general picture of the field. They describe behavior rather than benefit.
Case studies and descriptions
Reports from one course, department or project describe what was done and what happened. They can be rich in detail and useful for practitioners. They cannot show that the same results would occur elsewhere, and authors who were involved in a project may be inclined to describe it favorably.
Policy and theory papers
Conceptual and policy writing discusses definitions, ethics and strategies. This is a legitimate and large part of the literature, but it contains arguments rather than evidence, and it should be read that way.
Recurring weaknesses
Several limits come up often enough to be worth keeping in mind.
- Small samples. Studies of one class or one institution may not apply elsewhere.
- No real comparison group. Without a comparison, an apparent improvement might have happened anyway.
- Differences between groups. If students choose the course section, or if different instructors teach each version, the results may reflect those choices.
- Outcomes that are easy to measure. Satisfaction and cost are easy to measure; deeper learning is harder, so it is measured less often.
- Short time frames. A single term may tell us little about long-term effects.
- Publication bias. Studies with positive results are generally more likely to be written up and published.
- Mixed definitions. “Open” may mean a free online text in one study and an adaptable, openly licensed one in another.
Correlation and causation
A study may find that students who used open materials did better. That is not the same as showing the materials caused the improvement. Perhaps instructors who adopt open materials are also more engaged, or students with more free money to spend would have bought books but did worse for other reasons. The distinction between correlation and causation is one of the most useful ideas to hold when reading this literature.
Questions to ask of any study
- What exactly was compared with what?
- How many students or teachers were involved, and how were they chosen?
- Were the groups similar at the start?
- What was measured: opinions, cost, grades, completion?
- Over what period?
- Who did the research, and who funded it?
- Do the authors discuss the limits of their own work?
These are the habits that anyone reading papers outside their own specialism can use. The skills involved are similar to those taught under source evaluation: look at who is speaking, what they offer as evidence, and what they leave out.
Where the evidence is thin
On the whole, the research base is better at describing use and cost than at demonstrating learning effects. Areas that look less well covered include long-term outcomes, effects on particular groups of students, results outside the countries and languages that dominate publication, and the effects of open practices such as students creating content. Another thin area is research on quality, since few studies check the accuracy and teaching value of materials in a systematic way.
None of this implies that open materials do not work. It means the case for them rests partly on reasoned arguments, such as lower cost and flexibility, and partly on modest, mixed evidence. A reader should be comfortable with that.
Using this literature in practice
If you are deciding whether to adopt open materials, you probably do not need a definitive study. A more practical approach is to look at the particular resource, check its authorship and date, try it with a small group, and gather your own information on how it works. Published research can help frame the questions but rarely answers them for your own students and context.
Finding reliable reviews
Systematic reviews and meta-analyses, which collect and assess many studies by a stated method, are generally more trustworthy than a single paper, though they are only as good as the studies they include. Library databases and scholarly search tools can find them. Looking at several recent reviews, and noticing where they agree and where they note gaps, gives a fairer picture than any one source.