What this article does and does not do
This is not a summary of particular named studies. It sets out how to judge the reliability of the kinds of evidence that usually appear when open education and retention are discussed, and where those kinds of evidence tend to fall short. Where numbers appear below, they are invented purely to illustrate an idea and are labeled that way.
The kinds of evidence you will meet
Claims about retention and open materials usually rest on one of a few types of source.
- Institutional reports. A college switches some courses to free texts and compares persistence or pass rates with earlier years or with other courses.
- Student surveys. Learners are asked whether cost affected their choices or whether they feel more likely to continue.
- Comparative studies. Sections using open texts are compared with sections using commercial ones, sometimes with statistical adjustment for differences between students.
- Large data analyses. Records from many courses or institutions are analyzed to look for patterns.
- Reviews that summarize several studies.
Each has a different strength. A survey reveals what students say; institutional records show what happened; a review shows whether several studies point the same way.
Typical weaknesses
Self-selection. If students choose a section, they may differ from those who choose another. Perhaps they check costs more carefully, or enroll earlier, or have different schedules. Those differences, and not the materials, could explain part of any gap.
Other changes at the same time. Courses that adopt open texts are often being redesigned in other ways, taught by enthusiastic instructors, or supported by a grant. If so, it is hard to credit the textbook.
Narrow outcomes. Many studies measure completion rates for a single course: did the student finish, or pass? That is not the same as staying enrolled for several terms or finishing a degree. A gain in one course may not carry over.
Small samples. With few students, differences can appear or vanish by chance. A result based on one class of thirty is very different from one based on several thousand.
Selective reporting. Positive results about student retention are more likely to be written up and publicized than neutral ones. A reader seeing only favorable reports may get a distorted picture.
Weak comparison groups. Comparing this year with last year assumes nothing else changed. Enrollment shifts, economic conditions and new support services can all alter persistence.
Who is counted. If students who withdrew early are excluded from the analysis, the result may look better than reality. Whether withdrawals are counted needs to be stated.
A hypothetical example
Suppose, purely for illustration, a college reports that 80 of 100 students in courses using free texts finished the term, compared with 70 of 100 in other courses. That is a ten-percentage-point gap. It sounds notable, but the questions are immediate. Were the 100 students in each group alike? Did the free-text courses have lower prices that attracted different learners? Did the same instructors teach both? With only 100 students per group, how likely is such a gap by chance? Without answers, the figure establishes very little.
Now suppose the college also tracked the same students for the following term and found little difference. The first-term gain might not persist, which would change how the result should be read.
Correlation and causation
Many retention findings are associations. Students in courses with open materials persisted more, but that does not show the materials caused it. The distinction is familiar from any area of research and is worth keeping in mind: an association can arise because of a third factor, such as the type of course or the type of student, or because the cause runs the other way. Strong causal evidence usually requires comparing groups that are alike apart from the materials, ideally through random assignment or very careful statistical adjustment. Readers new to reading papers can start with the methods section, which usually says how groups were formed.
Signs of a more reliable study
Without suggesting any single study is flawless, some features generally make a finding easier to trust:
- It states clearly how students ended up in each group.
- It compares groups on prior grades or similar measures at the start.
- It follows students for longer than one term.
- It reports the numbers, not only percentages, and the uncertainty around them.
- It includes students who withdrew.
- It discusses results that did not fit the hypothesis.
- Similar results appear in different institutions, subjects and years.
Consistency across settings is probably the most persuasive feature. One striking result is a prompt for further work; many modest results pointing the same way are a stronger basis.
What can reasonably be said
Taking the weaknesses together, a cautious summary is available. Evidence that open materials raise retention is limited and mixed. Some reports find modest gains, particularly for students with limited finances; others find little difference. Many of the strongest-sounding claims rest on weak comparisons. There is little evidence that open materials harm persistence.
It is also reasonable to say that the lack of strong evidence is not evidence that nothing is happening. It may reflect how hard retention is to study, because it depends on so many factors that no single policy would be expected to explain much of it.
Using this as a reader
When you meet a claim, work through a short list. What was measured? Who was compared with whom? How many people? For how long? Did anything else change? Who produced the report and what might they want to show? If these answers are not given, treat the claim as a lead, not a conclusion.
For anyone running a program, the best contribution to the evidence is to record outcomes carefully before and after a change, keep comparison groups where possible, and report neutral results as readily as positive ones.