Two levels, one question
Institutions are interested in student success because it affects both the people they teach and their own aims. Open education adds a question: if an institution adopts openly licensed material, do its students do better? The institutional measures and the student measures are linked, but the link is not a straight line. This article is about that connection and, above all, about the gap between association and cause.
What “student success” usually means in this context
There is no single definition. In adoption studies it commonly covers some combination of:
- passing a course, or earning a grade at or above a threshold;
- staying enrolled from term to term;
- progressing through a program and, eventually, graduating;
- in some cases, satisfaction or a sense of belonging.
At institutional level these are summarized as rates and averages. A statement such as “the pass rate in introductory courses rose” aggregates many individual stories, which can differ greatly. Some students may have benefited, some not at all and some may have done worse.
Why people expect a link
The reasoning behind a connection is plausible. If materials are free and available on the first day, some students may keep up more easily. If instructors can adapt material to their course, teaching may fit better. If an institution pays attention to course design while adopting, the whole course may improve. Any of these might lift student results, which would then show up in institution-level figures.
Each step is a hypothesis. Evidence on teaching methods is mixed, and a plausible mechanism is not a demonstrated one.
Correlation versus causation in adoption studies
Imagine a hypothetical university in which courses using open materials have higher pass rates than courses that do not. There are several explanations besides the materials.
- Who adopts. Instructors who volunteer to adopt new materials may already be committed, well-organized teachers.
- Which courses. Open materials may exist for popular introductory subjects, which may differ from specialist courses in difficulty and student mix.
- Which students. Students may be steered into or choose particular sections.
- Parallel changes. Adoption may coincide with a redesign, new tutoring or revised assessments.
- Reverse direction. A department that already has strong results may feel more able to experiment.
The statistics describing this are covered by the general idea of correlation causation: two things moving together does not show one produced the other. An adoption study that cannot rule out these alternatives can describe a pattern but should not claim a cause.
Ways researchers try to get closer to cause
Comparison with similar courses. Matching adopters with non-adopters on subject, level and prior student attainment.
Statistical adjustment. Controlling for measured differences such as prior grades. This helps but cannot adjust for what was not measured, such as motivation.
Before-and-after with a comparison group. Showing that adopters changed more than non-adopters over the same period.
Natural or deliberate experiments. Sometimes sections are assigned by chance, or a policy rolls out to some courses first. These designs are stronger but uncommon.
Looking for dose-response patterns. If more extensive use of open material is associated with larger differences, that is mildly supportive, though it is not proof.
None of these removes uncertainty. Good studies say what they did and what remains unresolved.
A hypothetical illustration
A college reports that 80% of students in courses with open materials passed, compared with 72% in other courses. That is an eight-percentage-point gap. Before accepting it, consider the questions above. If the open-materials courses had a pass rate of 79% in the year before adoption, the change is about one percentage point, and most of the eight-point gap existed already. The figures are invented, but the lesson is real: a difference in levels is not the same as a change caused by the program.
Linking the levels sensibly
To connect institutional results with student success, it helps to track students, not only course averages. Following cohorts shows who stayed, who left and where. Disaggregating results by student group can show whether gains are shared or concentrated, and analytics of this type, used with care for privacy, can reveal where support is needed. Learning analytics offers tools for this, with its own limits.
It is also worth remembering the role of other factors in student success measures. Advising, financial aid, schedule flexibility and course design all affect outcomes and may interact with materials.
What to take from adoption research
A reasonable reading is that open materials are generally compatible with similar or somewhat better student results, that cost is the most consistent advantage, and that stronger claims need careful designs. Where a report says openness “improved student success”, ask what the comparison was, how groups were formed and whether other changes were occurring. Honest studies describe their limits, and honest readers allow for them.