What counts as a student outcome
An outcome is something that happens to a student as a result of, or alongside, their learning. In studies of open education the usual candidates are grades, completion, satisfaction and, less often, later results such as progression to the next course. Each measures something slightly different, and none captures everything a person gains from learning.
This article explains how each is measured, the common study types and the limits that should temper any conclusion. It is general information, not a verdict on any particular program.
Grades and test scores
How measured. Course grades, exam marks or standardized assessments, usually compared between students who used open materials and those who used conventional ones.
Strengths. Widely available and familiar. They connect to what institutions and students care about.
Limits.
- Assessments and grading standards differ between instructors and terms.
- Grades reflect many things besides materials: prior knowledge, effort, teaching and exam style.
- Only students who stay to the end get final grades, so leavers are missing.
- Grades can compress into a narrow band, making small differences hard to detect.
When you read a reported average difference, it helps to know how averages explained in basic statistics can mislead. A mean can be affected by a few extreme scores, and a one-point average difference may be too small to matter in practice.
Completion and passing
How measured. The proportion of students who finish a course or pass it, divided by those who started.
Strengths. Captures whether students reached the end, which grades alone cannot.
Limits. The denominator is a choice: registered students, active students or those who sat the first assessment. Course completion is covered in detail elsewhere on this site. The key point here is that two studies may report very different completion rates for similar courses simply because they count differently.
Satisfaction and perceived learning
How measured. Surveys asking students how they felt about the materials, the course or their own learning.
Strengths. Capture experience, such as usability, cost relief or accessibility problems that grades will not show.
Limits. Satisfaction is subjective, response rates are often low and responders may not represent everyone. Students may report they learned a lot without a matching change in test results, or the other way round. Student feedback surveys are most useful as a guide to what to fix, not as proof of learning.
Other outcomes sometimes studied
- Time to completion and credit accumulation. Whether students progress at a steady pace.
- Use patterns. How often and how long students access material, often from system logs. These show behavior but not learning.
- Later performance. How students do in follow-on courses, which is harder to track.
- Skills and confidence. Usually self-reported or measured through special tasks.
Types of studies
Randomized experiments. Students or sections are assigned by chance. The strongest design for cause and effect, but rare and typically limited to one setting.
Quasi-experimental comparisons. Existing groups are compared, often with adjustment for known differences. More common and reasonably informative, with the caveat that unmeasured differences remain.
Before-and-after studies. One course or institution is tracked across a change. Simple but vulnerable to anything else that changed.
Observational and correlational studies. Look for associations in existing data. Useful for hypotheses, weak for cause.
Surveys and interviews. Capture perceptions and context.
Systematic reviews. Combine many studies with explicit rules for what to include. They can give a broader view, though they inherit the weaknesses of the studies they pool.
A hypothetical example
Imagine a class of 120 students where 60 use an open textbook and 60 use a purchased one. The open-textbook group has an average mark of 71 and the other 69. A two-point difference is small. With 60 students per group it could easily arise by chance. If the open-textbook group also happened to come from a section that met in the morning with a different instructor, the difference might not reflect the textbook at all. The example is invented, but each of those questions applies to real studies.
Common cautions
- Correlation is not causation. A pattern in the data does not show why it occurred.
- Averages hide variation. Some students may do better and some worse.
- Context matters. Findings from one subject, level or country may not hold elsewhere.
- Positive results are more often published. The available literature may overstate typical effects.
- Outcomes interact. A cost saving might improve completion without changing grades.
Reading with care
If you are evaluating your own course, decide on your outcomes before you start, record a baseline and use the same measures afterwards. If you are reading someone else’s report, look for the design, the size of the groups, the way outcomes were defined and any discussion of limitations. A modest, carefully qualified finding is generally more trustworthy than a dramatic one.
Taken together, the evidence on student outcomes in open education is best described as mixed to modestly encouraging, strongest on cost and access and more uncertain on learning gains. Outcomes vary with context, and no outcome measure on its own tells the whole story.