Two numbers that carry a lot of weight
Research on open education and student success leans heavily on two measures: how many people complete a course, and what grades or scores they earn. Both are attractive because they are recorded routinely. Both are also easy to misunderstand. This article is about how they are produced and why comparing them is harder than it looks.
No specific studies are quoted here. The aim is to help you read whichever figures you encounter.
How completion is calculated
The basic formula is the number who completed divided by the number who started. Everything depends on what goes into each side.
The numerator might be people who passed, people who finished all the assessed work, people who watched the final lecture or people who earned a certificate. These are quite different.
The denominator might be everyone who registered, everyone who logged in at least once, everyone still enrolled after a deadline or everyone who attempted the first assignment.
A hypothetical example: 1,000 people register for a free course. Of these, 400 open the first lesson, and 100 complete the assessed work. Completion could be reported as 10% (100 of 1,000), 25% (100 of 400) or something else entirely, depending on the denominator. All of these are arithmetically correct. They answer different questions.
This matters most in open courses, where registering costs nothing and many people sign up to browse, with no intention of finishing. Low completion in such a course may not signal failure at all. In a credit-bearing course where students pay and intend to finish, the same rate would mean something very different.
How grades are used
Grades seem more straightforward, but they are less comparable than they appear.
- Different assessments. Two courses, or the same course in different years, may use different exams or marking schemes.
- Different graders. Marking standards vary between instructors.
- Only completers get grades. If weaker students leave before assessment, the average grade among those who remain will tend to rise. A better average may therefore reflect who stayed, not better learning.
- Grade scales compress. Many courses give most students a narrow range of grades, which makes small differences hard to detect.
When reading averages, remember what an average hides. Averages explained simply: the mean can be pulled up by a few high scores, and the median or the spread may tell a different story.
Why comparisons are tricky
A typical research question is whether students using open materials or taking an open course do as well as, or better than, comparison students. Several obstacles recur.
- Selection. People who choose open options differ from those who do not. Some are experienced learners who already hold qualifications, which tends to raise completion. Others face time or money constraints, which tends to lower it.
- Intent. Learners have different goals. Someone who wants to check one topic and leaves is counted as a non-completer.
- Context. A free self-paced course, a university course with open materials and a school class using shared resources are very different settings.
- Time. Completion at the end of a term and completion within a year are different numbers.
- Confounding. Other things change when materials change, including instructors, assessments and support.
This is the standard challenge of correlation causation. A higher completion rate alongside open materials does not show that the materials caused it.
Measuring uncertainty
Every figure comes with error. A small sample can produce a completion rate that shifts noticeably if the study is repeated. Measurement uncertainty is not only a laboratory concern: survey and enrollment data have gaps, misclassified records and rounding.
Ask for the sample size. A reported 60% from 10 learners (6 of 10) is far less informative than 60% from 1,000. Both are 60%, but the first could easily have been 40% or 80% on another day.
Reading charts and tables critically
Charts showing completion or grades can mislead. A bar chart whose vertical axis starts at 50% instead of zero can make a few points look dramatic. Pay attention to the scale, whether groups are comparable and whether the chart shows the number of people, not just percentages. Misleading charts are common even in well-meant reports.
What more reliable evidence looks like
- Clear, written definitions of the numerator and denominator.
- Comparison groups that are similar at the start, ideally with some form of random assignment or statistical adjustment.
- The same assessment used for both groups, or an explanation of how differences were handled.
- Follow-up over a meaningful period.
- Reporting of results across subgroups, with caution about small numbers.
- Open discussion of limitations.
Putting it together
The research on completion and grades suggests that open materials are often associated with similar grades to conventional materials, and that completion in free, open online courses is frequently low when measured against everyone who registered. Both statements need heavy context. They do not show that open education fails or succeeds. When you meet a figure, ask who is counted, what is compared and what else could explain the difference, and be cautious about drawing firm conclusions from one study.