Why academic achievement gets so much attention
Of all the outcomes people care about, grades and test results are the most visible. They are numbers, they are recorded by almost every institution, and they matter to students for practical reasons. It is no surprise that the question “does open education change academic achievement?” is asked more often than questions about confidence or later opportunities, which other articles on this site cover.
This article stays narrowly on achievement: grades, pass rates, and scores on tests and assignments. The aim is to explain how these are usually compared when materials are open, what can distort the comparison, and how modest the conclusions often need to be.
What counts as achievement data
Typical measures include:
- final course grades
- pass rates, meaning the share of students above a pass threshold
- scores on common exams or shared assignments
- results on standardized tests, where they are used
- grade point averages across terms
Each has strengths. A shared exam set across sections gives the most direct comparison, while final grades may include marks for participation or attendance that vary by teacher.
How comparisons are usually made
The most common design compares a course before and after switching to open materials, or compares sections of the same course that used different materials. Researchers then look at average scores or pass rates in each group.
A hypothetical illustration, with invented numbers: suppose a course had 200 students, and 150 passed, in the year before a switch, and 160 of 200 passed in the year after. The pass rate rises from 150 out of 200, which is 75 percent, to 160 out of 200, which is 80 percent. That is a five-point gain. It is tempting to credit the new materials. But before doing so, ask what else changed: the teacher, the exam, the intake of students, the weighting of coursework. A change this size is also small enough that a different year’s intake could produce it by chance.
Things that distort achievement comparisons
Who ends up in each group
If students pick their section, those choosing open-materials sections may differ in motivation or schedule. Differences in results may reflect those differences. Dropouts matter too: if weaker students withdraw from one version and not the other, the remaining group’s average rises for reasons unrelated to teaching.
The assessment itself
A different exam, or a revised marking scheme, can shift scores more than any material. Comparisons are most trustworthy when the same assessment, marked in the same way, is used for both groups. Even then, grades are relative to a particular course and cannot be compared across institutions without care.
Averages hide spread
A mean can stay unchanged while the distribution changes: more high scorers and more low scorers, for instance. Looking at the median, or at the share above and below a threshold, gives a fuller view, and the reasoning in averages explained applies directly. Reporting the range of results, not only the middle, is a sign of careful work.
Small differences and chance
Class results fluctuate naturally from year to year. A difference of a few points may lie within that normal variation. Researchers use statistical tests to judge this, but an ordinary reader can still ask whether the group was large and whether the result was repeated. The idea of measurement uncertainty is relevant: every score is an imperfect snapshot of what a student knows.
Teaching changes alongside materials
Adopting open materials often goes hand in hand with redesign, because teachers adapt content. If achievement improves, it may owe more to better course design than to openness itself. The reverse can also happen, where a poorly managed transition lowers results temporarily.
What the picture generally looks like
Reported comparisons are often described as showing results similar to those with conventional materials, with some studies finding modest improvements and some finding small declines. A careful summary is that moving to open materials does not usually appear to harm achievement, while firm evidence that it raises achievement is limited and varies by context. Subject, level of study, quality of the chosen materials and the support given to students all plausibly matter.
One mechanism is often suggested: students who might otherwise skip buying a book can read from the first week. This is plausible, and it connects to the cost argument, but how far it converts into higher marks is uncertain and depends on whether students actually use the materials.
What grades cannot tell you
Even well-run achievement studies have a limited scope. A grade does not say whether a learner will remember the material in a year, apply it, or continue studying. A test may reward recall over deeper understanding. Approaches like authentic assessment aim to judge performance on realistic tasks, and results from such tasks may tell a different story from a multiple-choice exam. For many learners, especially adults studying independently, formal grades are not even available.
For a broader look at progress, confidence and later opportunities, the wider definitions of success discussed elsewhere on this site are a useful complement.
Reading an achievement claim
When you see a statement such as “students using open materials scored higher”, it helps to ask:
- Higher than whom, and how were the groups formed?
- By how much, and is that difference large relative to normal year-to-year variation?
- Was the same assessment used for both groups?
- Did the same share of students finish in each group?
- Has the finding been seen in more than one course or institution?
If the answers are unclear, hold the claim loosely. Achievement is an important outcome and worth measuring, but a single number rarely settles whether a set of materials helped.