Why look at the institution as a unit
Student-level studies ask what happened to individual learners. Institutional studies ask a different question: what changed for the organization? Did costs shift, did enrollment change, did more students stay, did the way courses are developed change? Leaders usually need answers at this level because decisions about budgets, staffing and policy are made there.
Institutional outcomes are also harder to study. There is one institution, not hundreds of students, and many things change at once. This article describes the common measures and a way to compare fairly.
Measures worth considering
Direct and indirect costs. Students’ spending on course materials is the easiest to estimate, but the institution also bears costs, including staff time to select and adapt material, hosting, accessibility checks and updating. A fair view counts both. Over time, setup costs fall and maintenance continues. The wider economics of openness are covered elsewhere; here the key point is to state what is included.
Enrollment. Some institutions report that low-cost courses attract or keep students, but enrollment responds to demographics, pricing, competition and local conditions. It is rarely safe to credit it to open materials alone.
Retention and progression. The share of students who return, pass the course or move on to the next level. Definitions vary, so write down exactly how you count.
Course pass rates and withdrawals. These are routinely recorded and can show whether a course changed, though they are affected by grading practice.
Staff and process measures. The number of courses using open materials, the time it takes to develop a course, the proportion of staff who adapt rather than only adopt, and the proportion of material that meets accessibility expectations.
Reputation and partnership. Often discussed, rarely measured well. Counts of collaborations or external reuse can offer partial indicators, but they are not the same as reputation.
The central difficulty: what is the comparison?
To say an institution gained something, you need to know what would have happened otherwise. You cannot observe that directly, so you approximate it. Common approaches:
Before and after. Compare the same measures before and after adoption. Simple, but it assumes nothing else changed. A new advising scheme, a change in fees or a shift in the student population could all affect the results.
Compare adopting and non-adopting courses. Within the same institution, compare courses that adopted open materials with similar courses that did not. This controls for the institution but not for differences between the courses or instructors.
Compare institutions. Compare with similar institutions that did not adopt. This helps when institutions are truly alike, but they rarely are.
Combine approaches. Looking at change over time in both adopting and comparison groups, sometimes called a difference-in-differences design, is stronger than looking at either alone, though it still rests on assumptions.
A hypothetical example
Suppose a college has 2,000 students in introductory courses. Before an open materials program, 1,500 returned the following term, which is 75%. After, 1,700 of 2,000 returned, which is 85%. The change is ten percentage points.
It is tempting to credit the program. But suppose a comparison group of similar courses without open materials moved from 75% to 80% over the same period. The difference between the two changes is then five percentage points (10 minus 5), not ten. Half of the improvement happened anyway, perhaps because of a broader support scheme or a change in who enrolled. The example is invented, but it shows why a comparison group matters.
Designing a fair comparison
- Decide the question and outcome in advance. Choosing measures after seeing results invites selective reporting.
- Record the baseline across several terms, not just one, so normal fluctuation is visible.
- Choose comparable comparison groups by subject, level and student profile.
- List other changes happening in the same period.
- Define each measure in writing.
- Check group sizes. Small groups produce unstable percentages.
- Report uncertainty and limits.
- Share methods so others can check the reasoning.
Pitfalls to watch for
- Self-selected adopters. Enthusiastic instructors may be strong teachers regardless of materials.
- Moving definitions. If the institution changes how it counts retention, the trend breaks.
- Short windows. A one-term improvement might fade.
- Averages that hide gaps. Check results by student group, subject and level. Equity of opportunity is an important lens, and a headline average can conceal groups that did not benefit.
- Counting only successes. Include courses where adoption was dropped or did not work.
Reading the results honestly
Institution-level evidence is usually suggestive, not conclusive. Where an institution sees lower costs for students and similar or slightly better progression, that is consistent with the hope for open education, but it does not prove that openness was the cause. The most credible reports state their comparison, show their numbers, acknowledge other influences and describe what did not work. Outcomes vary across institutions, and a good evaluation helps a particular institution learn about its own situation instead of promising a general result.