Start with the goal, not the dashboard
An initiative can be anything from a department adopting an open textbook to a library program supporting staff who create openly licensed material. Before choosing indicators, write down what the initiative is meant to achieve. Lower costs for students? Wider use of shared material? Better course completion? Each goal points to different measures, and tracking everything at once tends to produce numbers nobody uses.
A useful habit is to pair every goal with one or two indicators and a stated baseline, meaning the starting position you will compare against.
Completion
Completion is the share of people who finish what they started. It is easy to calculate and easy to misread.
A hypothetical example: 400 learners enroll in a course and 300 complete it. Completion is 300 divided by 400, which is 75%. If the next year 800 enroll and 480 complete, completion is 480 divided by 800, which is 60%. The number of completers rose from 300 to 480 even though the rate fell, so which figure matters depends on your goal.
Pitfalls:
- Who counts as starting? Open courses can attract people who only browse. If they are counted as enrolled, rates look low without any failing on the course’s part.
- Different definitions. Finishing all assessments, reaching a score or viewing the last page are not the same thing.
- Course differences. Comparing a short voluntary course to a long credit-bearing one is unfair.
Course completion and graduation rates are covered in more detail elsewhere on this site, so treat this as the short version.
Reuse and adoption
Open material can be adopted, adapted and shared, so it makes sense to track how far it travels. Possible indicators include the number of courses or instructors using a resource, the number of adaptations, or downloads and page views.
Pitfalls:
- Downloads are not use. A download shows interest, not that anyone read or taught from the file.
- Counts depend on tracking. Material copied to other sites or shared by email is often invisible to the original publisher.
- Adoption is not quality. A resource can be widely used because it is free and familiar, not because it works well.
Where possible, supplement counts with short feedback from the people reusing the material.
Cost saved
Cost is often the headline measure, so it deserves care.
A hypothetical example: a course previously required a text costing 80 dollars, and 200 students take it. If every one of them would have bought the text new, the maximum saving is 200 times 80, which is 16,000 dollars. But if 50 of them would have bought a used copy at 40 dollars and another 30 would have borrowed one, the real figure is smaller. Counting 120 students at the full price (200 minus 50 minus 30) gives 120 times 80, which is 9,600 dollars, plus 50 times 40, which is 2,000 dollars, for a total of 11,600 dollars. The more realistic saving is therefore well below the maximum.
Pitfalls:
- Using the list price for everyone. Students buy used copies, rent, share or skip the book.
- Ignoring institutional cost. Staff time to select, adapt, host and update material is a real cost, even if it is not billed to students.
- One-time versus recurring. Setup costs fall over time, while maintenance continues.
The wider discussion of textbook costs and the economics of openness is covered elsewhere; here the point is simply to state your assumptions.
Learning results
Grades or assessment scores are tempting, but they are hard to compare across terms because exams, instructors and cohorts change. If you track them, compare like with like and be cautious about concluding that the initiative caused any change. Correlation causation is the trap: results and adoption can both move for unrelated reasons.
Experience and feedback
Surveys of learners and staff can show what is not working, such as missing sections, accessibility problems or confusing structure. Student feedback surveys are useful but only reflect those who respond, and response rates are often low. Report the response rate alongside any satisfaction figure.
Equity
An average can hide differences. If an initiative aims to widen access, check whether outcomes or use differ across groups, bearing in mind that small groups give noisy numbers and that privacy should be respected when reporting.
Practical rules for a sound evaluation
- Set the baseline before launch. You cannot show change without a starting point.
- Define each indicator in writing, including who is counted.
- Use a fair comparison such as a similar course or earlier cohort, and note what else changed.
- Mix numbers with explanations. A figure tells you what happened, comments help explain why.
- Report limits openly, including missing data and small samples.
- Review and adjust. If an indicator is not helping decisions, drop it.
Measures can also be gamed. If staff are judged on downloads, they may chase downloads. Choose indicators that reflect the real goal, and read them together rather than relying on any single number.