The problem
A large UK university was paying for a US-hosted tool to predict which students might drop out. It sat outside the university's Microsoft systems, its reasons were only given per group, and the data left the UK. An earlier rebuild had been trained to copy the old tool's risk bands rather than real outcomes, and was tested in a way that flattered it. The student support team needed a list they could trust: when they called the first group of students in September, those students had to be genuinely at risk.
What I did
- Led the data science and model build, working with a Microsoft data partner on the university's Microsoft Fabric platform.
- Rebuilt the model around what actually happened to students (official continuation records), using enrolment records, prior marks, Blackboard activity and timetabled attendance. Protected characteristics and deprivation measures were deliberately left out.
- Tested every version the honest way: train on past years, predict a year the model has never seen.
- Gave every student a calibrated risk score and the top reasons behind it, sorted into tiers sized to what the support team can actually handle (Urgent 50, Active 50, Watch 200).
- Made it a rolling model that re-scores every student each fortnight and feeds the results into the university's CRM (Dynamics 365).
- Cut the scoring run from just over 2 hours to 11 minutes, and handed over a full pack: model card, runbook and acceptance tests.
The result
- Tested on a year it had never seen, all 50 students it ranked most urgent went on to leave, and 294 of the top 300.
- ROC-AUC of 0.91 on unseen data, with run-to-run variation of about 0.001.
- Owned by the university, hosted in the UK, explainable for every student.
- Live: the first scoring run on current students went out on 12 October 2026.