Building data quality into the systems staff use every day
Flexible data capture systems let errors straight through, so every report started with a cleanup. We redesigned the databases so staff are guided to valid data as they enter it.
- improvement in data quality
- ~90%improvement in data quality
- errors prevented, not fixed later
- At entryerrors prevented, not fixed later
- rework for staff and analysts
- Lessrework for staff and analysts
The challenge
Research organisations capture large volumes of data in systems designed to be flexible. That flexibility has a cost: the systems rarely enforce the business rules staff are expected to follow, so incomplete or inconsistent entries go straight in.
The problems only surfaced when reports were produced. Analysts then traced and corrected records after the fact, and the same issues returned the next cycle, because nothing had changed at the point of entry.
Before and after
Before
- Systems accepted entries that broke business rules
- Rules lived in staff knowledge and guidance documents, not in the system
- Data corrected repeatedly after reports exposed problems
- Report quality depended on how much cleanup time was available
After
- Business rules built into data capture
- Staff guided to valid entries as they work
- Most data arrives clean, so reporting starts from a reliable base
- Time once spent on corrections returned to core work
Our approach
- 01
Review the existing databases
We reviewed how data was captured, where errors started, and which rules staff were expected to apply.
- 02
Capture the unwritten rules
We documented the business rules and logic that existed only in practice, and confirmed them with the people who rely on the data.
- 03
Redesign the systems
We redesigned the databases so those rules apply as data is entered, making the correct entry the easiest one.
- 04
Support the change
We worked with staff as they adopted the new design, so it fitted the way they already work.
The outcome
Data quality improved by about 90%. Staff achieve more with less effort, because the system now does much of the checking for them.
Analysts start from data they can trust, and reports are no longer held back by the same recurring errors.
