Skip to content
FocalIDFocalID

Seven data quality checks to run before an AI project

AI projects rarely fail because of the model. They fail because the data was incomplete, duplicated or out of date, and nobody found out until late. These seven checks take days, not months, and tell you what to fix first.

By the FocalID team in Colombo, , 2 min read

A printed report with bar charts beside a calculator and a laptop.

The seven checks

  • Completeness: for the fields the project depends on, what share of records are empty? A customer table with half the phone numbers missing cannot drive a messaging agent.
  • Duplicates: is the same customer, product or supplier entered more than once under slightly different names?
  • Consistency: do codes, units, dates and names follow one format, and do two systems agree on the same record?
  • Validity: are values possible? Look for negative quantities, dates in the future and prices of zero.
  • Freshness: how old is the data when it is used? Yesterday’s stock figure is useless for confirming today’s order.
  • History: is there enough past data, and was it recorded the same way throughout? A change of system often breaks the series.
  • Access: can the data be read by software, through an API or database, or only seen on a screen and exported by hand?

How to run them

Take one real extract of each table the project needs. Count empty values per field, group by the fields that should be unique and compare totals between systems. Then show a sample of the oddities to the people who enter the data. They usually know the cause in minutes: a mandatory field filled with a dot, a branch that uses its own product codes.

Fix at the source

Cleaning data inside the AI project gives you one clean copy and a source that keeps producing errors. Where you can, fix the cause: make the field mandatory, add a validation rule, merge the duplicates and name an owner for each key table. Then keep the checks running automatically inside your data pipeline, so new problems are caught as they appear.

Good enough to start

Data never becomes perfect, and waiting for it to do so delays the project for ever. Aim for the fields this project needs, in the tables this project reads, at a quality you have measured. Write down what is still wrong so the people testing the system know what to expect.

Common questions

How long does a data quality check take?

For the handful of tables one project needs, usually a few days, including a session with the people who enter the data.

Can AI clean the data for us?

It can help find duplicates and odd values, but people who know the business must decide what is correct, and the cause should be fixed at the source.

What if our data is in spreadsheets?

The same checks apply. Moving the key spreadsheets into a proper system is often the first project.

Related