CHIDOMASTER BLACK BELT · L6S

Statistics in practice · 25 February 2023

The chi-squared test — what it measures, and the fine print nobody reads

A test for counted things: complaints by site, passes by shift. It tells you whether a difference is bigger than chance — and stays silent about whether it matters.

Lots of operational data isn't measurements — it's counts in categories. Complaints by department. Pass/fail by shift. Yes/no answers by year. The chi-squared test (written , said "kye-squared") is the classic way to ask: are these counts different enough that chance alone probably didn't do it?

Chapter 1 · Statistics in practice

How it works, in plain words

First you work out what the counts would look like if nothing interesting were going on — if every shift, say, had the same underlying fail rate. Those are your "expected" counts. Then you compare what you actually saw against them:

Translated: for every category, take the gap between observed and expected, square it (so plus and minus gaps both count), scale it by the expected size (so big categories don't drown small ones), and add it all up. Small total: what you saw is about what chance would give you. Big total: something beyond chance is probably going on.

Chapter 2 · Statistics in practice

The fine print, which is where the trouble lives

  • "Beyond chance" is not "big enough to matter". The total grows with the amount of data. With a hundred thousand records, utterly trivial differences pass the test. Always look at the actual percentages next to the verdict, and ask: would this difference change any decision?
  • It needs enough data in each category. As a rule of thumb, every expected count should be at least 5. Sparse tables need a different test, not wishful thinking.
  • Two hundred forms from the same ten people are really ten opinions. The test assumes each count is independent.
  • *It never tells you why.* A significant result says "this pattern is unlikely to be luck". Which category drives it, and what's causing it — that's your job, not the test's.

Chapter 3 · Statistics in practice

Verdict

Chi-squared is a good first filter and a poor final answer. Use it to stop yourself over-reading noise in a table. Never let it substitute for looking at the table and thinking.

Talk this through with me

A free 30-minute discovery call — no commitment, no jargon.

Book a discovery call