The British Lord Kelvin once said: “When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.” This quote is often reproduced (with minor variations) in introductory textbooks on statistics and engineering, as a sort of encouragement I suppose. But is it true? For some, anything that can’t be counted and expressed in numbers is just not worth talking about. This is the “quantitative view” prevalent in much of contemporary science. Why take it? Because otherwise the ground for disagreement is just too great on any but the most trivial of observations, proponents say. Numbers are sharp and words are soft.
The opposite is the “qualitative view.” Proponents of the qualitative view claim that it is rare that you can summarize in numbers the things that are of real human interest. It’s the user experience that matters, isn’t it? Numbers can be, and often are, used as snake oil. People who do this don’t usually care about how the numbers were obtained (the so-called “counting rules”). The neat way data can be stacked in tables and graphed in three dimensions makes it look hard and objective. However if bits of the data are obtained by using different counting rules, it’s
actually nonsense to combine it.
What is a Counting Rule?
Counting is the art of assigning numbers to states or events in the world. For instance, measuring the length of your desk in inches, or timing someone logging in to your website. In science, we use such counting rules to communicate between ourselves how to assign numbers to states or events in a way that is reliable and accurate, so that anyone can replicate our work, so we all know what we’re talking about.
What do the Numbers Mean?
Here are some issues that are worth raising before we jump in with a ready-made interpretation of our numbers (such as “clearly, a menu in need of re-design.”)
- You can tweak counting rules—did you know that? So unless you explain clearly in words how you counted, all you have left are the brute figures, which may have been counted in different ways and, therefore, mean different things.
- How accurately have we counted? No raw statistic is interpretable without an estimate of its accuracy. An expert witness in a court of law is often asked, “What is the margin of error?”
- How representative is this data? Controlling the sample is vital in polls when it’s important to distinguish between various shades of opinion in the population. In Human Computer Interaction (HCI), we assume that the range of variation between users is minimal—so are our typically small sample sizes adequate?
- What statistical methods are used? Excel computes a mean and calls it an average. If the data is symmetrical around the central point, then the mean is fine as an average. If there are a few excessively high (or low) data points, then the mean is a poor estimator of the average. Darrell Huff’s famous How to Lie with Statistics showed that the methods we use to process data affect how we interpret it.
What we have to grasp firmly in HCI is that we are an engineering discipline. The field of statistics has developed to help scientists build, support, and demolish theories. We use statistics in a descriptive way to get an objective view of what is going on. Theoretical statistical models are all well and good, but to parody Wittgenstein, the data that you gather “is everything that is the case.” How did we manage to survive for all
those millennia before we invented statistics? We used common sense, counted where it made sense to count, and observed when we
knew that counting rules would fail. Let’s not forget that attitude of mind. I rather fear Lord Kelvin did.