The concept of measurement can be daunting. As a research and evaluation consultant I am often asked whether my approach will be ‘statistically significant’ or how reliable the outcomes will be. Here are a few basic elements of evaluation and measurement which should hopefully address some of these questions.
The concept of measurement
The definition of measurement itself is widely misunderstood. To understand the concept of measurement means a lot more things become measurable. It is not uncommon for people to think that measurement means meeting some nearly unachievable level of certainty, in which case, few things are measurable. In reality, measurement is not a certainty with no room for error. A better way of thinking of measurement is that it is a quantitatively expressed reduction of uncertainty based on one or more observations. When measuring anything, including conducting population surveys, unless the sample represents a 100% count of every person in the population, there will always be some risk of error, although the aim is to know how to reduce the risk of error.
The reduction of uncertainty is a critical concept to business. Major decisions can be made more easily, even if just slightly, by reducing uncertainty. In addition, a measurement doesn’t have to be about a quantity in the way we normally think of it. The uncertainty has to be quantified, but the subject of the observation might not be a quantity itself; it could be entirely qualitative, such as ways in which a regional town has been transformed.
Object of measurement
The thing being measured is not well defined. Sloppy and ambiguous language gets in the way of measurement. Even when the concept of measurement is well understood, some things seem immeasurable because we simply don’t know what we are trying to find out. For example, I have just completed an evaluation of the Small Town Transformations initiative for Regional Arts Victoria. It was important to clearly and fully understand what was intended by the concept of ‘transformation’. In this case, transformation was defined as the extent to which the arts contributed to the creative economy of the small towns, the building of their cultural identity and community development, and the extent to which the arts project left a lasting legacy. These measurement outcomes are not simply arrived at by economic figures or counts. They are complex, interrelated and convoluted concepts. Yet, they can still be measured to an extent that makes the outcomes meaningful.
Method of measurement
Many procedures of empirical observation are not well known. To become familiar with some of these basic methods makes it more apparent that many things though to be immeasurable are not only measurable but may already have been measured. Some things may seem immeasurable because the person considering the measurement is not aware of basic measurement methods- such as various sampling procedures or types of controlled experiments- that can be used to solve the problem. A common objection to measurement is that the problem is unique and has never been measured before. But it should be encouraging to know that several proven measurement methods can be used for a variety of issues to help measure something you may have at first considered immeasurable.
Most approaches to measurement are just variations on basic methods involving different types of sampling and experimental controls, and sometimes, choosing to focus on different types of questions.
So what?
Measurement is not an absolute. To consider measuring something of importance is about knowing how the measurement outcome intends to be used, knowing more than you knew before, assigning values that make sense and that are measurable, and being entirely clear about what it is you are wanting to measure in the first place.