This is part 2 in a series on program evaluation.  If you haven’t read part 1 yet, you can find it here.

It’s easy to design an evaluation poorly.

We don’t always know which questions to ask, what areas to explore, who to talk to, or which tools to use.  We know that we want things to improve, but uncovering and diagnosing the real hurdles that stand in our way can be surprisingly complex.

Survey questions can be misworded, we can measure the wrong areas, the right people might not participate, we could misinterpret what we find – any number of factors can disrupt our good intentions.

Despite the difficulty, evaluation design is a crucial skill to master if you want your organization to move forward.  The reason for this is simple:

If you want to know what your next step is, you have to know where you are.

If you want to know what your next step is, you have to know where you are. Click To Tweet

Most groups know the value of articulating their mission and determining their vision of the future – they want to know where they’re going.  This is good and important work, but it’s only half the equation.

If you were planning a trip, knowing your destination is obviously pretty key.  But there is another element that we often take for granted.  Before we can move anywhere, we have to know our current location.  In our modern GPS world, this is rarely an issue, so we forget how vital it is.  Without being able to identify ourselves on a map, we don’t know if we need to head north, south, east, or west.  Any step we take could be in the wrong direction.

It’s not enough to know where we want to end up, we also have to know where we’re starting from.

This is where a properly designed evaluation comes in.

 

Measures and Methods

Once you’ve laid some of the groundwork and you know why you need to do an evaluation (see part 1 in the series), your two primary tasks will be to determine your methods and measures.

Methods are concerned with how you’ll gather the information you need, while measures are all about what information is needed.

To visualize the difference, imagine evaluating a building project or home renovation.  What you’ll measure includes things like length, height, square, and level.  How you’ll measure is with tools such as a measuring tape, framing square, laser level, etc.

The what determines the how.

In the evaluation world, methods include things like surveys, interviews, focus groups, observation, records, etc.  Don’t start here.  Until you know what information you need, you’re not ready to choose a method.  Each method is best suited for a particular purpose – we’ll figure out that step later.

Today, we’re talking about measures.

What you need to measure will depend on your unique context and what you’re trying to accomplish.

In other words, I can’t tell you precisely what you should measure in your setting, but I can give you a few guiding points to get you thinking in the right direction.

 

1. Don’t measure everything

It’s easy to get carried away once you start measuring stuff. 

Sometimes, we just start measuring all sorts of things because we know that’s what “good leaders” do, right?  We’re like a 4-year old on a construction site who’s just been given their first tape measure – running around measuring everything that’s within reach.

Don’t be that 4-year old.  You’ll only end up with a mountain of data that you won’t know what to do with.  I know everyone is in love with data today, but if you’re excessive on this, the analysis phase later will be unnecessarily time-intensive and the chances of implementation will plummet.

The goal is action. 

An evaluation should help you make meaningful changes in your setting, not just to produce interesting bar graphs. Click To Tweet

Imagine your evaluation to be somewhat like a visit to a doctor’s office.  He or she will determine what to measure based on the symptoms you’re experiencing.  There is a particular area of health that they’re trying to improve.  At first, all they might want to know is your heart rate, blood pressure, and temperature.  That’s the first wave of measurement.  Based on what they discover, they’ll dig deeper and determine new measurements for wave two.  They don’t start with the endoscopy.

Reflect

• What are the broad, critical measurements you could make to give you a picture of your overall health?  I.e. What are the “vitals” of your organization?
• What are the “symptoms” your organization is experiencing?  What information do you need to understand what’s going on at a deeper level?

 

2. Measure what matters

When you limit what you measure, it helps you to clarify what really matters.  What are the elements in your program that are the best indicators of success?  What would you have to see that would give you a sense of whether or not you’re “winning the game?”

The right term here is validity.

In order for a measure to be valid, it has to clearly affect the outcome that we’re looking for. 

• If you’re building a home, you could measure the amount of sawdust produced during your project, but that doesn’t tell you anything about the quality of your build, does it?
• If you’re running a teen drop-in centre, you don’t need to record how many teens come in every week wearing black shirts.  It might be interesting, but it’s trivial. 

Again, this is where it’s important to be absolutely clear on what you’re trying to get out of this evaluation.  You’ll have a hard time choosing valid measurements if your goals are vague.

• If you’re really hoping to discover where the breakdown in communication exists among your team members, you don’t need to measure the volume of sales calls being made.
• If you need to learn why innovation isn’t happening in your organization, you don’t need a customer satisfaction survey.

Focus. You’re not looking to generate random data here.  You want measurements that clearly connect what you’re doing with the outcomes that you’re hoping for.

Whatever your particular measurements are, ensure that they are critical indicators of success.

Don’t just measure what’s interesting. Measure what matters. Click To Tweet

Reflect

• What particular outcomes are you hoping for in the area that you’re evaluating?
• Based on experience and external data (e.g. research studies), what are the behaviours most likely to produce those outcomes?  How can you measure the extent to which these behaviours are present in your organization?

 

3. Measure what you can control

Some of the easiest things to measure are those which we can’t control.

Think about two of the most common measurements in organizations – attendance and revenue.  Easy to track but impossible to control.  You can’t force people to show up to your event, and you can’t force people to give you money.  You only have an indirect influence over these numbers.  They may fluctuate significantly due to factors outside of your control.

The creators of the 4DX system call these “lag measures” because they are something you can quantify only after the fact.  By the time you know the number, you can’t do anything about it.  The event is over.  The fiscal quarter has passed.

Now, we still want to track lag measures, but we also need to be measuring the behaviours that are most likely to move those numbers in the right direction.

We need to be thinking of “lead measures.”

Unlike a lag measure, a lead measure is an action that is in your control.

For example, if you had a goal of losing weight, stepping on the scale would provide you with your lag measure.  Knowing this number would do nothing to improve it, however.  The lead measures, in this case, would involve tracking the frequency and intensity of physical movement in your day as well as the quality and quantity of the food you’re taking in.

The combination of tracking both lead and lag measures is powerful.  Not only do you now have a behaviour-based measurement that you can directly influence, but you also have a way of seeing which actions have the most potential to move the needle.  It becomes an ongoing research study on how to improve.  If a particular set of lead measures don’t get results, you simply tweak them or swap them out for something else and see if it makes a difference.  Additionally, you can track these numbers on the fly and don’t need to wait till after the fact.  Long before the financial quarter is over, you can monitor your lead measures to gauge your performance.

Here’s a bonus I’ve discovered through this: Measuring what you can control guards against beating yourself up over things that you can’t.  Don’t lose sleep over the things that are out of your hands.

Measuring what you can control guards against beating yourself up over things that you can’t. Don’t lose sleep over the things that are out of your hands. Click To Tweet

Reflect

• What are the most important lag measures in the area that you’re evaluating?
• What are the behaviours that you or your team can engage in that are most likely to move those numbers in the right direction?  Find a way to track those behaviours and create accountability measures to ensure they get executed.

 

4. Don’t turn measures into targets

Here’s a caution before you go too much further.  Even with valid measurements, you can derail your entire team if you focus too narrowly on them.

Let me introduce you to two important, related laws of evaluation.

Goodhart’s Law:

“When a measure becomes a target, it ceases to be a good measure.”

Campbell’s Law:

“The more a metric is used, the more likely it is to corrupt the process it is intended to monitor.”

As soon as we start to measure something, we tend to give a disproportionate amount of energy to anything that might move that measurement, and we can lose sight of the larger purpose behind our work.  A few examples:

• When advertisers start measuring the click-through-rate (CTR) of their online ads, it can easily cause them to begin creating “click-bait” ads that attract attention but don’t actually engage with the intended audience.
• “Teaching for the test.” A standardized test is a measurement tool, and it doesn’t take long before teachers know how to adjust their teaching content in order to help their students perform better on these tests.  Rather than increasing the learning quality of the classroom, it may deteriorate as they laser focus on the results of one test.
• The clean-up of the California wildfires in 2017 cost four times as much as previous clean-ups. Why?  Partly because contractors were paid according to the weight of the material that they could excavate in any given day.  This measurement led to shady practices like loading up their trucks with heavy, wet mud, mixing metals that should have been recycled elsewhere, and digging unnecessarily deep holes in order to inflate the numbers.  They then needed to spend over $3 million dollars to refill the holes that were left by these initial contractors.

Examples of people “gaming the system” and inflating numbers are everywhere.  In the business world, it could be sales quotas and billable hours.  In the non-profit world, it’s attendance and donations.  We can actually learn how to increase these numbers without becoming more effective as an organization.  In fact, by focusing on certain measures like these, we may even be neglecting more important indicators of success.

What do we do with this reality?  We can’t just stop measuring things because of this danger, but we can take some steps to counteract the effects of these two laws.

• Take multiple measurements. Don’t zero in on one area at a time, but have a variety of assessments.  Again, you don’t want too many measurements, but you do want enough to create a system of checks and balances.
• Adjust the frequency of how often you review certain metrics. If attendance at a weekly event isn’t the best indicator for whether a church is accomplishing its mission, then you might not want to review those numbers every week.  Maybe you only need an attendance update every quarter (or an alert when the numbers fluctuate by more than a predetermined percentage).
• Measure what matters. If your goal is increased sales, don’t just track CTRs, but rather look at what a customer does after they’ve clicked on your ad.  As a church, don’t just look at attendance, but monitor how many people are taking the next step into smaller circles of community such as ministry teams and small groups.
• Stay skeptical and frequently reassess the measures you’re using and what effect they’re having. Don’t be afraid to mix it up and try something else.  Just because a group down the street is tracking something doesn’t mean you need to be.

Reflect

• Are there metrics in your organization that have succumb to one of these laws?  What do you need to adjust to ensure that you don’t lose focus on the bigger picture?

 

Next Steps

Once you have a good idea as to what information you need to clarify where you are, then you can begin thinking about how to gather that information.  Keep your eyes open for a future article where we’ll walk through the various evaluation methods that exist and when to use them.

If you want to make sure you don’t miss out on future articles, sign up to the email list here.

Thanks for being here,

Dan