Stop Measuring Activity and Start Measuring Results
The Military Community Nonprofit Industrial Complex
The Good, the Bad, and the Ugly
Day 6: Stop Measuring Activity and Start Measuring Results
Nonprofits love numbers.
We served 25,000 veterans.
We conducted 700 congressional meetings.
Our advocates sent 250,000 emails.
We distributed 50,000 resource guides.
We trained 4,000 volunteers.
Our campaign reached 3 million people.
Those numbers sound impressive.
And they may represent an extraordinary amount of work.
But there is another question we should be asking:
What happened because of it?
Because doing a lot of something does not automatically mean the something worked.
Welcome to the glamorous world of outputs versus outcomes.
Try to contain your excitement.
Activity Matters
Let's start with the positive.
Organizations absolutely should track activity.
How many people did you serve?
How many calls did you answer?
How many claims did you assist with?
How many veterans attended the employment program?
How many volunteers completed training?
How many congressional offices did you meet with?
How many people received emergency assistance?
How many referrals did you make?
Those numbers help organizations understand workload, staffing needs, reach, demand, and program growth.
Funders need them.
Boards need them.
Managers need them.
Staff need them.
Donors may find them useful.
If a program served 500 people last year and 5,000 this year, leadership should probably know that before everyone wonders why the staff looks exhausted.
Activity metrics matter.
They just don't tell the whole story.
Outputs and Outcomes Are Different
Here's the simplest version.
An output is what you did.
An outcome is what happened because you did it.
You held 100 employment workshops.
That's an output.
How many participants found employment?
That's an outcome.
You distributed 10,000 benefits guides.
Output.
How many people used the information to access benefits they were eligible for?
Outcome.
You trained 500 peer mentors.
Output.
Did participants receiving peer support report better access to resources, stronger connections, or whatever result the program was designed to achieve?
Outcome.
You held 300 congressional meetings.
Output.
Did the policy change?
Outcome.
Both matter.
But one is considerably easier to count.
Guess which one ends up in the infographic.
Big Numbers Photograph Well
There is a reason organizations love activity metrics.
They're easy to communicate.
100,000 advocacy actions!
Looks great in giant letters.
Put an American flag behind it.
Maybe a silhouette of a service member.
Throw the logo in the corner.
Done.
Outcomes are messier.
Sometimes the answer is:
“We don't know yet.”
Or:
“The legislation didn't pass, but we gained 40 cosponsors and secured a hearing.”
Or:
“Only 37 percent of participants completed the program.”
Or:
“The pilot showed that half of what we designed didn't work, so we're changing it.”
Those don't always fit nicely onto a fundraising graphic.
But they can tell you considerably more about whether the organization is learning and improving.
“People Served” Can Mean Almost Anything
This is one of my favorite nonprofit statistics.
People served.
What does that mean?
Did someone receive six months of intensive case management?
Did someone receive $5,000 in emergency assistance?
Did someone attend a two-hour workshop?
Did someone receive a referral?
Did someone download a PDF?
Did someone walk past your booth at a resource fair and take a pen?
Depending on how an organization defines the metric, all of those could potentially become some version of “served.”
That doesn't necessarily mean anyone is being dishonest.
Organizations need practical ways to count program activity.
But donors, boards, and community members should understand what the metric means before treating it as evidence of impact.
“We served 50,000 veterans” sounds impressive.
Tell me what “served” means.
Now we can have a useful conversation.
Reach Is Not Impact
Social media has made this even more interesting.
An advocacy campaign reaches one million people.
Excellent.
What does “reached” mean?
Did one million people read the information?
Click the link?
Watch the video?
Contact Congress?
Change their understanding of the issue?
Take any action whatsoever?
Or did the content technically appear somewhere on their screen while they were scrolling past a recipe video?
Reach has value.
Awareness has value.
Communications metrics have value.
But visibility and impact are not synonyms.
Going viral can spread an important message.
It can also produce 700,000 views and absolutely nothing else.
Advocacy Is Especially Difficult to Measure
This is where I have some sympathy for advocacy organizations.
Measuring legislative impact is hard.
Really hard.
Suppose 30 organizations spend three years advocating for legislation.
Members of Congress are also hearing from constituents.
Committee staff are researching the issue.
Government agencies provide technical assistance.
Personal stories influence lawmakers.
Professional lobbyists are involved.
Political conditions change.
Then the bill passes.
Who gets credit?
Probably everyone sends an email saying:
“WE DID IT!”
But determining exactly how much influence any one organization had may be impossible.
Advocacy rarely has a clean cause-and-effect relationship.
That doesn't mean organizations shouldn't measure it.
It means they should be honest about what the measurements show.
Congressional Meetings Are Not Legislative Victories
This one deserves special attention.
Organizations often report how many congressional meetings they conducted.
That's useful.
Relationships matter.
Education matters.
Meetings can absolutely move policy.
But:
A meeting is not an outcome.
Who did you meet with?
Was it the relevant legislative staff?
Did the office understand the issue afterward?
Did the member cosponsor?
Did the office request additional information?
Did committee staff engage?
Did legislative language change?
Was a hearing scheduled?
Did an amendment get introduced?
Did opposition soften?
Did the meeting identify the actual obstacle preventing the bill from moving?
A hundred productive meetings may be extraordinary advocacy.
A thousand meetings where everyone politely nods, accepts a one-pager, and forgets you existed five minutes later?
That's a very impressive travel schedule.
Advocacy Emails Are Tools, Not Trophies
The same applies to electronic advocacy.
“We generated 200,000 emails to Congress.”
Great.
What happened?
Did offices respond?
Did congressional support change?
Did cosponsorship increase?
Did committee leadership take notice?
Did the messages reach the right offices?
Did advocates personalize them?
Were lawmakers receiving meaningful constituent communication or 20,000 identical messages?
Volume can matter.
A large number of constituents contacting Congress can demonstrate political interest.
But eventually advocates should ask whether increasing the number from 200,000 to 300,000 actually changes anything.
Sometimes the answer is yes.
Sometimes it might be better to generate 500 highly informed constituent contacts with strategically important offices.
More is not automatically better.
Effective is better.
Direct-Service Programs Need Outcomes Too
This isn't just an advocacy problem.
Consider employment programs.
An organization reports:
“We helped 5,000 veterans with employment services.”
Wonderful.
How many found jobs?
How long did it take?
Were those jobs full-time?
Part-time?
Temporary?
Did wages improve?
Did people remain employed six months later?
Did the program help participants who would otherwise have struggled to find employment?
Those questions help determine whether the program is actually accomplishing its mission.
The same principle applies to housing, education, financial assistance, mental-health programs, caregiver services, peer support, and almost everything else.
The outcome should connect to the problem the program exists to solve.
Not Everything Valuable Can Be Easily Counted
Now for an important warning.
Do not become so obsessed with outcomes that we pretend everything meaningful fits neatly into a spreadsheet.
Some nonprofit work is difficult to quantify.
What is the numerical value of a peer mentor answering the phone when a surviving spouse is having the worst night of their life?
How do you quantify someone finally feeling understood?
How do you measure the value of a volunteer spending two hours helping a veteran navigate a bureaucracy?
How do you calculate the long-term impact of connecting a military spouse with the right resource at exactly the right time?
Qualitative outcomes matter too.
Stories matter.
Feedback matters.
Interviews matter.
Case studies matter.
Human experience matters.
Data should help us understand impact, not erase everything that cannot easily be converted into a percentage.
Beware of Vanity Metrics
A vanity metric is essentially a number that looks impressive but tells you very little about whether you're accomplishing the mission.
Followers.
Impressions.
Website visits.
Emails sent.
Brochures distributed.
Event attendance.
Meetings held.
Media mentions.
Those metrics can be useful.
They become vanity metrics when organizations use them primarily to demonstrate success without connecting them to meaningful outcomes.
Ten million social-media impressions might be phenomenal if the goal is public awareness.
They may be nearly meaningless if the goal is getting 200 veterans into stable housing.
Measure what matters to the mission.
Set the Goal Before You Measure It
Organizations sometimes decide what success means after seeing the numbers.
That is convenient.
It is also backwards.
Before launching a program, ask:
What are we trying to change?
Then decide how you'll know whether it changed.
If you're launching an employment program, define success.
If you're creating a peer-support program, define success.
If you're running an advocacy campaign, define what progress looks like.
If you're holding training, define what participants should know or be able to do afterward.
That doesn't mean goals can never change.
Programs evolve.
Unexpected things happen.
But having a target makes evaluation considerably more meaningful.
Otherwise every result becomes a success because there was never a standard for failure.
Failure Is Data
This may be one of the hardest concepts in the nonprofit world.
A program can fail.
An advocacy campaign can fail.
An event can fail.
A strategy can fail.
That doesn't necessarily mean the organization failed.
Sometimes failure tells you exactly what needs to change.
Maybe participants weren't using the program.
Why?
Maybe the eligibility criteria were too restrictive.
Maybe outreach was reaching the wrong population.
Maybe the service duplicated something already available.
Maybe congressional offices weren't responding to the advocacy message.
Why?
Maybe the request was unclear.
Maybe the data wasn't convincing.
Maybe advocates were targeting the wrong offices.
Organizations that measure outcomes can learn from failure.
Organizations that only measure activity can always declare victory.
“We held 37 meetings!”
Lovely.
The strategy still didn't work.
Now what did we learn?
Donors Should Want Honest Results
Donors can accidentally contribute to bad measurement too.
We like impressive numbers.
We like success stories.
We like knowing our donation accomplished something.
Nobody particularly enjoys receiving an annual report that says:
“We tried this. It didn't work.”
But sometimes that may be evidence of a healthy organization.
Especially if the next sentence is:
“Here's what we learned, and here's what we're changing.”
Organizations should not be punished for honest evaluation.
If we demand that every program always be portrayed as wildly successful, we shouldn't be surprised when annual reports begin resembling campaign advertisements.
Boards Should Ask Better Questions
Boards have an important role here.
Not just:
How many people did we serve?
But:
What changed for them?
Not just:
How much did we spend?
But:
What did the spending accomplish?
Not just:
How many donors did we gain?
But:
Are donors staying?
Not just:
How many advocacy actions occurred?
But:
Did congressional behavior change?
Not just:
Did the program grow?
But:
Should it have?
Growth is not automatically success.
Sometimes serving fewer people more effectively is better than serving more people badly.
Don't Manipulate the Measurement
Here's where things can get ugly.
Organizations can choose definitions that make results look better.
They can count the same person multiple ways.
They can highlight percentages while hiding the denominator.
They can report people “reached” instead of people helped.
They can select favorable time periods.
They can celebrate outputs while avoiding outcomes.
They can quietly change the definition of success after a program underperforms.
And sometimes the statistics are technically accurate while still creating a misleading impression.
That isn't responsible communication.
Data interpretation is legitimate. Data manipulation is not.
If you have to make the metric confusing for it to look impressive, perhaps the metric isn't impressive.
Compare Cost With Outcome
Remember Wednesday's discussion about Form 990s?
This is where finances and outcomes meet.
Suppose two programs each spend $1 million.
Program A serves 10,000 people.
Program B serves 2,000.
Which is better?
You don't know yet.
Maybe Program B provides intensive services producing life-changing outcomes while Program A distributes informational materials.
Cost per person tells you something.
It doesn't tell you everything.
The better question is:
What result did the investment produce?
That is where nonprofit financial analysis becomes meaningful.
Not simply:
“They spent $5 million.”
But:
“They spent $5 million accomplishing what?”
Good Organizations Measure to Learn
The best reason to measure results isn't fundraising.
It isn't the annual report.
It isn't impressing the board.
It's learning.
What works?
What doesn't?
Who are we reaching?
Who are we missing?
Where are people dropping out?
Which services produce the strongest results?
Where are we wasting resources?
What should we expand?
What should we change?
What should we stop doing?
Good measurement helps organizations become better at the mission.
That's the point.
Good, Bad, and Ugly
The good: Organizations track activity and outcomes, combine quantitative data with human experience, learn from failures, improve programs, and communicate results honestly.
The bad: Organizations can become overly focused on easily measured activity because meaningful outcomes are harder, slower, and more complicated to evaluate.
The ugly: Vanity metrics can be used to create the appearance of extraordinary impact while avoiding the much harder question of whether anything actually changed.
Big numbers aren't bad.
They just need meaning.
Stop Counting Long Enough to Measure
Count the veterans served.
Count the congressional meetings.
Count the advocacy emails.
Count the volunteers.
Count the resource guides.
Count the website visitors.
Count the dollars raised.
Those numbers matter.
Then keep going.
Did people receive the help they needed?
Did their situation improve?
Did the program solve the problem it was designed to solve?
Did Congress move?
Did policy change?
Did participants find employment?
Did families become more stable?
Did the organization learn anything?
Did the investment produce a meaningful result?
Because activity tells us how busy an organization was.
Outcomes tell us whether the work mattered.
And sometimes the answer will be:
“We don't know yet.”
That's okay.
Sometimes the answer will be:
“It didn't work.”
That's okay too, assuming we learn something and change course.
What isn't okay is pretending activity automatically equals impact because the numbers look fantastic on an infographic.
So celebrate the 500 meetings.
Celebrate the 100,000 emails.
Celebrate the 50,000 people “reached.”
Then ask the question that should make every good nonprofit curious and every questionable one slightly uncomfortable:
What changed?
Because the mission isn't to stay busy.
The mission is to make a difference.
And if your biggest accomplishment is the number of times you told people about your accomplishments...
We may need another metric.
