Measuring the impact of ideas
We would like to reproduce in full an article published a few years ago, so it can hardly be described as “news”. Nevertheless, we believe it sets the tone and provides guidance for how think tanks should approach fundraising. Far from simply…
We would like to reproduce in full an article published a few years ago, so it can hardly be described as “news”. Nevertheless, we believe it sets the tone and provides guidance for how think tanks should approach fundraising. Far from simply seeking funding, think tanks must first deliver value through their work and then be able to demonstrate it by measuring their impact.
The article was first published by Arthur C. Brooks, then president of the American Enterprise Institute, in the March–April 2018 issue of Harvard Business Review (pp. 37–42). Brooks recalls how, after the Great Recession of 2008, the Washington think tank had to compete harder than ever for every dollar.
AEI’s output is straightforward: books, research papers, opinion articles, media appearances and so on. Determining its impact, however, meant measuring its competitive position in the marketplace of ideas.
Brooks describes two of AEI’s metrics: how many opinion articles its scholars publish in leading newspapers, and how often Congress calls on those scholars to testify. Compared with its four principal competitors, AEI accounted for 36% of opinion articles in 2015–2017 and the highest number of congressional testimonies, by a wide margin, in 2009–2017.
Article by Arthur C. Brooks, also published by the American Enterprise Institute.
In the summer of 2008, I was happily working as a professor at Syracuse University when I received an unexpected telephone call. For the previous year, the American Enterprise Institute, one of the country’s oldest and best-known think tanks, with which I had a part-time affiliation, had been searching for a new president. Would I be willing to be considered?
The think-tank industry is very small, so it has no established leadership pipeline. Boards are never entirely sure what sort of people should run these organisations, and executive searches are often difficult. I happen to know that I was not the first choice—or the second or third. For years I had taught and written about fundraising and nonprofit management, but I had never actually done either. I am convinced that the last thing AEI’s directors said before offering me the job must have been, “Oh, what the hell, let’s give him a chance.” But I was fortunate. They made the offer. I accepted.
Becoming a chief executive for the first time is challenging enough. But I faced more than the usual learning curve: between accepting the position in mid-2008 and starting the following January, the nonprofit economy collapsed as a result of the Great Recession. Inflation-adjusted charitable giving fell by almost 10% between 2008 and 2009, much of it during the final quarter of the year. AEI relies entirely on charitable donations, accepting neither government grants nor contract research. Its revenues therefore plummeted just as I walked through the door. The institute had to compete for every dollar as never before. My new colleagues and I had to show donors why their shrinking philanthropic investments should come to us rather than to others, and how investing in our work would produce a tangible impact.
Even before the recession, demonstrating impact was a growing concern in the nonprofit sector. For-profit businesses can quantify their impact through measures such as sales and shareholder returns, but nonprofits cannot. This is doubly true for organisations working in the realm of ideas, like AEI. What is our metric—intelligent thoughts per hour? Yet modern philanthropists, many of whom have made great fortunes in analytics-driven technology companies, demand evidence that their charitable dollars are working well. Mark Zuckerberg and others like him will not give to organisations that swallow their money without producing measurable results. Their generation insists on evidence that their giving creates value. They need to see data. Having an intangible product is no excuse.
Demonstrate impact or fail: that was my first great challenge as AEI’s president. It cost me many sleepless nights in my early years. But meeting that challenge proved to be about more than personal survival in a new role. It fundamentally changed how we operate.
A golden opportunity
Leaving a tenured job for an uncertain future as a nonprofit manager might seem reckless. In fact, it was consistent with the patterns I had established earlier in life.
I grew up in Seattle in a family of artists and academics. I showed an aptitude for music at an early age and soon found my way to the French horn. I tried university after high school, but left after a year and set out with my horn to earn a living as a classical and jazz musician in the United States and Europe.
A few years into my musical career, I met a woman from Barcelona and took a job with an orchestra in Spain, hoping to persuade her to marry me. We did not share a language when we met, but having recently celebrated our 26th anniversary, with three children approaching adulthood, I am pleased to report that our communication skills have improved.
I loved being a musician, but not having finished my education always bothered me, and that feeling grew stronger over time. While still living in Spain, I therefore returned to university through correspondence courses.
A month before my 30th birthday, I went to my mailbox and received my bachelor’s degree in economics. Shortly afterwards, I left music entirely and returned to the United States for a master’s in economics. That led to a doctorate in public policy analysis, which led to work as a university professor. Eventually, I arrived at Syracuse.
During my decade in academia, I taught many management and MBA students, and much of my research concerned nonprofit management and social entrepreneurship. I wrote a textbook on the subject. One central question kept recurring: how do nonprofits, apart from those offering the most tangible products or services, know that they create real value? The field, myself included, never seemed to have satisfactory answers. We spoke a great deal about “social return on investment” and the “double bottom line”, but these were more theoretical concepts than practical guidance. Moving to AEI seemed a golden opportunity to solve those theoretical puzzles through the real world of management.
But joining AEI was also a deeply personal decision: I sincerely believed in the organisation’s fundamental principles. AEI is strictly nonpartisan, avoiding political entanglements and institutional positions. What unites its scholars and staff is a shared commitment to a simple moral principle: free enterprise and American leadership in the world are pillars of the struggle to defend human dignity against poverty and tyranny, and to help lift up marginalised people. Learning about the extraordinary record of global capitalism and free trade in combating poverty had been a major reason for my interest in economics. Now I had an opportunity to devote my work to sharing that truth with the world.
Indeed, my deep admiration for AEI created a small problem. What if, as an inexperienced executive, I failed and damaged the organisation? While deciding whether to take the job, I had breakfast with Tully Friedman, a private-equity pioneer and longtime AEI board member who later became its chairman and one of my closest friends. I worried aloud about failing and jeopardising this 75-year-old think tank. “Don’t worry about that,” he said. “If you can’t raise money or motivate the scholars, we’ll fire you in a year and AEI will be fine.” That was strangely reassuring. With a mixture of excitement and panic, I gave up tenure and moved to Washington.
Impact, rather than outputs or inputs
Nonprofit leaders commonly make two mistakes when answering the question, “How do we know whether we are having an impact?” The first is what I call the sui generis error: the idea that our work is unique and our organisation so different from anything else that it cannot be compared or benchmarked against another organisation. It is remarkable how many intelligent people believe this about their own organisations or those they love. The claim is false, but I hear it constantly. Philanthropists hear it even more often.
The second is the “streetlight error”, named after the story of a man who loses his keys in the street and spends hours looking for them beneath a streetlight because the light is better there. Nonprofits struggling to measure effectiveness often fall back on whatever is easiest to see: usually inputs, such as the contributions they receive, or outputs, such as how busy they have been. This is plainly inadequate, because what interests us is impact.
The sui generis error leads to measuring nothing, while the streetlight error leads to measuring the wrong things. My colleagues and I set out to avoid both and find a better way to understand and describe our actual progress.
AEI’s output is straightforward: books, research papers, opinion articles, media appearances and public events. These products effectively constitute our supply curve. But nobody argues that writing an opinion article, publishing a peer-reviewed paper or getting a scholar on television automatically changes how leaders think and act. Our output measures are therefore not particularly interesting in isolation. To move from output to impact, we had to superimpose a demand curve on our supply curve. We needed ways, however imperfect, to measure how much leaders wanted and sought our work.
In the ideas industry, this kind of demand is almost never directly observable. However well an organisation educates leaders or raises the profile of an issue, no measure such as opinion polls, election results or legislative votes can isolate its role amid the noise of thousands of other variables.
We realised that think tanks must develop proxy measures of impact, looking for the point where their supply meets competitive demand in the marketplace of ideas. We needed to identify and track products that leaders consume at some cost to themselves and in direct trade-offs against alternatives. Alone, each metric is a single point of limited usefulness. Together, they form a pointillist picture that helps us assess our work’s impact.
Here are two examples from the set of proxy measures we developed.
The most prestigious national newspapers each receive around 1,000 unsolicited opinion submissions every week. Space on their opinion pages is an intensely competitive market, and editors reject everything except what they believe readers most need and want. That selectivity can reveal competitive demand for our product. The total number of opinion articles written each year is merely an output measure; the number our scholars place in a defined set of highly competitive outlets is a viable proxy for impact. Although far from a comprehensive measure, it has one particularly useful feature: the data are public by definition, allowing comparison with peers and competitors. AEI has maintained a lead in every year measured.
We do the same with congressional testimony. Most policy experts want to testify on Capitol Hill, but they cannot simply telephone the Senate switchboard and put themselves on the schedule. The Senate calls you. Although nobody believes that public testimony is the only way think tanks shape debate, it offers a useful angle for measuring impact. Again, everyone’s data are publicly available. When we first examined these figures, for the 110th Congress (2007–2009), AEI ranked fourth among think tanks for congressional testimony. In the 111th Congress, we became number one and have maintained that position since.
It bears repeating that none of these metrics is a direct measure of impact, let alone a perfect one. Every proxy can be manipulated or misinterpreted, and each reveals only one part of the truth. Nor can every useful proxy support competitive comparisons. Our scholars’ personal relationships and private briefings with policymakers and journalists, for example, are important indicators because leaders’ time and attention are scarce. Yet it is impossible to compare those private data across organisations.
The aim is not to find a perfect proxy. Nonprofits working in the world of ideas can instead build a dashboard containing a wide range of such variables, then use it to measure public leaders’ revealed preferences and the uptake of the organisation’s work.
Clarifying the mission
While we began building this impact dashboard, my management colleagues and I made several complementary changes. For example, we changed how the organisation described itself to reflect our definition of success. Think-tank mission statements are often dull lists of products—“We conduct high-quality policy research”—rather than genuine expressions of purpose. In consultation with our scholars and supporters, we wrote a new statement: “The American Enterprise Institute is a public policy think tank dedicated to defending human dignity, expanding human potential and building a freer and safer world. The work of our scholars and staff advances ideas rooted in our belief in democracy, free enterprise, American strength and global leadership, solidarity with those on the margins of society, and a pluralistic entrepreneurial culture.” We sought to make clear the moral purpose of our ideas: serving others, especially those on society’s margins.
Measuring impact
Opinion articles
Think-tank scholars write many opinion articles, but that measures production. To understand impact better, AEI began tracking how many appeared in three leading newspapers—The New York Times, The Wall Street Journal and The Washington Post—and comparing placements with competing think tanks over time. The percentages reflect annual averages for 2015–2017.
Congressional testimony
Testifying before congressional committees is another way AEI scholars can exert meaningful influence. When Arthur Brooks arrived and encouraged AEI to collect these data, the organisation ranked fourth. It has since moved into first place and increased its share.
We then launched a series of high-profile entrepreneurial ventures expressly designed to create impact and fulfil our new statement of purpose. Some were unusual, including happiness research, experimental multimedia ventures and a major collaboration with the Dalai Lama. The latest focuses on despair and human dignity in the United States. It aims to identify and propose long-term solutions to the underlying social and economic despair that shaped both sides of the most recent presidential election. It has four pillars: vocational and technical education to help people find good jobs; using free enterprise to strengthen families’ economic security; reforming the criminal justice system; and finding new strategies to curb the opioid-abuse epidemic.
Built and marketed as an internal start-up, this venture does not displace our other work. To present it to investors, we created a prospectus and pitch deck, just as any start-up would. We approach potential donors as we would venture capitalists—which many of them are. We discuss return on investment and set out the basket of impact measures we will use to demonstrate success.
Our new focus on impact also helped us refine audience development and segmentation. We believe potential consumers of our ideas can be placed in four groups according to their receptiveness: true believers, who already agree; the persuadable, who are open to hearing from us; the hostile, who think our perspective is foolish or evil; and the apathetic, who do not care. We cross these four attitudes with the five key groups in our audience: policymakers, business leaders, the media, community leaders and academics. This creates a four-by-five matrix, allowing us to balance our strategies and offerings to maximise impact and fulfil our mission more effectively.
Our strategy paid off
Unsurprisingly, these changes were not always easy to implement. I made many mistakes. Some colleagues complained that I was asking them to spend too much time and energy collecting data, and sometimes they were right. More than once, I measured entirely the wrong thing. For example, I became concerned when attendance at a series of live events fell. After a few months, someone pointed out that we had begun streaming those events online, where they attracted substantial traffic. This led to an even better measure: subscribers to events and original video programming on our YouTube channel, a measure on which AEI now leads the think-tank sector.
Despite some setbacks, our strategy has paid off. Since 2008, AEI’s operating revenue has grown by an average of around 10% annually; the institute bought and renovated a new headquarters in central Washington, D.C.; and we grew from 140 full-time scholars and staff to 220. More importantly, the impact of our research among policymakers and other leaders has increased dramatically. It is an exciting time at AEI.
Naturally, this growth required changes to our management structure. Managers now have greater authority and autonomy over internal administration, while my role has become increasingly outward-facing. These days, I travel around half the time, meeting prospective investors and giving 175 speeches a year about our work.
Looking back on my early days as AEI’s leader, I am grateful for the frightening demands that drove our changes. Because donors required proof that our intrinsically amorphous products were having an impact, we learned to measure and demonstrate it. Because we had to communicate our work in a crowded marketplace, we invested in communications systems that will pay off for years. I have clear evidence that crisis is the mother of invention.
Other nonprofits are asking about our systems, and several major foundations have asked our staff to help other grantees adopt some of our practices. That is satisfying. Above all, I am grateful to help our social enterprises increase the impact of their efforts in a world that needs them more than ever.
Arthur C. Brooks was president of the American Enterprise Institute from 2009 to 2019.