An Evaluation Culture
I took the idea of an evaluation culture from a wonderful paper written by Donald Campbell in 1969 entitled ‘Methods for an Experimenting Society.’ Following in the footsteps of that paper, this one is considerably more naive and utopian. And, I have changed the name of this idealized society to reflect terminology that is perhaps more amenable to the climate of the 1990s. For the term experimenting, I have substituted the softer and broader term evaluating. And for the term society, I have substituted the more internationally-flavored term culture. With these shifts in emphasis duly noted, I want you to know that I see the evaluation culture as one that a member of the experimenting society would feel comfortable visiting, and perhaps even thinking of taking as a permanent residence.
What would an evaluation culture look like? What should its values be? You should know at the outset that I fully hope that some version of this fantasy will become an integral part of twenty-first century thought. There is no particular order of importance to the way these ideas are presented – I’ll leave that ordering to subsequent efforts.
First, our evaluation culture will embrace an action-oriented perspective that actively seeks solutions to problems, trying out tentative ones, weighing the results and consequences of actions, all within an endless cycle of supposition-action-evidence-revision that characterizes good science and good management. In this activist evaluation culture, we will encourage innovative approaches at all levels. But well-intentioned activism by itself is not enough, and may at times be risky, dangerous, and lead to detrimental consequences. In an evaluation culture, we won’t act for action’s sake – we’ll always attempt to assess the effects of our actions.
This evaluation culture will be an accessible, teaching-oriented one that emphasizes the unity of formal evaluation and everyday thought. Most of our evaluations will be simple, informal, efficient, practical, low-cost and easily carried out and understood by nontechnicians. Evaluations won’t just be delegated to one person or department – we will encourage everyone in our organizations to become involved in evaluating what they and their organizations do. Where technical expertise is needed we will encourage the experts to also educate us about the technical side of what they do, demanding that they try to find ways to explain their techniques and methods adequately for nontechnicians. We will devote considerable resources to teaching others about evaluation principles.
Our evaluation culture will be diverse, inclusive, participatory, responsive and fundamentally non-hierarchical. World problems cannot be solved by simple “silver bullet” solutions. There is growing recognition in many arenas that our most fundamental problems are systemic, interconnected, and inextricably linked to social and economic issues and factors. Solutions will involve husbanding the resources, talents and insights of a wide range of people. The formulation of problems and potential solutions needs to involve a broad range of constituencies. More than just “research” skills will be needed. Especially important will be skills in negotiation and consensus-building processes. Evaluators are familiar with arguments for greater diversity and inclusiveness – we’ve been talking about stakeholder, participative, multiple-constituency research for nearly two decades. No one that I know is seriously debating anymore whether we should move to more inclusive participatory approaches. The real question seems to be how such work might best be accomplished, and despite all the rhetoric about the importance of participatory methods, we have a long way to go in learning how to do them effectively.
Our evaluation culture will be a humble, self-critical one. We will openly acknowledge our limitations and recognize that what we learn from a single evaluation study, however well designed, will almost always be equivocal and tentative. In this regard, I believe we too often undervalue cowardice in research. I find it wholly appropriate that evaluators resist being drawn into making decisions for others, although certainly the results of our work should help inform the decision makers. A cowardly approach saves the evaluator from being drawn into the political context, helping assure the impartiality needed for objective assessment, and it protects the evaluator from taking responsibility for making decisions that should be left to those who have been duly-authorized – and who have to live with the consequences. Most program decisions, especially decisions about whether to continue a program or close it down, must include more input than an evaluation alone can ever provide. While evaluators can help to elucidate what has happened in the past or might happen under certain circumstances, it is the responsibility of the organization and society as a whole to determine what ought to happen. The debate about the appropriate role of an evaluator in the decision-making process is an extremely intense one right now in evaluation circles, and my position advocating a cowardly reluctance of the evaluator to undertake a decision-making role may very well be in the minority. We will need to debate this issue vigorously, especially for politically-complex, international-evaluation contexts.
Our evaluation culture will need to be an interdisciplinary one, doing more than just grafting one discipline onto another through constructing multi-discipline research teams. We’ll need such teams, of course, but I mean to imply something deeper, more personally internalized – we need to move toward being nondisciplinary, consciously putting aside the blinders of our respective specialties in an attempt to foster a more whole view of the phenomena we study. As we consider the programs we are evaluating, we each should be able to speculate about a broad range of implementation factors or potential consequences. We should be able to anticipate some of the organizational and systems-related features of these programs, the economic factors that might enhance or reduce implementation, their social and psychological dimensions, and especially whether the ultimate utilizers can understand or know how to utilize and be willing to utilize the results of our evaluation work. We should also be able to anticipate a broad spectrum of potential consequences – system-related, production-related, economic, nutritional, social, environmental.
This evaluation culture will also be an honest, truth-seeking one that stresses accountability and scientific credibility. In many quarters in contemporary society, it appears that many people have given up on the ideas of truth and validity. Our evaluation culture needs to hold to the goal of getting at the truth while at the same time honestly acknowledging the revisability of all scientific knowledge. We need to be critical of those who have given up on the goal of “getting it right” about reality, especially those among the humanities and social sciences who argue that truth is entirely relative to the knower, objectivity an impossibility, and reality nothing more than a construction or illusion that cannot be examined publicly. For them, the goal of seeking the truth is inappropriate and unacceptable, and science a tool of oppression rather than a road to greater enlightenment. Philosophers have, of course, debated such issues for thousands of years and will undoubtedly do so for thousands more. We in the evaluation culture need to check in on their thinking from time to time, but until they settle these debates, we need to hold steadfastly to the goal of getting at the truth – the goal of getting it right about reality.
Our evaluation culture will be prospective and forward-looking, anticipating where evaluation feedback will be needed rather than just reacting to situations as they arise. We will construct simple, low-cost evaluation and monitoring information systems when we first initiate a new program or technology – we cannot wait until a program is complete or a technology is in the field before we turn our attention to its evaluation.
Finally, the evaluation culture I envision is one that will emphasize fair, open, ethical and democratic processes. We should move away from private ownership of and exclusive access to data. The data from all of our evaluations needs to be accessible to all interested groups allowing more extensive independent secondary analyses and opportunities for replication or refutation of original results. We should encourage open commentary and debate regarding the results of specific evaluations. Especially when there are multiple parties who have a stake in such results, it is important for our reporting procedures to include formal opportunities for competitive review and response. Our evaluation culture must continually strive for greater understanding of the ethical dilemmas posed by our research. Our desire for valid, scientific inference will at times put us in conflict with ethical principles. The situation is likely to be especially complex in international-evaluation contexts where we will often be dealing with multiple cultures and countries that are at different stages of economic development and have different value systems and morals. We need to be ready to deal with potential ethical and political issues posed by our methodologies in an open, direct, and democratic manner.
Do you agree with the values I’m describing here? What other characteristics might this evaluation culture have? You tell me. There are many more values and characteristics that ought to be considered. For now, the ones mentioned above, and others in the literature, provide us with a starting point at which we can all join the discussion. I hope you will add to the list, and I encourage each of you to criticize these tentative statements I’ve offered about the extraordinary potential of the evaluation culture that we are all in the process of creating today.