Archives
Book Review
Realworld Evaluation: Working under budget, time, data, and political constraints
Michael Bamberger, Jim Rugh and Linda Mabry
ISBN: 1-412909-46-5 2006 468 pp+xxxii pages Thousand Oaks, CA: Sage Publications
David McDonald
National Centre for Epidemiology and Population Health, The Australian National University, TAS
The last two years have seen a flowering of fine books on evaluation. These have included the 7th edition of the world's top seller in the field (Rossi, Lipsey and Freeman's 2004 Evaluation: A Systematic Approach); the 3rd edition of Australian scholar John Owen's 2006 Program Evaluation: Forms and Approaches; Davidson's 2005 Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (which may well become a core reference for students); and Mathieson's wonderful new 2005 Encyclopedia of Evaluation. These, and other new publications I could mention, support the contention that evaluation has become not only a profession, but also a discipline.
Bamberger, Rugh and Mabry's 2006 RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints is the latest offering from Sage Publication's evaluation booklist. Bamberger is a private consultant and a former Senior Sociologist at the World Bank; Rugh is with CARE International; and Mabry a Washington State University academic. They have much experience in conducting evaluations and teaching evaluation concepts and practice in developing countries and elsewhere, and this solid experience permeates the book.
The title coveys the way in which this book differs from so many others apparently covering similar ground: the numerous how-to evaluation resource books. The authors start with the real world of program evaluators, the world in which there is often insufficient money, time, data and political support to undertake sound evaluations. What, then, is the alternative to the types of evaluation research we would undertake were there sufficient money, time, data and political support? The authors point out that, for far too many evaluators, the alternative is a 'quick and dirty' study; meaning one which lacks methodological rigour and ignores threats to the validity of the study's findings.
'RealWorld evaluation' is defined as 'An approach developed for evaluations operating under budget, time, data and political constraints intended to maximise the rigor of methods and the validity of findings' (p. 439). RealWorld evaluation, as portrayed in this volume, has seven steps:
(1) planning and scoping the evaluation
(2-5) directly addressing the budget, time and data constraints, along with the political influences
(6) strengthening the evaluation design and the validity of the conclusions; and
(7) helping clients to use the evaluation's processes and products.
These are the features, the authors argue, which make this book different from the many others covering program and policy evaluation.
An example may assist. The chapter covering 'Critical information is missing or difficult to collect: addressing data constraints', deals seriatim with the following strategies: reconstructing baseline data, identifying and constructing comparison groups, problems of non-equivalent comparison groups, and collecting data on sensitive topics and from difficult-to-reach groups. In each area, specific research strategies to overcome the constraints are described and discussed.
The book is large: 468 pages in a fairly small font. The lengthy and detailed table of contents - it runs to 14 pages - is useful for readers looking for advice on a particular issue.
The book contains three parts. The first is an overview of RealWorld evaluation. The second comprises one chapter on each of the seven steps listed above: the how-to cook-book component. The third is a scholarly review of the social science research methods used in evaluation, and their application to the constraints with which the book deals. The final part, 'Pulling It All Together', deals briefly with capacity building in evaluation and then provides a detailed and beautifully-crafted 30 page summary of the whole book. It concludes with appendices providing checklists, pointers to additional resources and a glossary.
Many professional evaluators, students and those who commission evaluations will find this book just what they are looking for: sound, practical advice grounded in the authors' own experiences. Some evaluation academics and practitioners, though, will be irritated by what they see as a major omission, a fatal flaw in the volume. The omission links to The Great Debate in evaluation: is evaluation simply the application of standard social science research methods to answering evaluation questions, or is evaluation this plus another set of research processes for determining the merit, worth or value of something, as Scriven (2003) argues. RealWorld Evaluation is firmly in the former camp, omitting guidance on how to make evaluative judgments.
RealWorld Evaluation is a reference and text book that I highly recommend. Its structure means that the experienced evaluator who needs ideas on how to overcome a particular barrier can readily find what she or he is looking for. The advice provided is practical and methodologically sound. It will be used as a text book for students of evaluation as its comprehensiveness (save for the omission of research methods addressing the 'value' part of 'evaluation') and clarity will meet their needs well. The large number of case studies illustrating the points being made, along with many presentations in break-out boxes, checklists, summaries and lists of further reading, all facilitate its use as a training/capacity-building tool.
References
Davidson EJ (2005) Evaluation Methodology Basics: The Nuts And Bolts Of Sound Evaluation. Sage Publications: Thousand Oaks, California.
Mathison S (2005) (ed) Encyclopedia of Evaluation. Sage: Thousand Oaks, California.
Owen JM (2006) Program Evaluation: Forms and Approaches 3rd edition, Allen and Unwin: Crows Nest, NSW.
Rossi PH, Lipsey MW and Freeman HE (2004) Evaluation: A Systematic Approach 7th edition, Sage: Thousand Oaks, California.
Scriven M (2003) Evaluation in the new millenium: The transdisciplinary vision. In SI Donaldson and M Scriven (eds) Evaluating Social Programs And Problems: Visions For The New Millennium. Lawrence Erlbaum: Mahwah, NJ, pp.19-41.
eContent Home