Book Reviews
The Practice of Health Program Evaluation
David Grembowski
ISBN: 0-7619184-7-7 2001 323 pages Sage Publications, London
Anne Magarey
Primary Health Care Research and Information Service, Flinders University, Adelaide, Australia
Libby Kalucy
Primary Health Care Research and Information Service, Flinders University, Adelaide, Australia
David Grembowski's book is a welcome contrast to evaluation text books that contain fascinating theoretical concepts but which are very difficult to apply in practice. Grembowski intended the book for two purposes. The first is to increase knowledge and understanding about program evaluation for graduate students, program administrators, decision makers, health services researchers and evaluators. The second is to teach graduate students how to become evaluators by applying research methods in the practice of evaluation. Readers will find that the well organised presentation of essential evaluation concepts and approaches succeeds in making the practice of health program evaluation understandable and accessible. The book would be a useful text for graduate students with some understanding of health systems and programs, of their political nature, and of statistical methods. The book is practical and easy to understand but does not fall into the trap of suggesting that one-size-fits-all program evaluations are possible. It introduces the advantages and disadvantages of different approaches with reference to a wide range of resources and publications. Each chapter includes a brief summary, list of key terms and exercises.
The book is based on the view that politics and values are inseparable in health program evaluation. Therefore evaluators must be able to navigate the political terrain at all stages of an evaluation. The author uses the metaphor of a three-act play to illustrate health program evaluation as a series of interconnected steps designed to produce information for decision-makers and other groups. A variety of actors and interest groups with different roles enter and exit the process at different points. The evaluator, as one of the actors in this play, must not only understand their role and the steps and interconnections between the acts of the play, but also know how to apply quantitative and qualitative research methods.
Following a Prologue examining the worth of health program evaluation, and an overall introduction to the three-act play, the book has three major sections: 'Act I: Asking the Question', 'Act II: Answering the Question', and 'Act III: Using the Answers in Decision-making'. Act I occurs in the political realm, where evaluators work with decision-makers and others to define the questions that the evaluation will answer about a program. In Act II, Chapters 4 to 6 present designs for experimental and quasi-experimental evaluations, cost-effectiveness analysis and program implementation or process evaluation. Chapters 7 to 9 provide more detail on quantitative and qualitative methods for choosing populations and sampling, measurement and data collection issues, and data analyses for different designs to evaluate impact and implementation. Act III returns to the political arena for the important process of bringing the findings to the decision-makers and other groups who had roles in Act 1. The device of the three-act play is useful to emphasise the importance of the planning and knowledge-transfer stages of evaluation that sometimes receive less prominence than the middle 'act' of design, data collection and analysis.
The examples and figures used throughout this book relieve the potential dryness of describing evaluation concepts and approaches. The examples from published literature are relevant to Australian health services and include the delivery of preventive services to older adults, and establishing health clinics in high schools. The continued use of the same examples throughout the book is easier on the reader than trying to understand multiple, separate evaluation scenarios. In addition to some very useful figures illustrating program theory and hierarchies of objectives, Grembowski effectively uses the device (characteristic of his mentor Steve Shortell) of cross-tabulating two dimensions of a concept. Through this approach, Grembowski demonstrates that programs fail if there are faulty assumptions about the underlying theory of cause and effect, if the program was not implemented as intended, or both. While the author acknowledges the cross-tabulation device is simplistic, it can clarify concepts and possibilities.
The sections on delineating program theory are welcome and most important. Through these, the author emphasises that a key purpose of evaluation is to provide the information required by decision makers. This means that evaluation has a role in explaining as well as in describing the impact of a program. This involves providing information on how to fix programs that don't work, on how to identify and replicate successful elements of a program in other settings, and on how to avoid the unintended consequences of a program in future. The program implementation section includes a useful checklist of possible reasons for a program not achieving its intended outcomes.
Quantitative data collection methods receive much more space than qualitative methods, although the importance of the latter is emphasised and the author gives considerable attention to methods for ensuring their reliability and validity. It is pleasing to see cost effectiveness analysis included here because of its significance in comprehensive health program evaluation. Ethically challenging scenarios are raised in all three acts of this play, as part of the political realm in which evaluation takes place. The need for culturally appropriate evaluation methods is also addressed, if rather briefly.
This book makes it clear that the purpose of an individual health program evaluation is to meet information requirements for decision-makers, and reduce uncertainty about the best ways of solving health problems. Grembowski concludes that synthesising evidence from individual evaluations through meta analyses is vital in expediting the accumulation of knowledge about the performance of health services.
eContent Home