Evaluation Systems Award

Dr Julian King, Value for Investment approach

Value for Investment is an innovative, exemplary evaluation system that integrates theory and practice from evaluation and economic disciplines to inform judgements and decision making. Initially developed though Julian King’s doctoral research it is a flexible and collaborative approach that can be applied to all domains and program types. The letters of support demonstrate it has elevated the thinking and approaches of a wide group of practitioners across varied domains of evaluation practice including organisations and projects such as: Oxford Policy Management (an international development consultancy), The Indigenous Land and Sea Corporation, the Australia Indonesia Partnership for Economic Development, the International Atomic Energy Agency, the Australian Council for Education Research and Hikitia Consultants (a Māori evaluator). It has been used internationally including in New Zealand, Australian, Mozambique, Pakistan and the UK. It is highly practical and adaptable and promotes the use of mixed methods in value for money assessment for sense and decision making. It shifts the focus of value for money from costs and inputs to holistic, whole of program and system, considerations. Through the use of rubrics and multiple criteria to assess value for money it provides a robust mechanism to consider value and promotes the use of transparent evaluative reasoning. The Value for Investment integrated evaluation system is a significant contribution to evaluation theory and practice and is shared on an open access basis through the Oxford Policy Management’s value for money guide, Julian King’s published PhD dissertation, journal articles in American Journal of Evaluation, the Evaluation Journal of Australasia, New Zealand’s evaluation Journal Evaluation Matters, Evaluation and Program Planning, as well as open-access evaluation reports and blogs. It is also currently taught in the University of Melbourne’s Master of Evaluation program as a component of the course ‘Evaluation and Value for Money’ and in workshops through evaluation societies internationally. The Value for Investment system is an exemplary integrated evaluation system that has been implemented successfully in a variety of contexts and is a worthy recipient of the AES Evaluation Systems Award. 

Public Sector Evaluation Award

Victorian Department of Health (Centre for Evaluation and Research Evidence), Review of the North Richmond Medically Supervised Injecting Room 

This study was incorporated in a review of the first medically supervised injecting room set up in Melbourne. It took place within a challenging and politically charged environment and over several years. The study addressed a number of these challenges through allocating significant time and resources to consulting with the wide range of stakeholders involved in the study. In addition, there were significant ethical issues that the study dealt with through applying the AES Code of Ethics and seeking approval through an NHMRC ethics committee. A wide range of data collection methods were used to gather evidence to address both formative and summative questions. The work of the evaluation team was highly praised by members of the review team stating that the team developed and tested a multi-faceted framework for the review, drew on local and international evidence, conducted a complex project to support the review panel with clear evidence based findings on which to base recommendations. In addition, the Director of the of the Injecting Room Project praised the team’s work, stating the team helped build, “… a credible foundation for government to make decisions on the next phase of this life-saving trial. Importantly and immediately, the Victorian government accepted recommendations to continue the trial and expand it to include a second supervised injecting centre. The work of the Centre has helped inform other changes (that) have either already been made to improve the trial, such as the development of the approach to local community engagement in North Richmond, or are being considered (e.g. in future legislation)”.

EJA Publication Award

Ruth McCausland for her paper "‘I’m sorry but I can’t take a photo of someone’s capacity being built’: Reflections on evaluation of Indigenous policy and programmes”

The paper presents clear, robust research on evaluation with evidence presented and a number of 'tools' (e.g. rubric with a worked example) which could be used by practitioners in their work. Similarly, limitations and areas for further work are articulated, with a contribution to the evaluation evidence base. The article makes an important contribution and is relevant to key readership of the journal. The paper is very contemporary and adds value to the evidence base in an area that is sadly lacking.

Emerging New Talent Award

Amanda Mottershead, Toolkit to Engage Young and Emerging Evaluators (YEEs) in Voluntary Organizations for Professional Evaluation (VOPEs) Governance and Leadership

Amanda Mottershead has been selected for the Award based on a substantial piece of work she undertook in a difficult environment, with good references to how issues such as professionalism and ethics were managed. The process of working internationally across a large number of voluntary associations is extremely difficult and Amanda managed this with the skills of a much more experienced evaluator. The final product, a Toolkit to Engage Young and Emerging Evaluators in VOPEs, is comprehensive, engaging, practical and well designed. It is a resource that will be used across the world to encourage young evaluators to become more engaged in the profession. This is a goal we can all support.

Award for Enhancing the Social Good

ARTD and the Kinchela Boys Home Aboriginal Corporation (KBHAC)- KBHAC Practice Framework “The More I Talk, The Stronger I Get”

ARTD and KBHAC have developed a practice framework that brings together best practice trauma-informed approaches, is strongly evidence based and harnesses the voice of survivors to shape the work. Importantly, it articulates what survivors, descendants and program participants can expect from KBHAC. The framework is based on preliminary work that ARTD did with KBHAC in developing an outcomes framework to map KBHAC’s programs and services and identify intended outcomes and success measures of these programs and services for monitoring and evaluation purposes.

The work excelled against all of the criteria of excellence related to the award. It employed developmental and empowerment evaluation approaches to a high standard. ARTD’s consultation and project design was highly collaborative, informed by Aboriginal-led co-design practice and the recognition of KBHAC’s knowledge and expertise and the importance of ground-up, creative and iterative planning and design of project priorities. The KBHAC Practice Framework has been received with excitement and positivity by the KBHAC community. Staff have reported finding the tool useful in guiding their work and working with clients, showing it as a way of communicating KBHAC’s way of working. KBHAC staff have also found the workshops and processes for developing the frameworks to be valuable and reported feeling more confident and inspired in their work.

There is a lot of literature on the effects of trauma and intergenerational trauma, stories, reports and education material on the Stolen Generation. There is a gap, however, on documenting approaches that make a difference for the healing journey of those directly and indirectly effected by this multigenerational trauma. This is the gap that the practice framework fills.

The award is provided given the excellent use of evaluative thinking to develop a framework that is useful to the practice of a social justice focused organisation with great potential for impact across the sector and at community level. This is the first Practice Framework designed with and for Stolen Generations survivors, descendants and families. It aims to help restore and reconstruct the identity, dignity and integrity of Aboriginal men who were forcibly removed from their families and placed in the Kinchela Boys Home (KBH). The framework recognises that Aboriginal people and survivors are best placed to develop the healing frameworks to address the effects of the multigenerational trauma that adversely impacts on their lives. KBHAC’s work guided by the framework is at the forefront of healing work being done in Australia. Their work is critical to reckoning with Australia’s recent history of genocide, pain and injustice suffered by Aboriginal and Torres Strait Islander families and communities.

Evaluation Study or Project Award

Queensland Government’s Office of the Commonwealth Games, Department of Innovation, Tourism Industry Development, and the Commonwealth Games; and the evaluation team of Mark Douglas, Robert Grimshaw, Nicolette Pavlovski, Sean Conway, Kelly Reynolds, Joanne Ryan and Meghan Purcell – for the Evaluation and Monitoring Framework for the Embracing 2018 Legacy Program

The Queensland Government’s Embracing 2018 Legacy Program aims to ensure the Queensland community realised lasting benefits from hosting the 2018 Commonwealth Games. The reach of this event extends beyond the Games host and event cities to provide measurable outcomes for Queensland and Australia. An Evaluation and Monitoring framework for this program has evolved since 2013 and is designed to guide implementation of the Embracing 2018 Legacy Program and assess its outcomes over a 10-year period.

The assessors were impressed by the evaluation’s sound use of evaluation theory and approaches, including the ability to incorporate emergent findings into a results’ framework and ensure ongoing connections between projects. There was evidence of strong and sustainable connections being developed; for example, with the evaluation team building on work started in Glasgow with the 2014 Commonwealth Games, and their commitment to developing a framework that can be used by future hosts of the Commonwealth Games. The assessors noted the evaluation team’s commitment to publicly share the methods behind their framework. 

Evaluation Policy and Systems Award

The Victorian Department of Health and Human Services (DHHS) Centre for Evaluation and Research

The DHHS Centre for Evaluation and Research led the development and implementation of evaluation policy and systems in this large Victorian state government department. It played a pivotal role in transforming the evaluation culture of the department through policy guidance, advice and support, design and delivery, training and knowledge translation.

The Centre for Evaluation and Research demonstrated excellence in applying theory-based principles and approaches to developing and implementing evaluation policy and systems. The judges were impressed with the high-quality evaluation resources and pragmatic support that departmental staff have access to, including tools and tailored advice. The nomination included good evidence of stakeholder satisfaction with the team’s work. The judges were also impressed with the significant contribution of the team to transferring knowledge across the department and broader community and public sectors. Overall, the judges considered this Award winner as an exemplary case of evaluation policy and systems contributing to service delivery and improving community wellbeing, including among those most vulnerable to the risks of poorer outcomes. 

Community Development Award 

Palmerston/ Tiwi Island Communities for Children (C4C) Participatory Evaluation conducted by the Communities for Children (C4C) Local Committee, Pandanus Evaluation & Planning Services (Nea Harrison) and the Australian Red Cross (Rachel Dunne) 

The Award judges noted that the work was well-thought through, extremely thorough and comprehensive. A local committee, the elders and young women were all actively involved in the project in a strong collaboration with the facilitator of the evaluation. The nomination indicated an understanding of how an evaluation design contributes to community development goals and how the use of community development processes enables the accomplishment of the evaluation.

Areas that stand out as exemplary and examples of good evaluative practice in this evaluation include: 
- culturally appropriate evaluation design and methodology
- strong community engagement at all stages of the evaluation from inception to conclusion, including closure and reporting back to the community 
- a developmental approach working sensitively and in sympathy with the local community 
- outcomes resulting in sustainable benefits for local participants (e.g. capacity building) 
- use of advanced techniques –logic framework, quality rubrics, the community report.

Indigenous Evaluation Award

Aboriginal Evidence Building Partnership Pilot
ARTD Consultants Simon Jordan, Sue Leahy, Ruby Leahy Gatfield, Kieran Sobels, Christine Eastman, Stephanie Quail, Imogen Williams, Holly Kovac
Aboriginal Evidence Building Team, Their Futures Matter: Nattlie Smith, Sharon Macleod
Coffs Harbour Aboriginal Community Centre Inc (Abcare): Garry Mathews, Greg Bennett, Trent Matthews
Tirkandi Inaburra Cultural and Development Centre: Matt Watts, Damien Thorne, Beverley Tucker, Department of Communities, and Justice Troy Mott, Department of Education

This evaluation was conducted by a team committed to enabling Indigenous self-determination and empowerment from the outset. This was due in large part to the Indigenous leadership within the team combined with the skill and experience of the other team members in this context.

The evaluation demonstrated how collaborative and participatory approaches, together with a strong partnership model, enabled and paid respect to the roles and cultural imperatives of two organisational partners involved, as well as to the intended objectives of the NSW government agency involved. Overall, the judges concluded that this Award winner has made a strategic contribution to fostering a culture of evidence building and evaluation capacity for the ongoing monitoring and review of the Aboriginal services sector in NSW. The judges are confident that the work will lead to increased benefits for the two organisations’ service recipients, who are Aboriginal families whose children are at risk of removal, and Aboriginal boys aged 12–15 years who are seeking to develop their resilience and life skills. 

Evaluation Publication Award (Caulley Tulloch Award)

Samantha Abbato – for 'The case for evaluating process and worth: evaluation of a programme for carers and people with dementia' 

This is a book chapter published in the most recent volume of the prestigious series Advances in Program Evaluation, edited by Trisha Greenhalgh & Saville Kushner. Its thesis is the utility of the case study approach as a major component of a mixed-methods evaluation. What makes the chapter worthy of the award is the author's careful analysis and demonstration of the role of case studies in mixed methods evaluations, including the way she contrasts them with the quantitative methods more frequently used in the sector covered by the project (community health/dementia). This chapter will be particularly valuable for people new to evaluation, or to those coming from a quantitative background who wish to gain an understanding of the role of case study research in evaluation. The nomination explicitly and convincingly addresses the specified filters and the seven specified criteria of excellence.  

Outstanding Contribution to Evaluation Award

Jess Dart

Jess is a recognised leader in evaluation with over 25 years of experience in the collaborative design and evaluation of programs that seek to bring about a more equitable and just society. The judges were impressed by this nominee’s sustained application of authentic inclusivity, and her high level of ethical standards so clearly evidenced through her practice. The judges noted this nominee’s skill in being able to combine deep evaluation knowledge and theoretical understandings with straightforward communication. She has undertaken more than 30 external evaluations and overseen over 120 evaluations in countries around the world. The judges noted the high level of professionalism and stakeholder satisfaction evidenced in this nominee’s work. As an ‘evaluation entrepreneur’, the judges were impressed by this nominee’s capacity to constantly scan the horizon for where evaluation is headed, and to create fresh approaches and techniques. The judges were impressed by her pioneering work on no less than five innovative approaches.

The judges were impressed by this nominee’s other contributions to evaluation knowledge. She is a contributor to evaluation textbooks, an author of refereed journal articles including publications in the American Evaluation Journal and New Directions in Evaluation, and a prolific trainer who has reached more than 1,000 participants, including many AES members.

Jess has been an AES member since 1997 and a Board member since 2014 in the role of treasurer; and was involved in the aes18 International Evaluation Conference as convenor. The judges acknowledged this nominee’s significant contribution to the AES.