Evaluation Systems Award

Department of Education and Training Victoria Evaluation Practice

The Department of Education and Training Victoria has strengthened their internal evaluation capacity through a comprehensive system. Launched in early 2021, the Evaluation Practice now employs over 20 staff and has achieved considerable successes. For example, the Evaluation Practice team have led nine large-scale evaluations, synthesised sense-making and cross-initiative insights at the system level, enabled greater access and use of internal DET data to ensure more robust findings, influenced state-administered education surveys to further increase the quality of data available, and ensured appropriate resourcing are provided to evaluation across the agency.

This provides a good model of an internal capacity building system which could provide practical examples for other agencies and organisations. 

Public Sector Evaluation Award

Te Ihuwaka | Education Evaluation Centre, Education Review Office – for Evaluation of learning in residential care 

The Education Review Office is the public service department of New Zealand charged with reviewing and publicly reporting on the quality of education and care of students in all New Zealand schools and early childhood services. This evaluation study looked at the quality of education for students in Oranga Tamariki Care and Protection and Youth Justice residences and how it can be improved. Children and young people who are placed in Oranga Tamariki residential care are among the most at risk of poor outcomes later in life. This was the first nationwide, system-level evaluation undertaken by the ERO. The study developed an excellent theory of change based on a published systems framework and used this effectively to develop the evaluation methodology which was robust and of a very high quality. The study developed a unique set of indicators of education performance for the residences, using a 4-point rubric to define the quality of practice, which will be useful in future evaluation studies and service delivery. The rubric was developed in consultation with the education providers to ensure their support for its use. Given the target population of school aged children, the evaluation team developed processes to give voice to these students including an appropriate reporting process. The evaluation team recognized the need to strongly engage stakeholders in the design and implementation of the study. Importantly, the team communicated regularly with the stakeholders on the study findings and developed a range of hard copies and online publications including the substantive report, a summary document, and four guides for students; parents and whanau (community members); leaders and teachers; and social workers. The evaluation team used a range of innovative approaches to enhance utilization of the study findings and products. As a result, there are a number of impacts that have already resulted from the study both in developing improved evaluation practice and decisions about the future of the residential care program.

EJA Publication Award

Northern Land Council for 'Checking up to keep on track: An Aboriginal-led approach to monitoring well-being'

Authors: Linda Kelly, Mary Whiteside, Hayley Barich and Komla Tsey

The paper addresses an important issue—the need to develop monitoring systems that privilege Aboriginal led visions of community development in northern Australia. The authors were careful to ensure that their findings were framed using an appropriate theoretical framework for wellbeing. The findings provide valuable lessons for others working with Aboriginal communities where externally imagined assumptions about what is good for communities end up being imposed on them, rather than being developed from the ground up. The paper is well written, topical and provides a valuable point of reference for evaluators to draw from.

Emerging New Talent Award

Allison Clarke, CPE University of Melbourne

During her evaluation career to date, Allison has successfully used skills and abilities from her previous experience in other sectors to conduct high quality evaluations, strengthen the evaluation field and support other emerging evaluators with whom she works. Allison is clearly an important part of her organisation, which itself is an integral part of the evaluation landscape in Australia. The diversity of Allison's experience presented in the nomination shows that she is able to successfully apply good evaluation practice to several sectors and contexts. The very positive feedback of stakeholders who have worked with Allison is a testament to her professional standards and high quality of evaluation work. This diversity is impressive for a career of only a few years. In addition, the use of evaluation outputs which Allison has helped to produce demonstrates the high quality and relevance of the work. Delivering such high quality and varied results in addition to other organisational and evaluation capacity building activities is an excellent achievement.

The organisational and capacity strengthening activities that Allison has been involved with, particularly in response to the COVID-19 pandemic, are important and show Allison’s leadership skills and ability to use these skills to strengthen the sector. In particular, Allison’s ability to build and maintain professional relationships with both internal and external stakeholders is impressive. A particular strength is Allison’s work on Aboriginal cultural safety including contributions to her Centre’s uptake of the AES Cultural Safety Framework and evaluation activities which have produced positive results for stakeholders. Allison has demonstrated a significant contribution to strengthening the sector and great potential for doing more of this in the future.

Award for Enhancing the Social Good

Centre for Health Service Development, Australian Health Services Research Institute, University of Wollongong – for Evaluation of the Pathways to Community Living Initiative

The evaluation makes a strong contribution to the social good by building up knowledge and support for the deinstitutionalisation process for those with severe and persistent mental illness (SPMI). The evaluation informed and supported the development of new service models for appropriate care in the community. It is noted that community-based settings aid recovery of those with SPMI, and historically limited support to this context has reinforced inequalities experienced by those with long-term mental health challenges. The overall quality of the evaluation was excellent. The team adopted a useful formative approach with a strong commitment to add value and contribute to the on-going refinement of the program. The evaluation team's focus on the significant role of clinicians and the need to build support and reflective practice within rehabilitation psychiatry are impressive. Similarly, theoretical and methodological attention to the drivers of transformation in a complex health system were important to identify and maintain focus on areas critical to change. Attention to including consumer and carer perspectives were pivotal in building a valid, credible, and useful evaluation.  

Indigenous Evaluation Award

Curijo Research Monitoring and Evaluation Team 

This was an innovative submission for the AES Indigenous Evaluation Award. The submission was not for the conduct of a particular evaluation, nor for the commonly recognised types of evaluative thinking process documentation such as a monitoring and evaluation plan. This Award recognises the company’s holistic approach to developing its Research, Monitoring and Evaluation Framework and how it has shaped the organisational settings. We have awarded the nomination as an example of an excellent evaluation product that has been prepared and used by a privately-owned Indigenous Australian monitoring and evaluation company. The company is courageously embarking on influencing the qualities and competitiveness of the Indigenous evaluation market through a culturally specific research, monitoring and evaluation framework. It employs a model of economic independence to build upon the growing national commitment to improve the quality of Indigenous-related programs and services through better evaluation. Curijo’s commitment has translated into measured organisational staffing structures, staff training in philosophical and practical Indigenous methodological approaches, as well as provision of community support beyond the requirements of the commercial contract. The internal application of the model against the business is still developing but it is excellent to see an Indigenous research and evaluation company that has developed a holistic, cross cultural approach to centring Indigenous perspectives, priorities and knowledges. This company is making a significant contribution to promoting the accountability and learning functions of Indigenous evaluation, as a means to realising beneficial impacts and self-determination for the First Nations peoples of Australia. 

Evaluation Study or Project Award

Queensland Government’s Office of the Commonwealth Games, Department of Innovation, Tourism Industry Development, and the Commonwealth Games; and the evaluation team of Mark Douglas, Robert Grimshaw, Nicolette Pavlovski, Sean Conway, Kelly Reynolds, Joanne Ryan and Meghan Purcell – for the Evaluation and Monitoring Framework for the Embracing 2018 Legacy Program

The Queensland Government’s Embracing 2018 Legacy Program aims to ensure the Queensland community realised lasting benefits from hosting the 2018 Commonwealth Games. The reach of this event extends beyond the Games host and event cities to provide measurable outcomes for Queensland and Australia. An Evaluation and Monitoring framework for this program has evolved since 2013 and is designed to guide implementation of the Embracing 2018 Legacy Program and assess its outcomes over a 10-year period.

The assessors were impressed by the evaluation’s sound use of evaluation theory and approaches, including the ability to incorporate emergent findings into a results’ framework and ensure ongoing connections between projects. There was evidence of strong and sustainable connections being developed; for example, with the evaluation team building on work started in Glasgow with the 2014 Commonwealth Games, and their commitment to developing a framework that can be used by future hosts of the Commonwealth Games. The assessors noted the evaluation team’s commitment to publicly share the methods behind their framework. 

Evaluation Policy and Systems Award

The Victorian Department of Health and Human Services (DHHS) Centre for Evaluation and Research

The DHHS Centre for Evaluation and Research led the development and implementation of evaluation policy and systems in this large Victorian state government department. It played a pivotal role in transforming the evaluation culture of the department through policy guidance, advice and support, design and delivery, training and knowledge translation.

The Centre for Evaluation and Research demonstrated excellence in applying theory-based principles and approaches to developing and implementing evaluation policy and systems. The judges were impressed with the high-quality evaluation resources and pragmatic support that departmental staff have access to, including tools and tailored advice. The nomination included good evidence of stakeholder satisfaction with the team’s work. The judges were also impressed with the significant contribution of the team to transferring knowledge across the department and broader community and public sectors. Overall, the judges considered this Award winner as an exemplary case of evaluation policy and systems contributing to service delivery and improving community wellbeing, including among those most vulnerable to the risks of poorer outcomes. 

Community Development Award 

Palmerston/ Tiwi Island Communities for Children (C4C) Participatory Evaluation conducted by the Communities for Children (C4C) Local Committee, Pandanus Evaluation & Planning Services (Nea Harrison) and the Australian Red Cross (Rachel Dunne) 

The Award judges noted that the work was well-thought through, extremely thorough and comprehensive. A local committee, the elders and young women were all actively involved in the project in a strong collaboration with the facilitator of the evaluation. The nomination indicated an understanding of how an evaluation design contributes to community development goals and how the use of community development processes enables the accomplishment of the evaluation.

Areas that stand out as exemplary and examples of good evaluative practice in this evaluation include: 
- culturally appropriate evaluation design and methodology
- strong community engagement at all stages of the evaluation from inception to conclusion, including closure and reporting back to the community 
- a developmental approach working sensitively and in sympathy with the local community 
- outcomes resulting in sustainable benefits for local participants (e.g. capacity building) 
- use of advanced techniques –logic framework, quality rubrics, the community report.

Evaluation Publication Award (Caulley Tulloch Award)

Samantha Abbato – for 'The case for evaluating process and worth: evaluation of a programme for carers and people with dementia' 

This is a book chapter published in the most recent volume of the prestigious series Advances in Program Evaluation, edited by Trisha Greenhalgh & Saville Kushner. Its thesis is the utility of the case study approach as a major component of a mixed-methods evaluation. What makes the chapter worthy of the award is the author's careful analysis and demonstration of the role of case studies in mixed methods evaluations, including the way she contrasts them with the quantitative methods more frequently used in the sector covered by the project (community health/dementia). This chapter will be particularly valuable for people new to evaluation, or to those coming from a quantitative background who wish to gain an understanding of the role of case study research in evaluation. The nomination explicitly and convincingly addresses the specified filters and the seven specified criteria of excellence.  

Outstanding Contribution to Evaluation Award

Jess Dart

Jess is a recognised leader in evaluation with over 25 years of experience in the collaborative design and evaluation of programs that seek to bring about a more equitable and just society. The judges were impressed by this nominee’s sustained application of authentic inclusivity, and her high level of ethical standards so clearly evidenced through her practice. The judges noted this nominee’s skill in being able to combine deep evaluation knowledge and theoretical understandings with straightforward communication. She has undertaken more than 30 external evaluations and overseen over 120 evaluations in countries around the world. The judges noted the high level of professionalism and stakeholder satisfaction evidenced in this nominee’s work. As an ‘evaluation entrepreneur’, the judges were impressed by this nominee’s capacity to constantly scan the horizon for where evaluation is headed, and to create fresh approaches and techniques. The judges were impressed by her pioneering work on no less than five innovative approaches.

The judges were impressed by this nominee’s other contributions to evaluation knowledge. She is a contributor to evaluation textbooks, an author of refereed journal articles including publications in the American Evaluation Journal and New Directions in Evaluation, and a prolific trainer who has reached more than 1,000 participants, including many AES members.

Jess has been an AES member since 1997 and a Board member since 2014 in the role of treasurer; and was involved in the aes18 International Evaluation Conference as convenor. The judges acknowledged this nominee’s significant contribution to the AES.