Thinking about how you demonstrate the impact of your scrutiny work? Wondering how you might improve? Well, there are frameworks that can help. This post highlights six.
No doubt there are things you are already doing. Annual reports, updates to council meetings, features in council newspapers, press releases, videos, recommendation trackers are just a few of the ways that scrutineers show that they are making a difference.
This post is for anyone who wants to develop or review their approach. It suggests six solid frameworks that can be used and adapted. Hopefully you will find one that you works for you.
Why demonstrate the impact of scrutiny?
There are a number of reasons why you might want to demonstrate the impact of your scrutiny work. It might be a simple concern to know that you are making a difference. It might be that you want to demonstrate to the wider organisation that scrutiny is worth investment. Or it might be to provide assurance to the public that their concerns and issues are being properly addressed.
Whatever the reason, it’s important to know what exactly you want to achieve. After all, if time and effort is going to be spent then it’s good to know why.
Key questions before you start
Before getting into the frameworks here are three questions I think it’s worth asking to prepare the ground:
1.What works well for us already?
2.Who are our audience and what do they need?
3.Exactly what question should we be trying to answer?
From a research perspective, it’s always worth spending time to the right question. What aspect of impact do you want to focus on? What’s a meaningful question for your audience? What’s a manageable question to answer given the resources at hand?
Frameworks can help you in a number of ways. They can help you organise the work you are doing more coherently, provide you with a clear method, give you inspiration, offer something practical you can adapt or maybe just challenge you to think differently.
The frameworks below help you to think about the three main aspects of demonstrating impact; data collection, analysis and presentation.
So, here are the six frameworks (there will be others out there of course). I hope you find something useful.
1. Policy impact (Constitution Unit, University College London)
Described as the ‘gold standard’ by those in the know, this report details extensive research into the work of parliamentary select committees over a number of years. Usefully the report details eight aspects of policy influence that might be used as hooks to hang a discussion upon:
- Direct government acceptance of committee recommendations (discussed above)
- Influencing policy debate
- Spotlighting issues and altering policy priorities
- Brokering in policy disputes
- Providing expert evidence
- Holding government and outside bodies accountable
- Generating fear (anticipated reactions)
Selective Influence: The Policy Impact of House of Commons Select Committees, Meg Russell and Meghan Benton, Constitution Unit June 2011 (full report here)
2. Outcomes (Institute of Government)
Anything from the Institute of Government is likely to be excellent in my opinion, and this report on parliamentary scrutiny is no exception. It provides a framework for assessing impact that sets out a series of possible outcomes alongside the questions you might want to ask and the quantitative and qualitative evidence that you might need to demonstrate each. The outcomes are:
Parliamentary Scrutiny of Government, Hannah White, Institute of Government, (2015) (full report here)
3. Results scorecard (City and County of Swansea)
I’m very happy to recommend this approach as it was something I was involved in. ☺️ It’s essentially a mix of Results Based Accountability and the Balanced Scorecard approaches. Scrutiny’s annual report includes 24 indicators mapped against 4 simple questions. There is a narrative for each question and, because this has been done for a numbest of years, changes over time can be analysed. The 4 questions are:
- How much scrutiny did we do?
- How well did we do it?
- How much did scrutiny affect the business of the Council?
- What were the outcomes of scrutiny?
You can download the annual report for 17/18 here.
4. Return on investment (Centre for Public Scrutiny)
If you a looking for something a little more hard edged, this report from CfPS (a.k.a. the people who know about scrutiny) should be right up your street. It takes the idea of ‘return on investment’ (essentially a cost – benefit analysis) and applies it to scrutiny work. Calculating the cost of scrutiny work is perhaps the straightforward part. This report will also help you think about the ‘return’ in terms of:
- The process benefits
- The outcome changes
Tipping the Scales, Su Turner and Linda Phipps, Centre for Public Scrutiny (2012)
The report is here.
5. Governance (European Observatory on Health Systems and Policies)
This is my favourite governance framework (we all have one don’t we…). Drawing on a substantial review of the literature, this framework sets out five ‘themes’ that between them cover all of the aspects of governance. This is helpful if you want to show that scrutiny has an impact on the different aspects of good governance across the organisation. The framework is known as TAPIC for short which stands for:
- Integrity (things work as they should)
- Capacity (for policy development)
TAPIC has been published in Strengthening Health System Governance, Greer et al (2016) A shorter summary of the relevant chapter (which is by Greer, Wismar, Figueras and Mckee) can be found here (pdf).
6. Reach and significance (Research Excellence Framework)
This framework comes from the academic world where it has become essential for universities to be able to demonstrate their research impact. It’s not directly scrutiny or governance related but I think this is something that can be adapted. The key takeaway is that impact can be described in terms of the concepts of reach and significance (breadth and depth if you like) where:
- Reach = ‘the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact’.
- Significance = ‘the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries’
Consultation on the draft panel criteria and working methods, REF 2018/02 July 2018 (report here if you want to see it)