Home » Blog » Integrity

Category: Integrity

Six frameworks to help demonstrate the impact of scrutiny

Thinking about how you demonstrate the impact of your scrutiny work? Wondering how you might improve? Well, there are frameworks that can help. This post highlights six.

No doubt there are things you are already doing. Annual reports, updates to council meetings, features in council newspapers, press releases, videos, recommendation trackers are just a few of the ways that scrutineers show that they are making a difference.

This post is for anyone who wants to develop or review their approach. It suggests six solid frameworks that can be used and adapted. Hopefully you will find one that you works for you.

Why demonstrate the impact of scrutiny?

There are a number of reasons why you might want to demonstrate the impact of your scrutiny work. It might be a simple concern to know that you are making a difference. It might be that you want to demonstrate to the wider organisation that scrutiny is worth investment. Or it might be to provide assurance to the public that their concerns and issues are being properly addressed.

Whatever the reason, it’s important to know what exactly you want to achieve. After all, if time and effort is going to be spent then it’s good to know why.

Key questions before you start

Before getting into the frameworks here are three questions I think it’s worth asking to prepare the ground:

1.What works well for us already?

This solution focussed question invites you to notice the things that are already effective for you in your own context. What’s working well that might be developed or expanded? What’s working well in one committee that might be tried in others? Are there good examples of demonstrating impact being used for other aspects of council business?

2.Who are our audience and what do they need?

This design thinking question asks you to think about your final product and who will be using it. Whatever the final product you have in mind, you need to talk to whoever you want to impress and find out what they like and what they don’t.

3.Exactly what question should we be trying to answer?

From a research perspective, it’s always worth spending time to the right question. What aspect of impact do you want to focus on?  What’s a meaningful question for your audience? What’s a manageable question to answer given the resources at hand?

Six frameworks

Frameworks can help you in a number of ways. They can help you organise the work you are doing more coherently, provide you with a clear method, give you inspiration, offer something practical you can adapt or maybe just challenge you to think differently.

The frameworks below help you to think about the three main aspects of demonstrating impact; data collection, analysis and presentation.

So, here are the six frameworks (there will be others out there of course).  I hope you find something useful.

1. Policy impact (Constitution Unit, University College London)

Described as the ‘gold standard’ by those in the know, this report details extensive research into the work of parliamentary select committees over a number of years. Usefully the report details eight aspects of policy influence that might be used as hooks to hang a discussion upon:

  • Direct government acceptance of committee recommendations (discussed above) 
  • Influencing policy debate 
  • Spotlighting issues and altering policy priorities 
  • Brokering in policy disputes 
  • Providing expert evidence 
  • Holding government and outside bodies accountable 
  • Exposure 
  • Generating fear (anticipated reactions)

Selective Influence: The Policy Impact of House of Commons Select Committees, Meg Russell and Meghan Benton, Constitution Unit June 2011 (full report here)

2. Outcomes (Institute of Government)

Anything from the Institute of Government is likely to be excellent in my opinion, and this report on parliamentary scrutiny is no exception. It provides a framework for assessing impact that sets out a series of possible outcomes alongside the questions you might want to ask and the quantitative and qualitative evidence that you might need to demonstrate each.  The outcomes are:

  • Evidence
  • Analysis
  • Openness
  • Learning
  • Processes
  • Context
  • Democracy

Parliamentary Scrutiny of Government, Hannah White, Institute of Government, (2015) (full report here)

3.  Results scorecard (City and County of Swansea)

I’m very happy to recommend this approach as it was something I was involved in. ☺️  It’s essentially a mix of Results Based Accountability and the Balanced Scorecard approaches.  Scrutiny’s annual report includes  24 indicators mapped against 4 simple questions. There is a narrative for each question and, because this has been done for a numbest of years, changes over time can be analysed. The 4 questions are:

  • How much scrutiny did we do?
  • How well did we do it?
  • How much did scrutiny affect the business of the Council?
  • What were the outcomes of scrutiny?

You can download the annual report for 17/18 here.

 

4. Return on investment (Centre for Public Scrutiny)

If you a looking for something a little more hard edged, this report from CfPS (a.k.a. the people who know about scrutiny) should be right up your street.  It takes the idea of ‘return on investment’ (essentially a cost – benefit analysis) and applies it to scrutiny work. Calculating the cost of scrutiny work is perhaps the straightforward part. This report will also help you think about the ‘return’ in terms of:

  • The process benefits
  • The outcome changes

Tipping the Scales, Su Turner and Linda Phipps, Centre for Public Scrutiny (2012)

The report is here.

5. Governance (European Observatory on Health Systems and Policies)

This is my favourite governance framework (we all have one don’t we…). Drawing on a substantial review of the literature, this framework sets out five ‘themes’ that between them cover all of the aspects of governance.  This is helpful if you want to show that scrutiny has an impact on the different aspects of good governance across the organisation.  The framework is known as TAPIC for short which stands for:

  • Transparency
  • Accountability
  • Participation
  • Integrity (things work as they should)
  • Capacity (for policy development)

TAPIC has been published in Strengthening Health System Governance, Greer et al (2016) A shorter summary of the relevant chapter (which is by Greer, Wismar, Figueras and Mckee) can be found here (pdf).

6. Reach and significance (Research Excellence Framework)

This framework comes from the academic world where it has become essential for universities to be able to demonstrate their research impact. It’s not directly scrutiny or governance related but I think this is something that can be adapted.  The key takeaway is that impact can be described in terms of the concepts of reach and significance (breadth and depth if you like) where:

  • Reach = ‘the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact’.
  • Significance = ‘the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries’

Consultation on the draft panel criteria and working methods, REF 2018/02 July 2018 (report here if you want to see it)

 

Photo credit

The seven scrutiny kryptonites ☢️

When I wrote about The Seven Scrutiny Powers, I had a reply on twitter mentioning kryptonite.  And I thought, yes, we should be talking about this – aspects of local government culture that neutralise the seven scrutiny superpowers (transparency, accountability, participation, solutions, calling-in, assurance and capacity).

This may seem negative but think of it it as an anti-problem. If we wanted to ensure scrutiny was completely ineffective what would we do? Then we can do exactly the opposite of course.

I’m not claiming that any of this is necessarily intentional.  Rather I suspect that these are culturally convenient ways of doing things that develop, maybe even unconsciously, over time.  So, there is no supervillan, but these kryptonite will be recognisable to anyone who has worked in the scrutiny world.

Also remember, while kryptonite has a negative effect on scrutiny, it also does damage to the rest of the organisation.

And, while each type of kryptonite neutralises some powers more than others, each type of kryptonite has some effect on every aspect of scrutiny.

So, in no particular order there they are.

The Seven Scrutiny Kryptonites

1.  The party card

a.k.a. Political management

While party politics contributes to a healthy local democracy, party politicking, party cliques and the use of the party whip, can all have a negative effect.  Party management can undermine accountability and call-ins by ensuring that backbenchers give executive members from the same party an easy ride and ensure that the most controversial decisions pass unchallenged.

2.  The vault of secrets

a.k.a. Information restriction

The flow of information can be controlled in a number of ways including the overuse of confidentiality clauses, limited information about future cabinet decisions and a culture of keeping rather than sharing.  Information restriction can hinder scrutiny as it seeks to make the organisation more transparent, and as it seeks to plan its work.

3.  The deflector shield

a.k.a. Executive defensiveness

Scrutiny cannot work well if the cabinet are not open to be challenged, are inward looking or take a ‘we know best’ approach.  Executive defensiveness can seriously affect the scrutiny power of accountability and render the solutions proposed from in depth inquiries of little value.

4.   The room of doom

a.k.a. Poor meeting environments

The physical environment that scrutiny is given to work in can have a detrimental effect if the room or the seating arrangements are wrong.  Poor meeting environments can particularly affect the power of participation by isolating the public from councillors and creating an atmosphere of over formality.

5.  The cloak of invisibility

a.k.a. Limited publicity

The oxygen of publicity is important for scrutiny, not least so that the public can get involved.  By offering limited or even no publicity to scrutiny, and focusing instead only on what cabinet and council are doing, scrutiny is left in the shadows and the power of participation is neutralised.

6.  The game of disdain

a.k.a. Talking scrutiny down

The organisational stories that are told about scrutiny affect its status and the way people, particularly councillors, engage with it.  By talking scrutiny down the ability to provide serious accountability is affected as is the potential to offer the organisation additional policy development and problem solving capacity.

7.  The support stiffler

a.k.a. Restricted resources

Decisions about the resources and support available to scrutiny have a major bearing on scrutiny’s ability to work effectively.  Without access to its own advice, support and research, scrutiny becomes dependent on others and limited in what it can do.  When the resources that scrutiny has access to are restricted, at least in comparison, to cabinet, for example, the ability to exercise any of the seven superpowers are limited.

Thank you to @shelleyburke23 for the kryptonite idea and to @alanfinch4 @thomoli @tonybovaird @davidjabb @cfps_ed @jj_mclaughlin @vcusworthlabour @jonathanflowers @bryony1963 for their comments and suggestions.

The seven scrutiny superpowers ✊

As many new councillors are getting to grips with their new roles and many more old hands are getting back up to speed, here’s a reminder of the superpowers that scrutiny councillors have to change the world.

Sometimes it’s easy to get lost in the process, in the paperwork and the meetings, but don’t forget what scrutiny can do to help the good citizens of the area.

Of course scrutiny has some formal legal powers, but it’s the powers that come from the drive, creativity and commitment of the scrutiny councillors themselves that are far more interesting.

Far better than simply the legal powers – these are the superpowers, and there are seven of them.

The seven scrutiny superpowers

1.  The power of transparency

By asking questions and publishing the answers, by asking for reports and putting them in the public domain, scrutiny has the special ability to make local government more open and transparent.

2.  The power of accountability

By requiring cabinet members and other decision makers to give an account of themselves in public, by asking the questions that the public want the answers to and by publicly highlighting both concerns and praise, scrutiny is able to ensure that the public interest stays at the heart of decision making.

3.  The power of participation

By inviting the public and others outside the council to share their views and work with councillors, scrutiny can ensure that people’s voices are heard, that their involvement makes difference and that the issues that matter to citizens are acted on by decision makers.

4.  The power of solutions

By taking time to explore difficult and challenging issues in depth and talking to a wide range of people, scrutiny has the ability to find solutions to the most difficult policy problems and to suggest ways to improve the services that matter to people.

5.  The power of challenge

By acting as a critical friend, scrutiny can ensure that cabinet decisions are made as they should be – in the public interest.  Scrutiny can look at cabinet reports before decisions are taken and call-in decisions once they have been made.

6.  The power of assurance

By acting as a watchdog and by ensuring that services are delivered as they should be, scrutiny can give the public confidence that the local council operates with integrity and performs effectively.

7.  The power of capacity

By working on the things that really matter to the council and the public, scrutiny can make sure that all of the resources of the council are used to maximum effect.  In particular scrutiny can harness the power of the majority of councillors who are not in the Cabinet.

 

That’s the superpowers of scrutiny right there.

POW!

PS. Of course it’s important to be wary of the seven scrutiny kryptonites as these can neutralise scrutiny super powers. Read about them here.

 

Updated 19.12.18

The TAPIC framework: A really useful governance map

If you are looking for a method to organise your governance work, the TAPIC framework is one of the most robust in research terms and one of the simplest to use.  Drawing on an extensive literature review, the authors of this framework capture five categories of governance; transparency, accountability, participation, integrity and capacity. These five categories together provide a useful map of the issues that any public body will want to think about when considering their governance arrrangements.

About the Framework

The TAPIC framework comes from the world of health governance.  You can find the full details in chapter two of ‘Strengthening Health Systems Governance:  Better policies, stronger performance’ (2016) edited by Greer, Wismar and Figueras.  A shorter summary of the chapter (which is by Greer, Wismar, Figueras and Mckee) can be found here (pdf).

Although the framework has been developed in the health context, the authors have drawn on the wider public administration literature for their extensive review.  This identified 60 attributes of good governance which the authors then reduced to the five categories of the TAPIC framework.

Crucially the five categories do not overlap and and are independent of each other.  The result is a map that covers all of the governance terrain.  This is helpful because it gives confidence that all of the important aspects of governance will have been covered in any review.

It’s also important to stress that the framework is not prescriptive.  The five categories are not ‘ingredients of good governance’ as the authors put it, but rather boxes that tell us what needs to be addressed.  While simply seeking more of one of the categories may not necessarily be better, there may also be trade offs and conflicts between the different categories.

Nevertheless, the map provided by the framework will help governance practitioners to ensure that what needs to be covered, is covered.  In fact I’ve used the TAPIC framework to set the categories for this blog.

Here then are each of the five categories in brief.

Transparency

Transparency refers to the way that public bodies inform those outside the organisation about decision making and the decision making process.  At its best, say Greer et al, ‘transparency produces information that is available, useful and accurate so it can be used by those who would rely on, plan with or seek to influence the organisation.  The result is trust’.

Accountability

Accountability refers to the relationship between the public body and another organisation or forum that must be informed of decisions, is entitled to have those decisions explained to it and has the power of sanction.  Good accountability goes beyond ensuring compliance through punishment and instead sees public bodies being supported by the expertise and experience of those holding the body to account.

Participation

Participation ‘means that affected parties have access to decision making and power so that they acquire a meaningful stake in the in the work of the institution’ (Greer et al). As well as ownership, participation can also generate information to improve decisions and ensure that organisations operate in line with democratic principles.

Integrity

Integrity refers to the ethical dimensions of a public body such as legality and anti-corruption, but it also refers more widely to the good management of the organisation.  Processes should be understood and predictable, roles and responsibilities should be clear and rules should also be clearly stated and observed.  Beyond this integrity relates to the overall mission and purpose of the organisation and its coherence in pursuing this.

Capacity

Capacity, or more specifically policy capacity, refers to the ‘ability to develop policy that is aligned with resources in pursuit of goals’ (Forest et al cited by Greer et al).  This can mean the support provided, often by a small team, to those involved in the policy process that can ‘transform ideas into workable, well designed policies’.  Skills in areas such as analysis and research will be important as will the ability to manage networks of relevant stakeholders.

 

 

 

 

 

The International Framework: A great option for your governance review

If you are reviewing the work of your public body and thinking about how your review should be structured, the International Framework is a great option for you to think about.  It sets out seven key principles of good governance and provides plenty of suggestions about what good practice looks like.  

For any public body, change is is the only constant.  As a member of a public body, therefore, you will want to regularly reassure yourself that your governance arrangements are as good as they can be.  You will also want to make regular improvements.  For local councils this type of review is likely to be done by the audit committee.  For other public bodies, such as schools, the work might by done by a sub committee or by the governing body as a whole.

Either way, your governance review needs a structure to ensure that everything that needs to be covered, is covered.

The International Framework

One very robust framework you can use is The International Framework: Good Governance in the Public Sector (2014).  This was developed jointly by the Chartered Institute of Public Finance and Accountancy (CIPFA) and the International Federation of Accountants (IFAC).

It can be downloaded from the CIPFA website here.

The International Framework builds on previous similar work, draws on a substantial literature review and is informed by an international panel.  You can be confident that it is tried and tested.

While it has been used to assess the governance of bodies as large as the European Union, it is general enough to be employed by any public body down to the smallest parish council or school.

Seven Principles

This framework suggests seven principles of good governance:

  • Behaving with integrity, demonstrating strong commitment to ethical values, and respecting the rule of law
  • Ensuring openness and comprehensive stakeholder engagement
  • Determining outcomes in terms of sustainable economic, social, and environmental benefits
  • Determining the interventions necessary to optimize the achievement of the intended outcomes
  • Developing the entity’s capacity, including the capability of its leadership and the individuals within it
  • Managing risks and performance through robust internal control and strong public financial management
  • Implementing good practices in transparency, reporting, and audit, to deliver effective accountability

As the report suggests:

The Framework should be useful for all those specifically associated with governance— governing body members, senior managers, and those involved in scrutinizing the effectiveness of governance, including internal and external auditors. It also provides a resource for the public to challenge substandard governance in public sector entities.

Using the International Framework

The report breaks down each principle into sub principles (e.g. Behaving with Integrity) and provides a description of what good governance looks like for each including what, as a governing body, you should be doing, and some useful ‘implementation tips’.

You can use the International Framework to review one aspect of your governance arrangements or you can use it to review the whole lot.

The report is, of course, a general guide and your organisation, the context you work in and your governing body are all unique in many ways.  Good governance is not something that can be taken off the shelf.  General principles, no matter how well researched, always have to be adapted to work in a particular setting.

For this reason you should aim to use this framework (or any other one), as a broad guide to help your own exploration of good governance.

You can use the descriptions as benchmarks and ask a series of key questions:

Is this something we want to do?

What aspects of this do we already do well?

How do we get to the next level?

What practical next steps do we want to try?

How will we know if we have been succesful?

Answering questions like these is always done best in a group; ideally with the whole committee or governing body.  This way you can make use of the expertise that everyone has and their experience of what works.

Even better if you can have good conversations with people outside of your committee or governing body to ensure that you get a truly rich picture.

And if you are looking for a nice way to discuss those questions, why not try zero to ten scales – I’ve written about them here.

 

 

Image: Relationships between the Principles for Good Governance in the Public Sector, from The International Framework: Good Governance in the Public Sector (2014), by the Chartered Institute of Public Finance and Accountancy (CIPFA) and the International Federation of Accountants (IFAC). p.11