Peer Review and Self Assessment, VAGA Response

VAGA welcomes the advent of more self directed and better informed
method of assessment for RFO’s as advocated by the McMaster
Report and reflected in the recommendations of the McIntosh Report,

and in particular the move towards a renewed relationship of ”mutual
respect, trust and openness” between ACE and its RFOs. Following
consultation with members we have a number of observations.
 
2. The definition of ‘excellence’ proposed by McMaster is personal and
subjective; there is a danger that this could become self defining if the
scope of peer review is too narrow, lacks sophistication, vision or
relevant expertise. For the process to be credible there must be a
clearer understanding of how excellence is defined and total
transparency. There is concern not to replicate the bias that can creep
into academic peer review.
 
3. ACE needs to be more confident in asserting the value of the arts
across government and articulating the excellence value agenda
upwards. There is an inherent conflict with a performance management
driven culture and with PSA targets – especially given DCMS’ failure to
meet all but one of its Arts related SR04 targets and in particular those
around participation  
 
4. There is overwhelming consensus that both self-assessment and peer
review should focus on process rather than product and be able to
accommodate potentially conflicting outcomes such as reach and
depth, innovation and diversity, and failure.   
 
5. One size will not fit all and needs to take account of how an
organisation fits locally, regionally nationally, internationally, its type
and form of delivery, the percentage of ACE funding etc. 
 
6. Innovation by its very nature may only be recognised by a small peer
group  - if at all. The shifting parameters of user led and user generated
content across practice and organisations needs to be accommodated.  
A project-by-project approach may be the most manageable and
effective. 
 
7. There not only needs to be greater understanding of ‘excellence’ per se
but also a move to more robust ways of assessing cultural value/impact
and the development of a meaningful evidence base. (Art Angel is for
example currently undertaking research into this with Queen Mary
College, University of London) The cognitive cultural maps undertaken
by Liverpool 08 are cited as one potentially interesting model; as are
models developed around broader agendas such as well-being. 
 
8. There is concern as to the pressure it would put on an already
overstretched ACE staff especially at a time of 15% cost savings. The
capacity and work load of RFOs needs to be respected as the
downturn takes effect.
 
9. The long-term success of self-evaluation and peer review is dependent
on getting a renewed relationship of mutual respect and value between
ACE and RFOs right. The sector needs to be valued for what it does
well.  It must be a collaborative process and not simply a reporting to or
of ACE strategies. How will ACE respond to outcomes that do not ‘fit’ a
national or regional strategy but may be pushing the envelope?
 
10. There is concern as to the clarity of purpose of appraisal of RFOs and
how these would relate to the funding cycle. In the past ACE have not
always acted on or responded to the outcomes of self initiated reviews
of organisations. How would priorities be set between organisations
requesting an organisational review and ones being imposed by ACE?
 
11. The process must have the same value and level of meaningful
interrogation across the regions.
 
12. Most organisations already undertake regular self-assessment,
involving stakeholders and peers.  The best of this practice should be
built on.
 
13. The quality of self-assessment, its ambition and rigour depends in part
on the capacity and expertise of staff and therefore ACE needs to
invest in what are traditionally seen as ‘softer ‘areas’ of self directed
professional development and leadership. If staff do not have the time
to meet peers, to travel and keep up there own art form expertise, to
discuss policy and understand the broader environment within which
they work their ability to be rigourous and ambitious in their self-
assessment or peer review of others will be curtailed.  
 
14. The newly formed and forming Turning Point Strategy groups provide
an opportunity for peer review and supporting self-assessment but
cross regional and international fertilisation/comparators is also
necessary as is that with non RFOs also engaged in high quality work.
The Renaissance in the Regions programme for regional museums has
for example delivered high quality engagement and learning
programmes. The thinking needs to be 360 degrees and joined up to
maximise the potential both of these networks and self-assessment
and peer review. 
 
15. The practice of matching like organisations/creating new partnership
groups (as across universities) is seen as a potential way forward but
with the drawback of some organisations having a very small
immediate like-for-like peer group. (The recent Visual Arts Cultural
Leadership Network between three like RFOs in different regions
provides an interesting insight into the developmental value of like
organisations being networked particularly in operational as well as
artistic areas) This should be seen as an opportunity to share and
develop good practice as well as a means of assessment.
 
16. A rolling review over a longer time span with known reviewers/critical
friends is proposed as one model.
 
17. Given the amount of annual reporting that organisations already
undertake across a range of stakeholders (eg. Companies House,
Charity Commission, plus often more than one local authority and
through their own governance) this as an opportunity to look at ways in
which common reporting models might be developed. For many RFOs
ACE investment represents a very small amount of their income and
therefore to be effective self-assessment and peer review must have
value across the organisation,  be seen as relevant by other
stakeholders and not just become another very time consuming
demand.
 
Questions
 
1.  a) strongly agree
 
A good board will already be acting as a ‘critical friend’ and providing
elements of peer review to the organisation.  Board development should be
supported.
 
2. This is complex and it was generally felt that greater understanding of ways
of measuring cultural value is required for benchmarking to be really useful
(See para 7 above). It is felt that comparators would be useful across areas of
management such as salaries, pension provision, job specifications etc. There
will be some RFO data that may be commercially sensitive and should be
treated as such.
 
3.  The development of a common form of reporting would be welcome. (see
para 17 above.) The range of different LAs needs to be recognised, some
RFOs report to several for different parts of their programme and in differing
degrees. Local authorities may only fund one part of a programme eg
education., or be the largest funder.  The scope of this reporting will depend
on broader and very variable local authority policies and how culture has been
taken up by LAAs. 
 
4. Strongly agree, however organisations are already doing this. This needs to
take on board local variance – how is parity across different areas to be
achieved as well as across different types of organisation,  there will be for
example a greater range and type of external voices for example in
Westminster than say in the South West.  Good practice needs to be shared. 
 
5. We agree with statement a) and feel that grading will not be helpful.
Templates would be difficult to apply across a very diverse and evolving
sector.
 
6)  e) strongly disagree and believe a more sophisticated approach is required
that working with an organisation.
 
7.  We advocate a process-based approach, particularly given the nature of
visual arts and the amount  of new work commissioned and produced as
opposed to the performing arts,  rather than a reporting on a final product.
Reviewers should be both known critical friends and anonymous if this will
give meaningful assessment; a reviewer needs to know where the review ‘fits’
within the sector and for there to be clarity of purpose.
 
8. Sector reviews should be led by the sector and build on practice, expertise
mutual respect and transparency.   
 
9. b but depending on the capacity of the organisation and level of support
provided to enable the organisation to fulfil this role.
 
10. In principle but not as currently proposed. 

This response was made following consultation with members and a focus group meeting.