The Programme Pilot & Community Consultation

In 2018 the UK Council for Graduate Education agreed an initiative to develop and trial a scheme to recognise good practice in doctoral supervision, and this was undertaken in 2019.

To better understand the appropriateness of this approach, and the demand for such a programme, the Council undertook a pilot and a community consultation. The background and results from each of these follow below:

The Piloted Programme

The programme pilot involved:

  • A set of statements of good supervisory practice.
  • Written applications by supervisors demonstrating the alignment of their practice to the statements.
  • Review by a panel of reviewers drawn from the UKCGE Executive.

Statements of good practice were devised on the basis of the literature on ‘effective supervision’ and following the model of the generic UK Professional Standards Framework for teaching and supporting learning, and clustered into three dimensions, namely:

  • Supervisory activities,
  • Knowledge and understanding
  • Principles.
The original programme framework
The Piloted Programme Dimensions of Good Supervisory Practice

In all, there were 22 separate criteria. In order to demonstrate the alignment of their practice to these criteria, participants were asked to write a reflective account of up to 5000 words, addressing these criteria and giving at least one example of their own practice under each main heading.

Applications were reviewed against the criteria by a panel of at least two reviewers. Where reviewers agreed, their decisions were final, but in the event of disagreement, the application would be referred to another reviewer for adjudication.

Pilot Participants

13 institutions from England, Scotland & Wales agreed to participate in the pilot project. An HEI in Northern Ireland was approached to participate but did not respond.

The participating HEIs were:

  • Bournemouth University
  • Coventry University
  • Liverpool John Moores University
  • Sheffield Hallam University
  • Swansea University
  • University College London
  • University of Brighton
  • University of East Anglia
  • University of Edinburgh
  • University of Glasgow
  • University of Manchester
  • University of Wales Trinity Saint David
  • University of York

These institutions provided 52 volunteer participants from a variety of disciplines, experience levels, and seniority.

Job Title#
Senior Lecturer 8
Professor of… 7
Reader 6
Associate Professor 5
Lecturer 5
Head of… 2
Principal Academic 2
Programme Director 2
Senior Research Fellow 2
Chair of… 1
Principal Lecturer 1
Research Associate 1
Academic Field#
Professions & Applied Sciences 12
Medicine 10
Education 5
Natural Sciences 5
Social Sciences 5
Arts 4
Computer Sciences 2

List of Academic Fields: https://en.wikipedia.org/wiki/List_of_academic_fields

Number of doctoral candidates supervised to completion

The pilot formally began on 1st March 2019 when the Good Supervisory Practice Framework, a Guide for Applicants, and a sample application were sent to the participating institutions for distribution to supervisors.

This was followed up on March 8th 2019 with an hour-long webinar for participants in which they were taken through the criteria in detail with suggested examples.

Participants were given until the 3rd May 2019 to write and submit their applications.

The Application Review Process

43 of the original 52 participants in the pilot institutions completed their applications, a response rate of 83%.

These were reviewed by volunteers drawn from the UKCGE Executive Committee plus Professor Stan Taylor, with each application being assessed by two reviewers. Reviewers were given a guide to reviewing the applications and a standard recording template.

The overall standard of the submissions was extremely high and in nearly all cases the reviewers agreed that they fully met the criteria for recognition. In the handful where reviewers agreed that the criteria had not been met, this was for two main reasons.

The first was a failure to engage with the literature on research supervision; a few accounts of practice contained no reference at all to the literature and hence were unable to demonstrate that their practice was grounded in the scholarship of supervision.

The second was a focus upon describing institutional policies and procedures rather than upon giving personal examples of practice which meant that a judgment could not be formed on the practice of the individual in relation to the criteria.

Verbal feedback on the review process from two reviewers was positive. Most felt that the process had gone relatively smoothly, although it was perhaps unnecessarily complex in view of the number of criteria and unpacking evidence relating to those for knowledge and understanding and professional values within the areas of activity.

Evaluations of the Pilot Project by Participants

34 of the participants in the pilot completed an evaluation questionnaire, although not all responded to every question. The key findings were:

Criteria

Participants were asked whether the criteria represented good practice in research supervision; 91% agreed that they did while 9% thought that they did partially (N=33). Of those who added explanatory comments (N=20), the vast majority were supportive but several felt that it would be helpful to consolidate the statements into a smaller number.

Do you think the criteria accurately reflect good supervisory practice?

Clarity of Guidance Notes

48% of the participants thought that the notes provided were very easy to follow, 45% that they were fairly easy, and 6% that they were difficult (N=33).

Participant comments (N=27) were overall highly supportive but some asked for greater clarity in the amount of evidence required and how far there was a need to demonstrate engagement with the scholarly literature.

Ease of Application

6% of participants felt that it had been very easy to produce the application, 72% that it had been fairly easy, and 22% that it had been fairly difficult (N=32).

Of those who offered comments about how it could be improved (N=30), the main issues were the number of criteria and the complexity of interweaving knowledge and understanding and the professional values into the areas of activity.

How easy was it to complete the application?

Time Taken to Complete the Application

The frequency distribution of reported times taken to complete the application is set out below:

Days
<112345>5 
1467365N=32
3%13%19%22%9%19%16%101%

The median time taken was just under three days, but there was a spread from less than a day through to several weeks.

Benefits of Completing the Application

All of the 32 participants who responded said that completion of the application had been beneficial to their practice.

Did you find the exercise of completing the reflective application beneficial in itself?

Improving Research Supervision in Future

89% of participants thought that completing the application would improve their research supervision in future, while 12% were not sure (N=33).

All of those answering the question added their comments, nearly all stressed the value of reflection in itself and a large number also mentioned the usefulness of being exposed to and following through the scholarly literature on supervision.

Do you think completing it will improve your research supervision in the future?

Value of Recognition

Participants were asked about how beneficial it would be for them to have their practice recognised in the following terms:

Very beneficialFairlyNot veryNot at allN=30
Professional development7327100
Attracting candidates533810101
Career advancement434314100

In all cases, the large majority thought that recognition would be ‘very’ or ‘fairly beneficial’ in these contexts.

The Community Consultation

In addition to the pilot, the UKCGE conducted a sector-wide Community Consultation on the proposed scheme between the 6th – 31st May 2019.

Members of the postgraduate community were invited to download the criteria and answer questions relating to the criteria, the potential benefits of supervisory recognition, and about any other comments they had on the proposed scheme.

In all, the consultation attracted a wide range of, in all, 116 responses although not all answered all of the questions.

Criteria

Respondents were asked how far the criteria reflected good supervisory practice:

  • 43% said ‘very well’,
  • 49% ‘well’,
  • 5% ‘not well’ and
  • 4% ‘not well at all’

(N=108).

How well do you think the criteria reflect good supervisory practice?

Value of Recognition

As with the pilot participants, respondents to the community survey were asked how beneficial it would be to have their practice recognised in a range of aspects:

  Very BeneficialBeneficialNot Very BeneficialNot Beneficial At AllN=108
Professional development45301213100
Career advancement32411611100
Attracting candidates3232132299
Supporting funding applications27351721100
  • 75% of respondents saw recognition as ‘very’ or ‘beneficial’ to their professional development,
  • 73% in the advancement of their careers,
  • 64% in attracting candidates,
  • 62% in supporting funding applications.

Other Comments

Finally, respondents were asked an open question about if they had any other comments about the proposed recognition programme. The comments of the 75 who gave useable responses fell into three clusters:

  • 25 (33%) expressed strong support for the proposed recognition scheme
  • 21 (28%) offered conditional support
  • 29 (39%) were opposed to it.

Among those expressing strong support, the proposed scheme was widely welcomed as offering a long-delayed opportunity for supervisors to be recognised for their professionalism.

Among those expressing conditional support, the conditions included simplifying the scheme and making applications less onerous and adding specific criteria, for example in relation to mental health and advising on non-academic career development.

Among those who were opposed, two main reasons were given. The first was a concern about possible duplication with existing recognition schemes, and both those associated with the HEA and the Staff and Educational Development Association (SEDA) were mentioned in this context. In all, 7 respondents (11% of the total) were opposed for this reason. The remaining 22 (28% of the total) rejected the whole proposal, in most cases on the ground that it was designed to generate metrics for use by institutions to measure and manage supervisory performance.

Updating the Programme

In response to both pilot and the community consultation the UKCGE, under the guidance of Professor Stan Taylor, updated the Research Supervision Recognition Programme.

Simplified Structure

The 22-attributes across the three dimensions of good supervisory practice were condensed to just a single set of 10 criteria. This simplified both the process of completing the structured self-reflection and the reviewing of applications.

Enhanced Criteria

The ten criteria were also enhanced to include references to supporting the personal wellbeing of candidates, and advising candidates on non-academic careers.

By the Sector, for the Sector

The UKCGE has made a commitment to make the Good Practice Framework and related resources freely available, and will continue to review and evaluate the Programme based on feedback from the postgraduate research community.

Programme Approved by UKCGE Executive Committee

Following these amendments to the scheme and presenting the above report, the Executive Committee of the UKCGE approved the programme on the 17th September 2019.