"Further work"

The phrase "we will need to undertake further work..." crops up several times in HEFCE's consultation document on the new Research Excellence Framework.

The biggest area that is still up for debate is how to deal with the arts, humanities, social sciences, mathematics and statistics. For these, HEFCE will "develop a quality assessment regime involving a light touch form of peer review informed by quantitative indicators". But, so far, "we have not undertaken significant development work on this," says HEFCE. The consultation is seeking preliminary input on how this system should work. The time line for this is not as tight though, it only needs to be in place by 2013, to inform funding from 2014.

Other areas that are still under development include how researchers should be assigned to institutions and subject groups, the algorithm to produce the quality indicator based on citations, and indicators of research income and numbers of research students will be used. HEFCE is also looking for suggestions for quantitative indicators that can be used to capture user value and the quality of applied research.

HEFCE has commissioned further technical advice, to report in early 2008, on some of the data and methodological issues, and will run a "substantial" pilot exercise next year.

All this shows that the REF is still very much a work in progress, and although the general framework has been decided, there is still room for academics to tweak it here and there to try and deal with the problems that are bound to crop up.

But the timetable is tight, as HEFCE acknowledges:

November 2007 to February 2008 - Consultation on key elements of the framework and on bibliometric indicators. In parallel, further work on developing bibliometric techniques.

March to August 2008 - Substantial pilot of proposed approach

Autumn 2008 - Further consultation and decisions on the framework and indicators to be used for the science-based disciplines

Early 2009 - Launch of full bibliometrics exercise for science-based disciplines

November 2009 - Output of bibliometrics exercise available for use in funding; decisions on new funding approach to be phased in from 2010

From late 2009 - Consult on light touch peer review to run in 2013

The science-based disciplines

Here are HEFCE's proposed subject groups for the science-based disciplines for the new Research Excellence Framework. The six groups map to Main Panels A to E and G in the 2008 RAE, except for Unit of Assessment 23, Computer Science and Informatics, which has been taken from Panel F and combined with the engineering subjects covered by Panel G. The rest of Panel F, mathematics and statistics, will be assessed with the light-touch peer review system being developed for the arts and humanities.

Subject group	RAE 2008 units of assessment	HESA cost centres
Clinical Medicine (RAE Panel A)	1 Cardiovacular Medicine 2 Cancer Studies 3 Infection and Immunology 4 Other Hospital based Clinical Subjects 5 Other Lab based Clinical Subjects	1 Clinical Medicine (part)
Health Sciences (RAE Panel B)	6 Epidemiology and Public Health 7 Health Services Research 8 Primary Care and Other Community Based Clinical Subjects 9 Psychiatry, Neuroscience and Clinical Psychology	1 Clinical Medicine (part)
Subjects Allied to Health (RAE Panel C)	10 Dentistry 11 Nursing and Midwifery 12 Allied Health Professions and Studies 13 Pharmacy	2 Clinical Dentistry 5 Nursing and Paramedical Studies 6 Health and Community Studies 8 Pharmacy and Pharmacology (part)
Biological Sciences (RAE Panel D)	14 Biological Sciences 15 Pre-clinical and Human Biological Sciences 16 Agriculture, Veterinary and Food Sciences	1 Clinical Medicine (part) 3 Veterinary Science 4 Anatomy and Physiology 8 Pharmacy and Pharmacology (part) 10 Biosciences 13 Agriculture and Forestry
Physical Sciences (RAE Panel E)	17 Earth Systems and Environmental Science 18 Chemistry 19 Physics	11 Chemistry 12 Physics 14 Earth, Marine and Environmental Sciences
Engineering and Computer Science (RAE Panel G, with part of F)	23 Computer Science and Informatics 24 Electrical and Electronic Engineering 25 General Engineering and Mineral & Mining Engineering 26 Chemical Engineering 27 Civil Engineering 28 Mechanical, Aeronautical and Manufacturing Engineering 29 Metallurgy and Materials	25 Information Technology, Systems Sciences and Computer Software Engineering 16 General Engineering 17 Chemical Engineering 18 Mineral, Metallurgy and Materials Engineering 19 Civil Engineering 20 Electrical, Electronic and Computer Engineering 21 Mechanical, Aero and Production Engineering

HEFCE launches post-RAE consultation

The consultation on replacing the Research Assessment Exercise with a metrics-based approach, at least in the sciences and medicine, has just been launched by the HEFCE.

HEFCE's published a Consultation document and two reports that it has been looking at, a general one from the bibliometric team at the University of Leiden in the Netherlands and a smaller one on interdisciplinary research from Evidence in Leeds. The main web page for what's now termed the "Research Excellence Framework" is here.

There'll be more here later in the day, but my first impressions are that there are few surprises in the proposals. Outside of six subject groupings (three for medicine, three for the rest of science), "light touch" peer review will be retained. In the sciences, universities will choose the staff to submit and assessment will then be based on bibliometrics, research income and numbers of postgraduate students.

There's no mention of selecting papers, so although it's not stated, it seems HEFCE is leaning towards assessing all publications, which could have a big impact on the willingness of academics to publish work they expect to get a low citation count, often for users. This isn't tackled directly, but HEFCE's flagging behavioural consequences as a concern and asking people to suggest additional metrics that could allow it to measure the value of research to users. 

Rather alarmingly for those who hoped the end of the RAE would eliminate a huge swathe of work, HEFCE talks about increases in workload in the short term. The issue seems to be in verifying who has actually published what, despite the promise of new technology.

And HEFCE's already flagging that the timetable for doing all of this is short and that it may struggle to find the expert resources it needs to make it all a success.