Too Sexy For My Books

One of the first acts of the new Australian federal government was to scrap the RQF (Research Quality Framework - similar to the UK’s RAE).  Yesterday its replacement process was announced - Excellence in Research for Australia:  ERA.

One of biggest issues concerning the defunct RQF was that of metrics - in particular citation analysis.  As I have previously mentioned, the debate over the validity of citation analysis and metrics has long been called into question.  Due to more and more countries looking at citations and metrics to assist in determining the research quality (and grabbing division of funding) of higher educational institutions debate on this issue is gaining considerable attention.

The skewing of authority through citations and references is well recognised, yet the solution is far from evident.  The main concern in using such metrics to allocate funding is the considerable degree of citation game playing conducted by authors and even institutions.  Such game playing tricks include:

• excessive self-citation (where the author will cite to their own works in an extravagant, indulgent manner)
• citing to co-workers, friends and other researchers in their clique - ‘links for the boys’
• only citing to ‘rock stars’ in the field
• citations to papers that do not really support their arguments or even have absolutely nothing to do with their research at all.
• Negative and even downright malicious citations that attack another authors research (and sometimes personal attributes and behaviours - “Your mother was a hamster and your father smelt of elderberries”).
• There are also considerable language, geographical and even gender, race and religious issues and biases that can contort the way in which in author cites.

As long as such behaviours are present (and encouraged) using citations and reference metrics as the base for determining a publications quality and authority and in turn, determining funding and reputation of the publication and research institutions, is always going to be controversial and flawed.

So whats the solution?  Well, frankly there isn’t.  The RQF was criticised for its metrics focus (along with other things), yet one of the keystones of the new proposal is, yup, you guessed it - metrics:   The ERA would use research metrics, such as citation data.  You cannot get away from the fact that despite the flaws, despite the bias and blatant gaming, citation metrics are one of the strongest methods in determining quality and authority of a publication.

“Never mind the underlying social reality, feel the data“  - Cronin, B; A Hundred Million Acts of Whimsy