Assessing students, assessing ourselves

It's time we started measuring third level research - The Irish Times - Tue, Feb 26, 2013: "Research is akin to venture capital. We need to do a lot to get a little output. Success is rare and failure the norm.

Acceptance rates (success) in decent journals or in gaining leading research grants are often less than 10 per cent. Research grants can take months to complete with no guarantee of success.

And doing a paper takes time. It takes typically in excess of 100 hours of work to get a paper to a stage where one is happy to put it out for even working-paper review. And then it takes a long time to get it published. The paper is usually under review at symposiums and conferences and further hundreds of hours are spent refining and tweaking.

Typically a paper gets published after a couple of anonymous referees have commented on it. Doing this referee job takes time – typically up to five hours reading and writing per paper. If one does 12 reviews a year, that’s 60 hours, or the best part of two working weeks. Imagine the work of an editor of a journal receiving 300 or more papers a year.

Teaching also takes time beyond the classroom. For every hour spent in the class you need three hours to prepare, review, reflect and redo the work. Even if one has a stack of slides and a good strategy, it’s imperative to do a mock runthrough (that takes about the same time as teaching), noting as one does what is outdated, what is wrong, what doesn’t flow and so on. So a 16-hour class contact in the IoT doesn’t leave a lot of time in teaching term to do any research. There are always evenings and weekends, but any IoT staff who wish to research in a significant sense use the summer breaks to do it – the Christmas and Easter are usually taken up with marking essays.

By all means let’s look at metrics of activity. But let’s recall that these metrics typically measure output, not input, and therefore can’t be used as such for the measurement of efficiency or, still less, effectiveness. That’s not to say we should not measure research output. We should.

We have experience in the UK of several generations of research measurement. Many of us have been involved in these as assessors or as units of measurement. We can and should design a model that builds on these and improves on them, that measures research output, across units and the sector. This will show that there are many who simply do not engage in research (as measured). Every academic knows of people who simply turn up, teach and disappear. This puts an unfair burden on those who do research, and it is they who should be shouting loudest for such a metric.

Until universities, the Department of Education and the Higher Education Authority determine where the balance should lie between research , teaching and other roles we cannot begin to reward or discipline academics for “success” and “failure”.

As it is, we are not measuring research in any manner in Ireland.

Perhaps we should start..."