Thursday, January 13, 2011

Journal editors are poor selectors of best papers

Journal editors are thought to be exceptional scholars who are capable of identifying the best papers and in particular those that will have the largest impact, typically measured by citation counts. But editors have considerable help: first peer reviewers make reports that should be informative, second authors to some extend self-select in the submission process, they would not send a paper where it has no chance of getting published. But it is difficult to evaluate whether an editor is doing a good job. However, when editors choose to put as lead paper the one they consider the best in the issue, one can in retrospect check whether these papers are cited the most. As I reported earlier, editors turn out not to be particularly good.

What about those papers that get the "best article of the year" award? Tom Coupé does a similar exercise and finds that they are more cited than the median article, and slightly more than the runner-up articles, but they are rarely the most cited in a year. You just cannot trust editors, even though they are even supported by a committee for such awards.

2 comments:

PB said...

Citations are a very partial measure of paper quality. Ordinary papers on a popular topic typically receive more citations than superb papers on a less popular topic.

Anonymous said...

As an editor, I try to put first the articles that I think are going to be the most cited, because I want them to be even more cited. They are the ones that are going to drive up the impact factor, so I want to promote the best the most, as this is where the highest return is.

Now, I may be wrong sometimes, but I definitely go for the citations, not where the article is superb.