How not to distribute research funds

Citation counts are often used to proxy for the quality of an article, researcher or journal. They are not a perfect measure, everybody agrees on that, but they have proven to be a useful starting point for evaluation. Sometimes they are taken very seriously, too seriously, for the distribution of funds and pay. But at least this is done within a field, as it is obvious that citing conventions and in particular frequencies differ from field to field.

Javier Ruiz-Castillo goes further in trying to infer how budget priorities should allocated across research fields by using citations counts. Of course, for this one first needs to have a good understanding of how citations are distributed. Roughly, citations are distributed following power laws with fields and subfields. This means that few articles garner a lot of citations, while many go empty (especially in Business, Management and Political Science). And if I understand the paper right, one can apply readily a multiplier to compare the citation frequencies across fields. And these multipliers then make it possible to compare researchers or research units across fields within, say, a country, as long as one assumes that an adjusted citation is equally worth citing. For example, is political science worth the same support as biomedical engineering after using these multipliers, to take two randoms fields? And the "size" of the field is important as well. Here the author makes an attempt at some definitions of size which I frankly did not understand.

That said, I wonder why I forced myself in reading this paper. First is it indigestible because it is poorly written and uses very bad analogies. Second, because trying to compare fields and use citations for the allocation of funds or prizes across then is impossible because you have no identification: in statistical speak, the fixed effects capture all the variance. You can only compare how well a field does in a country compared to the rest of the world, but this cannot measure how important the field is. You need more information than just citations.