Sunday, September 16, 2012

Is the Danish/Norwegian Model a Step Forward?

In Denmark, a new scheme for assigning resources to the universities according to their research performance was recently rolled out. One aspect of this scheme is a mechanism for calculating a score that captures the publication performance of a university. This mechanism is based on a similar mechanism used in Norway.

It works roughly as follows: Each journal is assigned to a subject area and to level 1 or level 2. Within a subject area, at most 20% of the world production (in number of papers) can be in a level 2 journal. Thus, level 2 roughly contains the 20% best journals. A level 1 publication yields 1 point, and a level 2 publication yields 3 points.

The following journals in my area (places where I have published or where it would be natural for me to publish) are at level 2:

ACM Transactions on Database Systems
Communications of the ACM
The Computer Journal
Data and Knowledge Engineering
GeoInformatica
IEEE Transactions on Knowledge and Data Engineering
Information Systems
International Journal of Geographical Information Science
Journal of Database Management
Journal of Intelligent Information Systems
SIGMOD Record
Software: Practice & Experience
The VLDB Journal

(This is according to a list that may no longer be completely up to date.)

There is some debate as to whether this ranking is intended to regulate the behavior of researchers. Thus, some pertain that the ranking should only be used at high levels of aggregation, e.g., at the level of universities. However, others believe that the ranking should be applied to each individual researcher.

In my view, if this ranking significantly affects the funding that universities and faculties at universities receive, which will be the case, then deans who care about funding are likely the take steps towards improving the performance of their faculties. The opposite seems to be simply irresponsible. A scalable way of improving performance is to reward those departments that perform well. So at the end of the day, each individual researcher will be confronted with his or her score and will be rewarded for improving it.

Is this good or is this bad? To shed light on this, let me consider what we have been doing in my old group at Aalborg University. Each five years, an evaluation of the research in my department is being carried out. In connection with a recent evaluation, my group prepared this ranking of database journals:

General Journals

1a. ACM Transactions on Database Systems
1b. The VLDB Journal, IEEE Transactions on Knowledge and Data Engineering
2a. Information Systems
2b. Data and Knowledge Engineering, the Computer Journal
3. Journal of Intelligent Information Systems, Knowledge and Information Systems, Journal of Database Management, Journal of Data Semantics, Information Sciences, etc.

Specialized Journals

1. Geoinformatica, Transactions in GIS

We prepared this ranking based on the prestige we associated with the different journals. Thus, we considered publication in ACM TODS to carry the highest prestige and publication in The VLDB Journal and IEEE Transactions on Knowledge and Data Engineering to carry similar, but less prestige than publication in TODS. Together, we viewed these at the top-3 general database journals. We expected few to disagree – and this expectation has held true. Then follows a group of three journals with clearly less prestige, and among these, we felt that Information Systems was the better one. The listing of journals at the fifth level is incomplete and consists of respectable journals.

Comparing the national ranking to our own, several observations may be made:

First, our ranking is much more detailed than the national one. Second, the journals in our four top levels are in the top level in the national ranking. Of the 5 journals mentioned in our fifth level, 2 are in the national top level, 1 is not listed, and 2 are at the national level 1. For the specialized journals, 1 is at the top level, and 1 is at level 1.

How do I perform well according to the two schemes? First, consider our own scheme. The pars of our work that fit in the general journals, we need to publish in quite specific journals. And publishing in Journal of Database Management is not a substitute for publishing in ACM Transactions on Database Systems. Other parts of our work are too specific to be publishable in general database journals. For example, some work is specific to geographic information systems and is thus published in specialized outlets in the GIS area.

Second consider the national scheme. How can I improve my current performance? First, I should avoid publishing in level 1 outlets. But I already avoid level 1, for which reason this will not change my current behavior and thus will not improve my performance. So what can I do? I think there is a lot to be gained to stop publishing in the best level 2 outlets and start aiming for the bottom ones. Publication in the bottom ones is much easier. The results need not be as good or novel or significant, and one can slice things thinner and thus write shorter papers that build extensively on previous papers. One can publish things that one could never publish in the best outlets. So I imagine that by choosing my outlets carefully, I could easily triple my score.

However, if I did that, my colleagues would wonder what went wrong. And I worry that many would no longer bother to talk with me. The currency of the science is reputation, and my reputation would go down the drain. Would this be in the best interest of Denmark? I think not.

And frankly, if I had to publish in the bottom outlets, my motivation to do science would be seriously affected – this is not something I would be likely to spend some 20 hrs of my spare time on each week.

One response is that we simply need to refine the national model. I worry that a committee that needs to cover all of computer science is unable to do this in a meaningful way, even if the committee members are great scientist. And too many compromises need to be made. So I worry that the entire approach is problematic.

Perhaps a bottom-up approach is better. With such an approach, one can let the scientists in a specific research area in a department (or across several departments in different universities) quantify their performance based on the best possible insight into their research area.

In closing, I note that I have not touched upon the unusual importance of conferences in computer science. This causes its own host of problems when creating a simple system that is to apply across all subject areas of science.

[Note: I prepared the above some time ago, and so it is not entirely up-to-date with current developments. Notably, conferences are now in the process of being considered.]