Archive for July 17, 2012

Computer science publication culture: where to go from here?

Computer science publication culture and practices has become an active discussion topic. Moshe Vardi has written a number of editorials in Communications of ACM on the topic that can be found here and here, and these have generated considerable discussion. The conversation on this issue has been expanding and Jagadish has collected the writings on Scholarly Publications for CRA, which is a valuable resource.

The database community has pioneered discussions on publication issues. We have had panels at conferences, discussions during business meetings, informal conversations during conferences, discussions within SIGMOD Executive and the VLDB Endowment Board – we have been at this since about 2000. I wrote about one aspect of this back in 2002 in my SIGMOD Chair‘s message.

The initial conversation in the database community was due to the significant increase in the number of submitted papers to our conferences that we were experiencing year-after-year. The increasing number of submissions had started to severely stress our ability to meaningfully manage the conference reviewing process. It became quite clear, quite quickly, to a number of us that the overriding problem was our over-reliance on conferences that were not designed to fulfill the role that we were pushing them to play: being the final archival publication venues. I argued this point in my 2002 SIGMOD Chair’s message that I mentioned above. I ended that message by stating that we “have been very successful over the years in convincing tenure and promotion committees and university bodies about the value of the conferences (rightfully so), we now have to convince ourselves that journals are equally valuable and important venues to publish fuller research results.” The same topic was the focus of my presentation on the panel on “Paper and Proposal Reviews: Is the Process Flawed?” that Hank Korth organized at the 2008 CRA Snowbird Conference (the report of the panel appeared in SIGMOD Record and can be accessed here).

This discussion needs to start with our objectives. In an ideal world, what we want are:

  • Fast decisions on our papers so we know the result quickly;
  • Fast dissemination of the accepted results;
  • Meaningful and full reviews of our submissions; and
  • Fuller description of our research.

The conventional wisdom is that conferences are superior on the first two points and the third point is something we can tinker with (and we have been tinkering with for quite a while with mixed results) while the fourth objective is addressed by a combination of increasing conference paper page limits, decreasing font sizes so we can pack more material per page, and the practice of submitting fuller versions of conference papers to journals. Data suggest that the first issue does not hold – our top journals now have first round review times that are competitive with “traditional” conferences (e.g., SIGMOD and ICDE). The second issue can be addressed by adopting a publication business model that relies primarily on on-line dissemination with print copies released once per volume – this way you don’t wait for print processing, nor do you have to worry about page budgets and the like. Note that I am not talking about “online-first” models, but actually publishing the final version of the paper online as soon as the final version can be produced after acceptance. Journals perform much better on the last two points.

In my view, in the long run, we will follow other science and engineering disciplines and start treating journals as the main outlet for disseminating our research results. However, the road from here to there is not straightforward and there are a number of alternatives that we can follow. Accepting the fact that we, as a community, are not yet willing to give up on the conference model of publication, what are some of the measures we can take? Here are some suggestions:

  1. We should move away from a single submission deadline. With a single deadline per year, we tend to submit papers that may not yet be ready since the overhead of waiting for an entire year is far too high. This has many drawbacks as one can imagine. Multiple and frequent submission deadlines encourage authors to continue working and submit at the next deadline. The frequency of submissions should be at least bi-monthly. Less frequent submissions are problematic in that the temptation to submit even if a paper is not yet ready will be too high.
  2. We should move to a journal-style reviewing process. This means in-depth reviews with multiple rounds where the authors can engage in a discussion with the reviewers and editors. This is perhaps the most important advantage of journals as it encourages reviewers to invest the time to do it right rather than the uneven conference reviews that we frequently complain about. With multiple submission deadlines, the review load at each cycle should be more manageable and should allow for more proper reviews.
  3. Publication of the accepted papers can happen in a number of ways. Although I have a preference to publishing as soon as the papers are accepted, this is not the most important issue – if necessary for other reasons, the papers can be accepted throughout the year, but they can be published all at once at the conference, as proceedings are done today.

These are things that we currently do – Proceedings of VLDB (PVLDB) incorporates these suggestions. It represents the current thinking of the VLDB Endowment Board after many years of discussions. Although I had some reservations at the beginning, I have become convinced that it is better than our traditional conferences. However, I am suggesting going further:

  1. We should not have conference PCs that change every year. We should have have PCs that serve longer and provide some continuity – just like journal editorial boards do. In this context, we need to change our culture. My experience with serving on PVLDB as a reviewer this year was quite enlightening. It appears to me that people are still in the PC mode of thinking and not in journal review board mode of thinking – many still focus too much on acceptance rates and making binary decisions. Those of us who serve in these roles need to change our thought process – our job is not to reject papers, but to ensure that the good research results get to see the light of day.
  2. We should reorganize our PCs. We should have a small group of senior PC members and a larger group of reviewers. Senior PC members should be the senior members of the community. Together with multi-year service that I advocate above, this provides a means of training the junior members of the community to achieve a broader view of the field, how to evaluate novelty and originality, and how to write meaningful reviews. The senior PC members should really work as associate editors of journals and truly oversee the process, not just manage it.  Malcolm Gladwell, in his book, Outliers, talks about the “10,000 hour rule” indicating that success in any endeavor requires 10,000 hours of practice. I am not suggesting that junior members should serve 10,000 hours as reviewers, but doing this right so that it helps the science and the community takes practice and we should build it into our modus operandi. PVLDB is doing something this year along these lines – they have associate editors and reviewers, and associate editors assign papers to reviewers and then make decisions (like in a journal). I view this as a very positive step towards true journal-style reviewing.
  3. We should rethink the structure of our conferences. I have always been somewhat surprised that we allocate 20-25 minutes to presenting a paper whose text is already available but only 5 minutes for questions and discussions, all the while claiming that conferences are valuable for one-on-one discussion of research  (in addition to networking). The organization seems to me to be a one-way monologue rather than a discussion. It may help to reduce the presentation time and allocate more time for discussion of each paper (or group of papers). We should consider more extensive use of poster sessions where real discussions usually take place. To enforce the view that journal publications are no less valuable than conference publications, and to encourage direct submissions to journals, we should allocate some sessions at our conference to selected papers that have appeared in TODS in the previous year. This gives an opportunity for those papers to be discussed at the conference.

As I said earlier, my personal belief is that we will eventually shift our focus to journal publications. What I outlined above is a set of policies we can adopt to move in that direction. For an open membership organization such as SIGMOD, making major changes such as these requires full engagement of the membership. I hope we start discussing.

Blogger’s Profile:
M. Tamer Özsu is Professor of Computer Science at the David R. Cheriton School of Computer Science of the University of Waterloo. He was the Director of the Cheriton School of Computer Science from January 2007 to June 2010. His research is in data management focusing on large-scale data distribution and management of non-traditional data. His publications include the book Principles of Distributed Database Systems (with Patrick Valduriez), which is now in its third edition. He has also edited, with Ling Liu, the Encyclopedia of Database Systems. He serves as the Series Editor of Synthesis Lectures on Data Management (Morgan & Claypool) and on the editorial boards of three journals, and two book Series. He is a Fellow of the Association for Computing Machinery (ACM), and of the Institute of Electrical and Electronics Engineers (IEEE), and a member of Sigma Xi.