July 17, 2012
Computer science publication culture and practices has become an active discussion topic. Moshe Vardi has written a number of editorials in Communications of ACM on the topic that can be found here and here, and these have generated considerable discussion. The conversation on this issue has been expanding and Jagadish has collected the writings on Scholarly Publications for CRA, which is a valuable resource.
The database community has pioneered discussions on publication issues. We have had panels at conferences, discussions during business meetings, informal conversations during conferences, discussions within SIGMOD Executive and the VLDB Endowment Board – we have been at this since about 2000. I wrote about one aspect of this back in 2002 in my SIGMOD Chair‘s message.
The initial conversation in the database community was due to the significant increase in the number of submitted papers to our conferences that we were experiencing year-after-year. The increasing number of submissions had started to severely stress our ability to meaningfully manage the conference reviewing process. It became quite clear, quite quickly, to a number of us that the overriding problem was our over-reliance on conferences that were not designed to fulfill the role that we were pushing them to play: being the final archival publication venues. I argued this point in my 2002 SIGMOD Chair’s message that I mentioned above. I ended that message by stating that we “have been very successful over the years in convincing tenure and promotion committees and university bodies about the value of the conferences (rightfully so), we now have to convince ourselves that journals are equally valuable and important venues to publish fuller research results.” The same topic was the focus of my presentation on the panel on “Paper and Proposal Reviews: Is the Process Flawed?” that Hank Korth organized at the 2008 CRA Snowbird Conference (the report of the panel appeared in SIGMOD Record and can be accessed here).
This discussion needs to start with our objectives. In an ideal world, what we want are:
1. Fast decisions on our papers so we know the result quickly;
2. Fast dissemination of the accepted results;
3. Meaningful and full reviews of our submissions; and
4. Fuller description of our research.
The conventional wisdom is that conferences are superior on the first two points and the third point is something we can tinker with (and we have been tinkering with for quite a while with mixed results) while the fourth objective is addressed by a combination of increasing conference paper page limits, decreasing font sizes so we can pack more material per page, and the practice of submitting fuller versions of conference papers to journals. Data suggest that the first issue does not hold – our top journals now have first round review times that are competitive with “traditional” conferences (e.g., SIGMOD and ICDE). The second issue can be addressed by adopting a publication business model that relies primarily on on-line dissemination with print copies released once per volume – this way you don’t wait for print processing, nor do you have to worry about page budgets and the like. Note that I am not talking about “online-first” models, but actually publishing the final version of the paper online as soon as the final version can be produced after acceptance. Journals perform much better on the last two points.
In my view, in the long run, we will follow other science and engineering disciplines and start treating journals as the main outlet for disseminating our research results. However, the road from here to there is not straightforward and there are a number of alternatives that we can follow. Accepting the fact that we, as a community, are not yet willing to give up on the conference model of publication, what are some of the measures we can take? Here are some suggestions:
1. We should move away from a single submission deadline. With a single deadline per year, we tend to submit papers that may not yet be ready since the overhead of waiting for an entire year is far too high. This has many drawbacks as one can imagine. Multiple and frequent submission deadlines encourage authors to continue working and submit at the next deadline. The frequency of submissions should be at least bi-monthly. Less frequent submissions are problematic in that the temptation to submit even if a paper is not yet ready will be too high.
2. We should move to a journal-style reviewing process. This means in-depth reviews with multiple rounds where the authors can engage in a discussion with the reviewers and editors. This is perhaps the most important advantage of journals as it encourages reviewers to invest the time to do it right rather than the uneven conference reviews that we frequently complain about. With multiple submission deadlines, the review load at each cycle should be more manageable and should allow for more proper reviews.
3. Publication of the accepted papers can happen in a number of ways. Although I have a preference to publishing as soon as the papers are accepted, this is not the most important issue – if necessary for other reasons, the papers can be accepted throughout the year, but they can be published all at once at the conference, as proceedings are done today.
These are things that we currently do – Proceedings of VLDB (PVLDB) incorporates these suggestions. It represents the current thinking of the VLDB Endowment Board after many years of discussions. Although I had some reservations at the beginning, I have become convinced that it is better than our traditional conferences. However, I am suggesting going further:
1. We should not have conference PCs that change every year. We should have have PCs that serve longer and provide some continuity – just like journal editorial boards do. In this context, we need to change our culture. My experience with serving on PVLDB as a reviewer this year was quite enlightening. It appears to me that people are still in the PC mode of thinking and not in journal review board mode of thinking – many still focus too much on acceptance rates and making binary decisions. Those of us who serve in these roles need to change our thought process – our job is not to reject papers, but to ensure that the good research results get to see the light of day.
2. We should reorganize our PCs. We should have a small group of senior PC members and a larger group of reviewers. Senior PC members should be the senior members of the community. Together with multi-year service that I advocate above, this provides a means of training the junior members of the community to achieve a broader view of the field, how to evaluate novelty and originality, and how to write meaningful reviews. The senior PC members should really work as associate editors of journals and truly oversee the process, not just manage it. Malcolm Gladwell, in his book, Outliers, talks about the “10,000 hour rule” indicating that success in any endeavor requires 10,000 hours of practice. I am not suggesting that junior members should serve 10,000 hours as reviewers, but doing this right so that it helps the science and the community takes practice and we should build it into our modus operandi. PVLDB is doing something this year along these lines – they have associate editors and reviewers, and associate editors assign papers to reviewers and then make decisions (like in a journal). I view this as a very positive step towards true journal-style reviewing.
3. We should rethink the structure of our conferences. I have always been somewhat surprised that we allocate 20-25 minutes to presenting a paper whose text is already available but only 5 minutes for questions and discussions, all the while claiming that conferences are valuable for one-on-one discussion of research (in addition to networking). The organization seems to me to be a one-way monologue rather than a discussion. It may help to reduce the presentation time and allocate more time for discussion of each paper (or group of papers). We should consider more extensive use of poster sessions where real discussions usually take place. To enforce the view that journal publications are no less valuable than conference publications, and to encourage direct submissions to journals, we should allocate some sessions at our conference to selected papers that have appeared in TODS in the previous year. This gives an opportunity for those papers to be discussed at the conference.
As I said earlier, my personal belief is that we will eventually shift our focus to journal publications. What I outlined above is a set of policies we can adopt to move in that direction. For an open membership organization such as SIGMOD, making major changes such as these requires full engagement of the membership. I hope we start discussing.
M. Tamer Özsu is Professor of Computer Science at the David R. Cheriton School of Computer Science of the University of Waterloo. He was the Director of the Cheriton School of Computer Science from January 2007 to June 2010. His research is in data management focusing on large-scale data distribution and management of non-traditional data. His publications include the book Principles of Distributed Database Systems (with Patrick Valduriez), which is now in its third edition. He has also edited, with Ling Liu, the Encyclopedia of Database Systems. He serves as the Series Editor of Synthesis Lectures on Data Management (Morgan & Claypool) and on the editorial boards of three journals, and two book Series. He is a Fellow of the Association for Computing Machinery (ACM), and of the Institute of Electrical and Electronics Engineers (IEEE), and a member of Sigma Xi.
Copyright @ 2012, M. Tamer Özsu, All rights reserved.
Comments are closed
@Tamer: Thank you for a great post!
@Data Management Community:
As part of the ongoing discussion within the SIGMOD community on possible changes to the current SIGMOD conference publication process, I have prepared a quick survey to capture the preferences of the data management community in relation to this post.
Could you please go to http://bit.ly/sigmodsurvey to voice your opinion? Thank you!
Tamer – I like the idea of increasing the times assigned to the poster mode where there is more opportunity for discussion. I also like the idea of asking people to serve on a committee for a longer period of time – say 2 years.
Nice blog, Tamer! I would also like to see our community doing one more thing: De-focus on papers, and paper counts, and focus on results instead. I.e.: Produce cool results. Build systems that embody them. Measure them. Be proud of them. THEN publish them. Back in the early days, back when we were working on coal-fired databases, the interest seemed to be on SOLVING PROBLEMS and/or BUILDING THINGS that worked well (e.g., by finding good new algorithms). These days the focus seems to be almost solely on writing papers; we are now dying of metricitis as a result. I HATE IT when I hear colleagues raving about how many papers researcher R published, or how what R’s XYZ-index (pick your favorite metric) is, etc. Once upon a time the measure of success discussed, e.g., at tenure time, was IMPACT, usually as measured by what one’s tenure letters said about the candidate’s major results and their reach. Then yes, there were sanity checks: Has enough journal papers, and/or papers in major conferences? Teaching ratings above the bar? Service above the bar? Check, check, check. Impact high, all boxes checked -> tenure. Those were the days… (And hopefully still are at the top schools? Not sure.) Papers are a way of sharing results – not the end in themselves – we need to get back to that way of thinking.
I fully agree with you Mike. A full discussion of this particular aspect requires a lot more space (or some discussion over beers) as there are a number of contributing factors. One that I wanted to highlight here, in hopes of getting some conversation going, is our changing attitude to hiring in academia. Our expectations have grown unrealistically from the days you and I entered the field to now. My experience with many institutions is that applicants who do not have 3-4 publications in top places have no shot at even getting an invitation for an interview. You take into account our low acceptance rates, and working backwards, grad students should start writing papers about their second term of studies. And they are aware of this, or we make them aware in an attempt to assist them. So, there is a laser-like focus on improving one’s publication record to have a decent shot at an academic job. Unfortunately, this doesn’t work well in research that involves system building or even extensive validation/experimentation (which is yet another topic for discussion).
I like the spirit of this Tamer, particularly as an author. But I fear that we don’t have the capacity to support it long term. Having chaired the first VLDB under the new monthly-submission regime, I can say that by the end of the year most of the people involved were exhausted by the pace and the volume of reviewing.
It would be useful to run the numbers on the status quo, and on your proposal. Do we have the aggregate reviewing bandwidth in the community to take on this project? Or will the net effect simply be collective “burnout” in the course of a couple years?
That is a concern Joe. I may be an outlier, but I served on the PVLDB last year and I was perfectly fine reviewing 1-3 papers each month. I thought I had enough time to read the paper, have it read by my students and compare notes before I finalized the reviews. I think over the entire year I was delayed only one month. I wonder if one of the problems is that we are not conditioned to operate this way. I was talking to colleagues from other sciences and their review times for each article is 2-4 weeks and they do this all the time. Our PVLDB papers are not that much longer than theirs. I wonder how much of our current difficulty is because we have not gotten used to the new system and have not adjusted our personal workflows. I don’t know. I have been asked to continue serving on PVLDB for one more year and I suspect I’ll have a better perspective at the end of the second year. So far, I have not had any difficulty keeping up with the workload for the past 14-15 months.
Wecan run the numbers but that misses the point I am making. My major point is that we will (eventually) shift to a journal-first-and-foremost model consistent with all the other science and e gineering disciplines. There is no good argument why we should not and many good arguments why we should. The other suggestions are intermediate steps that we can take in getting there from here. When we are in the journal publishing model, we will handle this many papers within is time frame anyway. I was trying to get that main point across.
Tamer, thanks for this great post. As a PhD student, I have found the publication process overwhelming. Given a rolling (bi-)monthly deadline takes the pressure off and makes sure that as researchers, we are submitting the best work possible.
In the Semantic Web community (my research area), one of the main conferences, ISWC, has the senior pc member structure. I find this very valuable because during the review process, the senior pc member makes sure that sufficient discussion about the paper and reviews is made. I wish ISWC would follow the same rolling deadline as VLDB.
I agree with your proposal of changing the structure of paper presentations at conferences. I’m very concerned that workshops, which are suppose to spark discussion, are starting to become mini-conferences: 20 min talk and 5 min questions. Sometimes they even have lower acceptance rates than the main conference. As the organizer of a workshop, I want to make sure that there is discussion and I think the idea of having the poster presentations would work.
A final question: what is your opinion about the open review process, for example Semantic Web Journal (http://www.semantic-web-journal.net/)
You make a number of good points Juan. I think some of the db conferences a also moving into the senior PC structure – I served as one for CIKM this year. It does help somewhat if the senior members are on top of things and push for discussion.
The issue of workshops morphing into conferences has been going on for some time and I don’t like it. At some workshops discussion time is less than what they are at conferences, which is already low as I argued. My favorite workshop model is the Dagstuhl workshops where there are no proceedings, not every attendee has to “present” a paper, but everyone is heavily engaged in discussions on the topic. Give me one of those any day.
Finally your question about the open review process – I don’t know what it is, but I’ll go figure out and postmy views then.
Very nice blog, Tamer. I particularly like the idea of the presentation getting shorter and the discussion getting longer. Indeed, to devote adequate time to discussions in conferences these days, one has to skip sessions — which kind of defeats the purpose of having the talks. Which — being a chain PC chair these days — beings me to a related thought: arm-wrestling the authors of the papers to come to the conference and present seems a bit absurd. Yes there are strong reasons, e.g. conference attendees expect to see the authors of accepted papers and discuss with them, but as we move a to a more meaningful paper-conference relationship we can decouple the publication from the attendance/presentation (perhaps even credit them separately).
Natassa, this is. It a problem that I paid too much attention to until now, because the number of authors who fail to attend and present their papers have been low in the database conferences in my experience. However, your experience seems to differ, and to the extent tha this is happening it is disturbing, and I agree that we might look at decoupling the two concerns.
A thought that occurs to me is whether this is the result of proliferation of conferences. I don’t know if this could be the reason.
Sorry, I don’t know what happened, but the first sentence should have read “…this is not a problem that I paid too much attention to until now…”
Thank you for a thoughtful post, Tamer. I thought I would add a couple more observations from PVLDB.
w.r.t. your point regarding immediate publication, PVLDB does do this now, in batch mode, approximately monthly. The production process still takes a few weeks from acceptance to publication.
w.r.t. your point regarding PC continuity — this was discussed by the VLDB Endowment trustees. There are benefits to continuity, and benefits to new blood. Currently, there is a recommendation to reappoint approximately 2/3 of the review board from the previous year, but it is not an enforced requirement, and is left completely up to the Editor-in-Chief each year. In practice, a substantial fraction, but well under 2/3 appear to be reappointed.
A related point has to do with how much year-to-year change is good. Journals have Editors-in-chief serve for multiple years, and each strives to leave their legacy. Conferences have a new program chair each year, and some PC chairs have innovative schemes they need the room to try out. Yet, I feel that many conferences, and even PVLDB, have too much change year-to-year. I am not sure what the right balance is, but this is an issue worth thinking about too.
Finally, in response to Joe Hellerstein, I’d like to say that I don’t see the numbers change w.r.t. the number of reviewers if the aggregate total number of submissions doesn’t change. There is a change in the distribution of workload — which some reviewers may prefer and others not. I think that year-round submissions, and management of revisions, definitely are more work for the editors, so authors should be grateful to the few who willingly take on this task.
Thanks for the clarifications regarding PVLDB Jag. With respect to the question of how much change should happen year-to-year, my main point is that we will eventually be moving to a journal publication model, so these are steps that we can take to get ready for that time when we will have to manage these submissions in a very different environment.
The issue of whether we will (or should) move to a journal publication model is one that we may wish to debate and discuss — that is really the main point. Besides the reasons I have enumerated here and elsewhere, I have been told recently that NIH discounts conference publications. If that is true, we do have a problem with joint work in life sciences.
There was a related panel at the recent CRA Snowbird conference that Moshe Vardi organized. There is a very nice summary of the panel at CACM Blog that SIGMOD members may wish to read: