Saturday, January 16, 2010

The Paper Formatting Gestapo

I've recently had two conference submissions "rejected with prejudice" for violating the formatting requirements. In both cases, it was because the lead grad student on the paper didn't read the call for papers carefully enough. (I'll take my share of the blame for not checking it, but sometimes it's pretty hard to check without pulling out a ruler and counting lines of text by hand.) Now, I totally agree with those papers being rejected: it's essential that papers adhere to the formatting requirements. On the other hand, it certainly does not help that every conference uses slightly different guidelines. Here's a typical formatting convention:
Submitted papers must be no longer than 14 single-spaced pages, including figures, tables, and references. Papers should be formatted in 2 columns, using 10 point type on 12 point leading, in a text block of 6.5" by 9".
Another one:
Submissions must be full papers, at most 14 single-spaced 8.5" x 11" pages, including figures, tables, and references, two-column format, using 10-point type on 12-point (single-spaced) leading, with a maximum text block of 6.5" wide x 9" deep with 0.25" intercolumn space.
Yet another (from SIGCOMM 2009):
  • Submissions MUST be no more than fourteen (14) pages in 10 point Times Roman (or equivalent font). This length includes everything: figures, tables, references, appendices and so forth.
  • Submissions MUST follow ACM guidelines: double column, with each column 9.25" by 3.33", 0.33" space between columns.
  • Each column MUST contain no more than 55 lines of text.
  • NOTE: For the submission, you may use the following LaTeX template to ensure compliance.
  • The final copy will be 12 pages using the SIGCOMM standard 9 pt format; this is less than what you might be able to fit in 14 pages at 10pt, and so there is no value in pushing the envelope.
  • Provide an abstract of fewer than 200 words.
  • Number the pages.
  • Do not identify the papers' authors, per the Anonymity Guidelines.
  • On the front page, in place of the authors' names, the paper MUST indicate: the paper ID number assigned during the paper registration process and the total number of pages in the submission.
  • The paper MUST be submitted in PDF format. Other formats (including Postscript) will not be accepted. We must be able to display and print your submission exactly as we receive it, using only standard tools (Adobe Acrobat Reader), with no loading of special fonts.
  • Make sure that the paper prints well on black-and-white printers, not color printers. This is especially true for plots and graphs in the paper.
  • Make sure that the output has been formatted for printing on LETTER (8.5" by 11") size paper.
  • Make sure that symbols and labels used in the graphs are readable as printed, and not only with a 20x on-screen magnification.
  • Try to limit the file size to less than 15 MB.
This is getting silly. Even conferences sponsored by the same organization (say, USENIX or ACM) have different formatting guidelines. In one case, our paper was rejected because it was a resubmission from a previous conference that used an oh-so-slightly different format.

Now, I am all for having a consistent, and firm, formatting requirement for conference submissions. It's really unfair to authors that take pains to adhere to the guidelines when someone violates them by squeezing too much text into the paper. But isn't it time that we define a single standard for conference paper formatting that everyone uses?

Yes, I know there are various templates out there, but most of these are for the final proceedings, which can vary substantially from the submission format. Even worse, many of the "standard" templates floating around out there don't adhere to the submission guidelines anyway. What would help tremendously would be to have a canonical standard template (in various formats, e.g., LaTeX and Word) that everyone uses. Then it would be trivial to tell if someone had tweaked the formatting since their paper wouldn't look the same as the rest. This is the model used by EWSN 2010 (which required submissions to be in the Springer LNCS format) and it worked very well -- all of the submissions had consistent formatting.

A word on automatic format checkers. Both of the conferences we submitted to were supposed to be using an automated format checker that should have flagged the paper as violating the requirements when we submitted it. In both cases, this failed, and the program chairs (quite rightly!) rejected the paper once they discovered that the formatting was wrong. Unfortunately we were not careful enough and assumed that passing the automated check meant that we had done everything correctly. I like Geoff Voelker's Banal system, but it doesn't always work (mostly the fault of the underlying Ghostscript tool that it's based on). Even doing this kind of thing manually is a big pain -- Adobe Acrobat Pro lets you measure things like text blocks and font sizes, but it's a lot of manual effort.

Finally, as a TPC chair I have been on both sides of this, and I always hate rejecting papers due to formatting violations, especially when I know the authors have done good work. Dealing with formatting problems is a huge time sink when you're running a conference, and a standard format would save everyone -- authors, program chairs, reviewers, publication chairs -- a lot of trouble. I think it's time for the systems and networking community to simply define a single standard and get all conferences to use it.

15 comments:

  1. Very interesting Matt. I've very rarely seen papers rejected because of format. If possible, can you share a bit about the conference and just how strict the PC chairs were?

    ReplyDelete
  2. Wait -- what makes you think SIGCOMM is a systems conference?

    ReplyDelete
  3. In my experience, I have often found that the templates provided by conferences often do not meet their own formatting requirements!

    ReplyDelete
  4. I don't know which conferences you're talking about, but I know at least one reasonable counterexample. During the SOSP '09 process, Tom asked his grad students to measure the text area of all the suspicious papers. I think this was principally the set of papers that failed the format checker. Tom only rejected on formatting if the authors managed to submit papers that violated the total text area requirements by more than a few percent, and that ended up being only a few of the total set of submissions and around half of those that failed the format check. I can't imagine you could ask for more lenience.

    The templates out there exist, but of course people change the documentclass and assume everything else is fine, forgetting about their baselinestretch-ing or the margin formatting commands that have to go in the main .tex file anyway. (Why does the default SIGCOMM template *not* force letter paper??) I don't think that problem would go away with any 'canonical' template.

    Given how many hours people spend trying to adjust whitespace to make that last bib item fit on the fourteenth page, I figure they can spend the 20 minutes it takes to carefully read the CFP.

    ReplyDelete
  5. formatting requirements enforce length requirements, a heinous obsolete vestige.

    think how much better your papers would be if you didn't have to make all the compromises imposed by length requirements!

    ReplyDelete
  6. @peter: getting rid of the length requirement (which is both a max and a min in practice) is a great idea, but it seems like the community is stuck in this mode. We should take the SIGGRAPH model and weight papers by normalized (contribution/length). So many systems/networking papers, especially SIGCOMM papers, are 4-6 pages bloated into 14. And several each year are 16 pages crammed into 12, with a linked Tech Report and multiple appendices.

    ReplyDelete
  7. Ben - I'd rather not say, but let me say that some TPC chairs seem to enjoy being extremely pedantic. Of course it is well within their right.

    Dan - Tom's approach sounds very reasonable to me. SOSP was not one of the conferences I was referring to.

    Peter - I am all for length requirements. As a reviewer there is only so much of one paper I can take!

    ReplyDelete
  8. Why don't we just say that the paper is technically lousy?

    ReplyDelete
  9. Hi Matt,
    It was an interesting post and very helpful for a grad student like me. I would also love to see a post from you regarding the review process for conference submissions. Recently I got reviews from a top sensor network conference where I had reviews from two expert reviewers with totally opposite verdict (strong accept versus strong reject). I am not trying to say that I have done everything right and hence I disagree with the second reviewer. I think the current review process for conferences is fundamentally flawed (especially when a reviewer does a lousy job). Your analysis/comments may help us a lot.
    Thanks....

    ReplyDelete
  10. Anonymous - getting diametrically opposed reviews is not that unusual! And it is absolutely true that some reviewers are just plain wrong - the system is not perfect. But feel free to email me about it and we can discuss in more detail. I have an idea of which conference you may be talking about ;-)

    ReplyDelete
  11. How far from the formatting requirements was your paper? Was it an issue like going 5% over the margins, or was the formatting grossly off? I'd be pretty upset if I had a paper rejected for exceeding the margins by an amount that could only be detected using a ruler.

    ReplyDelete
  12. Jonathan - with automated tools there is little difference between eyeballing it and using a ruler. In one case our margins were off by about half an inch. In the other the line spacing was too tight. I guess both of these infractions allowed us somewhere in the ballpark of an extra page worth of text. Anyway, I'm not griping about having those papers rejected -- we were clearly violating the formatting requirements -- just the inconsistency across conferences.

    ReplyDelete
  13. And I thought the research world was as nice as the smurfs ;-)

    Hopefully my poster abstracts at EuroSys won't be both rejected due to this... I searched for the correct template to use as I couldn't recall where I saw the link to the good one, until I noticed where it was on the homepage, and had to axe the text out to use 10pt...

    ReplyDelete
  14. It took five years, but we finally got SIGPLAN to agree on a single LaTeX style file. Although the result has been an enormous boon to SIGPLAN authors, the process of developing the style was unbelievably bureaucratic and consumed a lot of volunteers' time. Having put in all this work, we would fight to the death against anything mandated by ACM. It was exactly because ACM mandated something idiotic that my SIG finally got off its butt and did something.

    If you really want to make something happen I think you need to go out and fight this battle one SIG at a time. And who knows what you do about IEEE, USENIX, ETAPS, and other meetings. The good news is that most people I talk to are willing to agree that there is a problem. But resolving the problem is *very* time-consuming.

    You have my sympathies about your papers.

    ReplyDelete
  15. I've a great affinity for this matter, and also I've been buy a lot of books about it, history about it and this kind of issues as you posting here. Thanks for share all this with us.

    ReplyDelete

Startup Life: Three Months In

I've posted a story to Medium on what it's been like to work at a startup, after years at Google. Check it out here.