Fudging journal statistics

It’s time for a rant again, one that has been simmering for a long time now. The topic: the way some journals pervert the review process in order to make their stats look better than they are.

Good statistics? What’s that?

Basically, most journals want to present themselves as

  • publishing only the best research, and
  • publishing quickly, and
  • getting cited a lot.

Let’s ignore the last point for today, and focus on the first two. There is an easy trick for “improving” both in one go. And that’s to do with how you handle papers that need revisions.

Normally, when an editor sends a paper out for review, it comes back with one of the following recommendations:

  • Publish as is
  • Minor revisions
  • Major revisions
  • Reject

For arguments sake, let’s say there is a paper that both reviewers recommend be accepted pending minor revisions. And the reviewers did a good job: the paper is indeed almost perfect; it needs but a few days work to make it ready for printing.

In this case, I’d expect the editor to send the paper back to the authors with the request to either make the changes the reviewers want, or rebut them in a letter to the editor. That’ll take a week or two (or maybe three if the author(s) happen(s) to be in the field or on holiday), and that’s that.

How a “no!” may mean a “yes!”, but make the journal look better

What in fact often happens is that the paper is rejected. Yep, rejected! Not because the journal doesn’t want to publish the paper. Not because the revisions would take very long, so that there is a high risk of the paper requiring a second round of reviews to check if new publications published in the interim need to be addressed, too. Not because there may be a major change in the paper’s content caused by revisions.

No, the reason is usually two-fold:

  1. it drives up the journal’s rejection rate
  2. it shortens the journal’s turn-around time.

And this works this way: the paper may be re-submitted, and quite often the editor asks the author(s) to re-submit, but it still counts as rejected. Some journals are honest, and count papers that are later published not as ‘rejected’, but most don’t. And then the journal turns around as says “Look at our high rejections rate – we publish only the BEST stuff there is!”

Ah, no, I say, in fact you publish a lot of stuff that you still shove into your “rejection” rate. You may actually be publishing any BS sent your way, but your reject it once or twice first, which gives you a 50% or so rejection rate outright. DUH!

As for 2., that’s equally easy to see: the paper will very likely be altered and re-submitted. The date on the final version, the one that appears in print, should then be the one of the original, first submission – but whenever I have seen this entire thing happen as a reviewer, it was usually the date of the second submission that was on the paper. This cuts out all the initial editor assessment, review, review collation times. That’s nothing but cheating, but there you go.

What to do about this?

Well, I wouldn’t know. As an author, if you get one of those “rejected, please re-submit” emails, you know your paper needs only minor changes, and very likely will either not require a review (I have seen that happen!), or the review will be by the very same reviewers (provided they agree), thus fast and positive if you make the changes. So why would you send your paper elsewhere? You’d just risk additional trouble – stupid reviews, stupid editors, and so on.

So all we can do is make “blacklists” of journals that play this stupid game – but that needs to be done very cautiously, so that there are no erroneous entries. Thus, we can’t really do this publicly. I, for one, know a journal or two I will never submit to (again, in one case). At least not while the current editorial team is still there. And that’s about all I can do.

Aargh!

About these ads

About Heinrich Mallison

I'm a dinosaur biomech guy working at the Museum für Naturkunde Berlin.
This entry was posted in rants. Bookmark the permalink.

9 Responses to Fudging journal statistics

  1. Mickey Mortimer says:

    By reading comments in the numerous open access threads lately, I’ve noticed that one thing traditional publishers keep emphasizing as a “flaw” of PLoS is their low rejection rate. I don’t understand how this is supposed to be a bad thing. As Mike Taylor said in one thread, rejected papers just waste time getting reformatted and are eventually published _somewhere_. As long as it’s written by a competent scientist and not a crackpot/BANDit (which review doesn’t filter out anyway; ditto for terribly miscoded and incorrectly run analyses), it’s in the scientific community’s interest to have the paper published and accessable.

    • Indeed – pointing fingers at open access publishers is a fun game for certain members of BigPublishing. But that means you need to find a difference to point to – maybe this is the reason for the recent extreme increase (as far as I can see) in the stats fudging?

  2. sharmanedit says:

    I have worked as an in-house editor on two journals (Current Biology and Genome Biology) and have talked to the editors of a lot more, over the course of 13 years in journal publishing. I’ve never yet seen this practice nor heard of it happening. I’m not saying it doesn’t, because I can only talk about the journals I know about. But on the journals I worked on, it didn’t happen when I was an editor.
    I can’t see how it would work. Let’s imagine the editors decide that a paper needs only minor revisions, but they then decide to reject it purely to make their stats look good. How will they word the letter to the authors? It’s hard enough explaining rejection when you have good reasons, never mind when you don’t. How will they avoid the author coming straight back to them with a rebuttal asking for the rejection to be changed to an acceptance in principle? I can’t see any journal getting away with this and keeping any kind of reputation.
    If you get a rejection when it seems to you from the reviewers’ reports that the revisions needed are only minor, the reason might not be manipulation of statistics or anything dodgy. The editors may have interpreted the reviewers’ comments differently from you and may have good reasons for rejecting the paper, or at least reasons that make good sense to them on the information they have. I suggest reading the editors’ letter carefully and seeing if their reasons make sense. If they don’t, you are totally within your rights to write back asking for an explanation of why the paper has been rejected. And if you have definite proof that a journal has been engaging in false rejection for these reasons, I think a letter of complaint to the publisher or someone higher up the chain would be in order. Perhaps the journal should also be named and shamed publicly (though I understand why you personally might not want to do the naming).

    • sorry, this was held back for moderation becauseit was your first comment on this blog. Also, I was on holiday. Back now, will reply in a separate post (I guess) tomorrow.

    • Alright, got a few minutes here.

      First of all, I am glad to hear that this practice is not widely used. I didn’t think it was.

      You say you can’t see how it works, and until I saw it happen I would have said the very same thing: which author is so stupid to re-submit to a journal that gave a ridiculous rejection?

      The answer, however, is fairly simple: there are journals that are so uber-cool that people will swallow any shit to get their papers in there. Additionally, re-formatting a paper for a different journal is serious work. Often, it is simply easier to call the BS by re-submitting a barely altered manuscript and a lengthy reply to the editor. That’s what happened in the cases I was talking about. Also, I have heard, from funding agencies, that people should “avoid publishing in the same journals over and over” – we third-party-funding-dependent are good at listening to such “suggestions”. So there is motive to ‘make it into’ certain journals.

      The rejections letters were very easy to write (i.e., you just choose from the standard replies): the editor found both reviewers making critical remarks on some part of the manuscript, and felt these had to be addressed . “Substantial revisions” is what became of the “minor revisions” boxes we reviewers checked. The trick here is pretending that we (or one of us) used the wrong check box, and that the long-form ‘comments to the editor’ contained serious enough criticism. (OK, theoretically I am an idiot and the editor was right – but it is *my* area of expertise, and not his, and I read both reviews in full)

      That easy!

      As an author I’d be looking over the list of ‘comments for the authors’ in confusion, think “what the heck?” and re-submit after making a few changes. “Crazy reviewer? Crazy process? maybe…. let’s get this paper out.”

      You gave advice how authors should handle this – I wholeheartedly agree! And I do not think that any journal does this purely to get the rejection rate up. But there is a gray area between being tough (Naturwissenschaften does not accept ‘major revisions’) and doing anything you can to be seen as an “elite” journal. And in at least two cases, at least one editor took things a tad too far.

  3. SanSaurio says:

    Some months ago, a researcher told me that in a good jounal the referees suggest only minor revisions, but the editor finally reject the paper. This editor, in the beggining, decided acept the paper to revision…
    Finally, he published it in a “better” journal, but he lost a year.

  4. Mike Taylor caught Historical Biology Biology Letters in the act.
    http://svpow.com/2012/10/03/dear-royal-society-please-stop-lying-to-us-about-publication-times/

    Historical Biology Biology Letters happens to be one of the journals I had in mind when I wrote this post.

    Two strikes – congrats, BL, you just made my black list!

    EDIT: Freudian slip here, sorry – I am currently unhappy with HB for a totally different reason (and no, they did nothing wrong!). Thanks for catching my mistake, Mike!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s