Error message

  • Deprecated function: The each() function is deprecated. This message will be suppressed on further calls in _menu_load_objects() (line 579 of /var/www/drupal-7.x/includes/menu.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Deprecated function: implode(): Passing glue string after array is deprecated. Swap the parameters in drupal_get_feeds() (line 394 of /var/www/drupal-7.x/includes/common.inc).

A Little Rough Data About Journal Refereeing in Philosophy

Published by Anonymous (not verified) on Fri, 01/04/2022 - 8:32pm in

Is there a refereeing crisis in philosophy? There has been a fair amount of discussion about this over the past couple of months. What was missing from much of this discussion, though, was data. So I asked for some.

I heard from around 40 philosophy journals, but only about half of them were able to provide the kind of information I was after in order to figure out how difficult it is for journal editors to find enough qualified and willing referees: the percentage of those they invite to referee a paper who accept the invitation and write the review.

20 or so journals is not as large a pool of useful data as I was hoping to dive into, but it’s not nothing. What did they say?

The journals that provided helpful data were a varied group in terms of prestige, breadth of subject matter (general/specialist), and popularity of subject matter in the profession. They also varied in the time periods from which they drew their answers, but most of the information was from within the past year (with just a couple stretching further back in time than that). I told the editors that I would not reveal journal-specific information, so in what follows, no particular journals are named.

Across the whole sample I found that, on average, about 40% of the invitations journals send to prospective referees are accepted. About half the journals in this group were within ±7 points of that average. Among the other half, a few were down between 20% and 29% referee acceptance rate, and a few were up between 55 and 60%.

Were there any patterns to which journals had which rates of referees’ acceptance of invitations to review? Given the relatively small sample, it is hard to come to any definitive conclusions. I will say that one thing that’s true of all the journals with rather low referee invitation acceptance rates is that they are not the the best known versions of the type of journal they are. (And by type I mean general or specialist, and if specialist, the specific area.) That said, the generalist journals in this subgroup are definitely known to most philosophers, and the specialist journals in this subgroup are definitely known to the philosophers working in those areas of specialization. So journal obscurity is not doing any explanatory work here. What is? Perhaps it’s higher competition (i.e., more journals) in that space relative to the number of potential reviewers.

One reason to think this explanation has merit is that it’s consistent with the following data, which struck me as odd at first. Since I’m not naming particular journals, I’ll use nicknames for three of them: the Very Good Journal in an Unpopular Area, the Possibly Best Journal in a Popular Area, and the Great Generalist Philosophy Journal. Of these three, the Very Good Journal in an Unpopular Area has the highest referee invitation acceptance rate (in the high 50s): 4 points above the Possibly Best Journal in a Popular Area and 18 points above the Great Generalist Philosophy Journal. The first journal is one of very few journals of its type, and faces little competition for reviewers, while the other two, despite having better reputations in the profession at large, are each one of many of their respective types, and so each face a lot of competition for reviewers.

That said, the supply of potential reviewers is lower for the Very Good Journal in an Unpopular Area than for the other two, owing to differences in the popularity of the areas they cover, so that may count somewhat against the competition explanation. There’s also the possibility that there is a bit of an “underdog” mentality among those working in the unpopular area that motivates them to be more inclined to accept referee requests; or perhaps, knowing that they work in a relatively small area, they feel a relatively less diffuse sense of responsibility for helping with credibility-enhancing institutions in that area, like the journal.

A different possibility is that the Very Good Journal in an Unpopular Area is seen by its specialist community as “their journal”—the one that matters to them and is somewhat definitive for the area it covers, and so members of that community are more motivated to referee for it. Such a possibility was suggested by a journal editor (not associated with any of the above three journals) in reflections they shared with me about their experiences at two different journals: a top generalist journal and a top specialist journal: “Whereas people just about never refused [to referee] for [the specialist journal], the very same people did perhaps a third of the time for [the generalist journal]. It could… be that [the specialists in this area] saw [the specialist journal] as ‘their’ journal in a way that they did not so regard a [top] generalist journal.”

The “competition” and “their journal” explanations, plus some observations about the kinds of journals that had a 30% or smaller referee invitation acceptance rate, suggest (but hardly prove, given the small amount of data) something like the following generalization: if the journal you edit is not among the very first places authors might send their manuscripts, you are going to have more trouble finding referees.

This explanation is compatible with there being multiple reasons for why a journal might be among the very first places an author sends their manuscript: the journal might be especially prestigious or high quality, or be focused on distinctive content, or be the center of professional gravity in a particular subfield (owing to its content, or perhaps to the editor’s influential role in the subfield, apart from the their work for the journal). Correlatively, it suggests possible routes for how journals might improve their referee invitation acceptance rate: excellence, distinctiveness, and influence.

Another possible explanation for disparate rates is offered by the aforemention editor: “the personal touch made all the difference.” At the specialist journal, the editor would personally email potential referees, and though they used a form letter template for such messages, he thinks they still were more likely to garner positive responses than the auto-generated messages from editorial management programs: “I’m inclined to blame publishers’ insisting on seeking referees through the editorial management software as a major (if not the only) reason editors are having so much more trouble procuring referees.”

Some further information:

  • the journal that reported the highest referee invitation acceptance rate is also a journal that has a reputation for a high rate of desk rejections
  • very few referees agree to referee a paper and then never submit a report (except at one of the responding journals, about which the editor said their rate of referee ghosting was between 10% and 20%)
  • almost no papers at the 20 journals that sent in information were rejected simply because of difficulties in securing referees and reports (again, except at the same journal referenced in the previous bullet point, at which this happens “occasionally”)

Lastly, it’s worth noting that very few journals maintain this kind of data. For almost all of the journals that contributed data for this post, the data had to first be produced. In many cases, editors judged it impossible or too time-consuming to do so, or believed the information was inaccessible owing to publishers’ practices or editorial management software. As with questions about submission topics, demographic data about prospective authors, and related matters, it seems it would be useful for our understanding of our profession (what is happening in it, its challenges, its history, its progress, and so on) were journals in the habit of maintaining and occasionally publishing data relevant to their editorial operations.

Reminder to Journal Editors Re: Refereeing Data

Published by Anonymous (not verified) on Wed, 16/03/2022 - 10:30pm in



Several journal editors have emailed me responses to my request for data regarding referee requests at their journals. If you haven’t yet, please at least provide an answer to this one question:

What percentage of the potential referees who you ask to referee a new submission agree to do so? Or, to get at the matter another way, on average, how many referee requests must you make per new submission?

[Jasper Johns, Numeral Series]

Some journals have provided figures for the past two years, others have done so for some subset of that time, others have picked what they take to be a representative sample of papers and provided information based on them. I know that the ease of acquiring this information varies from journal to journal, so I’m not being too picky in what I’m requesting. The idea is to gather enough information from a wide enough range of journals to be able to get a sense of whether there is indeed a “refereeing crisis,” and if so, whether there are any patterns to it.

As I mentioned in the previous post on this, when I report on this data I will not be revealing journal-specific data, nor will I mention the titles of the responding journals without first seeking permission to do so. Any information provided will be kept confidential. Please send the information to me at dailynouseditor@gmail.com.

Thank you.

Call for Refereeing Data from Journal Editors

Published by Anonymous (not verified) on Tue, 01/03/2022 - 5:38am in

Over the past couple of weeks we’ve seen complaints from journal editors about the difficulty of finding referees and managing the refereeing process in a timely way but also some commentary suggesting that there may not be a problem.

[Jasper Johns, Color Numeral Series]

Is there a crisis or not? What are the facts? It would be great if we could say, “let’s turn to the numbers,” but we don’t have them. In light of this, a reader asked me to try to get said numbers. Good idea.

Philosophy journal editors, can you help us out?

Since for various reasons you might be reluctant to share such information publicly, I’m not asking you to provide your stats in the comments here; instead, please email (dailynouseditor@gmail.com) them to me. If I get some useful information, I will share it with DN’s readers in a follow-up post; however, I won’t release any journal-specific data without the explicit permission of the editor who provides it.

What we’re looking for is the referee request acceptance rate. That is, what percentage of the potential referees you ask to referee a paper agree to do so? (For our purposes let’s, if we can, consider only new submissions, and not R&Rs.)

Additional questions, if you have the information and time to share it: (a) what percentage of referee requests go completely unanswered? (b) what percentage of referees who agree to referee a paper fail to do so before you decide to find an alternate referee? and (c) how many papers, of those that made it through the desk-rejection phase, have you ended up rejecting owing to an inability to find enough suitable referees for it?

If you can’t get to these latter questions, don’t let that deter you from answering the main one, please. Thank you!