I'm generally in favor of double blind reviewing (DBR) since, in my experience, it can at least help avoid some bias in the review process and "level the field" between well known and less well known researchers and groups. The use of DBR is often quite controversial though. Below I try to collect empirically based research on DBR. If you know of any results/papers that should be listed here please contact me.
Not much of the literature on DBR is from the software engineering (SE) or computer science fields. The ACM Transaction on Database Systems (TODS) is an exception though and both did an extensive review of the research on DBR and installed a DBR policy in 2006. However there seems to be a current trend that SE conferences are trying out or transitioning to DBR so below I also added the conferences and journals I know of.
To be clear, let's first define DBR and SBR (from H. Wallach):
- Double-blind peer review (DBR) = identities of a paper's authors and reviewers are concealed from each other.
- Single-blind review (SBR) = reviewer identities are concealed from authors but not vice versa
Empirical evidence in relation to Double Blind Reviews
- Reviewers can guess the actual authors about 25-42% of the time (as summarised in Budden et al 2008)
- US papers are evaluted more favorably (by both US and non-US reviewers), with US reviewers showing a stronger preference for US papers than non-US reviewers (Link 1998)
- Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review (van Royen 1998)
- Reviewers were more critical in DBR (Blank 1991)
- Female authors fared somewhat better in DBR than in SBR but effects were small (Blank 1991)
- Authors at near-top-ranked institutions and non-academics had lower acceptance rates but other groups largely unaffected (Blank 1991)
- SB reviewers were significantly more likely than their DB counterparts to recommend for acceptance papers from famous authors and top institutions (WSDM 2017 DBR Experiment). The paper quantifies the effect and says: "...estimated odds multiplier is around 1.5×, so the result is quite strong."
Double-blind review in Software Engineering (SE)
Summaries of results on Double-blind review
Empirical evidence on arbitrariness of peer review
- On a related note the NIPS 2014 experiment found that "conditional probability for an accepted submission to get rejected if examined by the second committee ... was 60% (acceptance rate 24%)". This re-analysis found that "The result
suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates
in future review processes.".
Computer Science Journals that have DBR
- ACM Transactions on Database Systems (TODS) - from 2007.
- ACM Transactions on Computer Education (TOCE) - from April 2017.
If you know of more CS/SE journals that use DBR please contact me and I'll add them here.
Software Engineering Conferences that have DBR
- SSBSE (Symposium on Search-Based SE), from 2014
- ASE (Automated SE), from 2016
- ISSTA (Int Symposium on SW Testing and Analysis), from 2016
- FASE (Fundamental Approaches to SE), from 2016
- FSE (Foundations on SE, incl ESEC), from 2017
- MSR (Mining SW Repositories), from 2017
- ICST (Int Conf on SW Testing...), from 2018
- ICSE (Int Conf on SE), from 2018
If you know of more SE conferences that have or are planning to use DBR please contact me and I'll add them here.