Those of you in Communication Research may have noticed something strange about the NRC R-rankings for the field on our site: until yesterday, the confidence intervals for programs almost all started at 1. That means that in our simulated rankings, almost all programs ended up in first place at least 25 times. Why?

The reasons are a bit technical: the regressions used to generate ranking weights for Communication Research programs had little predictive power, and the NRC's algorithm set all the weights to 0 in 39 iterations of their simulated rankings. The result was that all programs received the same quality score. How do you rank programs in the case of a tie? The NRC's methodology document does not say. We chose to report all programs as being tied for 1st place. They chose to report all programs as being tied for 41st (there were 83 programs). Both are reasonable choices, but ours leads to some strange side effects in rare cases like this; theirs has problems in other cases. We've since changed our algorithm and have dropped the degenerate rankings altogether.

The point is not to criticize the particular algorithmic choices the NRC made. Rather, it's that the NRC's methodology document, detailed as it may look, does not fully specify their algorithm. (This is not the only example, either). While it's valuable to have the sort of document the NRC produced that explains the algorithm and the rationale for the various components, without access to their source code, it's very difficult to reproduce what they did.

Increasingly science involves computation. If one want others to be able to reproduce what one has done in an experiment (a core feature of scientific research), one must describe not only the methods, but also the computational environment used, including the source code.