Many researchers rely on grants to fund their work. Typically, they write a proposal for a future project. Proposals are scored and ranked by a panel, and the top-ranking proposals are funded. The percentage of proposals that receive funding is referred to as the "payline," and has fallen dramatically over time. Recent papers in mBio and PLOS Biology have called for a change to this process. They propose using a lottery, rather than a ranking system, to select proposals for funding. Although this idea may seem strange, they argue that the existing system The idea of using lotteries is gaining steam: it was discussed in Vox, and has been adopted by several funding agencies. In this post, we will explore whether a lottery might actually be better than a ranking system. Although I am sympathetic to all four concerns above,1 I will focus on the last, as it benefits most from a model. Let's sketch a simple model. There are \(n\) researchers. Each researcher \(i\) has an idea of quality \(q_i\), and chooses the fraction of time \(t_i \in [0,1]\) to spend writing a grant proposal. Once all proposals have been received, they are scored and ranked: the payline \(p\) determines what fraction receive funding. The score of a proposal depends on both the quality of the underlying idea and the time spent on the proposal. For simplicity, let's assume that the score of proposal \(i\) is \(t_i q_i\). If researcher \(i\) gets funded, they spend their remaining time conducting research. This advances science by \((1-t_i)q_i\), and gives this same benefit to the researcher. Researchers whose proposals are not funded get a payoff of zero.2 Let's suppose that we have 8 ideas, with qualities \(q_1 = 10, q_2 = 8, q_3 = 6, q_4 = q_5 = q_6 = 2, q_7 = q_8 = 1\), and that the payline is \(p = 3/8\), so three proposals will be funded. Researchers 1, 2, and 3 can assure themselves of funding by writing just enough to earn scores of two. Researcher 1 spends 20% of her time writing grants, and with the other 80% conducts research that advances science by 8. Researcher 2 spends 25% of her time writing grants, and advances science by 6. Researcher 3 spends 33% of her time writing grants, and advances science by 4. In total, science advances by 18. If we were to use a lottery instead, researchers would have no incentive to polish their proposals, and would have more time for research. However, most of the funds would be directed towards bad ideas. On average, science would advance by only 12. Suppose that the payline falls to \(p = 1/8\), so that only one idea will be funded. We have the following result: Proposition For any fixed qualities \(q\), as the payline \(p\) decreases, winning researchers spend more time writing grant proposals. Now Researcher 1 must spend mosst of her time (80%) polishing her grant proposal to ensure that it beats out the proposal from Researcher 2. This leaves her only a 20% of her time for science, which advances by 2. Had we awarded the grant by lottery, then one randomly chosen researcher could spend all of her time pursuing her idea. On average, science would advance by 4. This model illustrates that there is a tradeoff between these approaches. Ranking proposals ensures that the best ideas are funded, but forces researchers to waste time writing proposals. Using a lottery allows researchers to dedicate themselves to research, but may fund bad ideas. Ranking works well when the number of good ideas matches the number of proposals that can be funded: it identifies proposals that deserve funding, without requiring researchers to spend too much time writing. However, if there are more good ideas than funding, researchers with good ideas must spend most of their time competing with each other for funding. A lottery spares this effort, resulting in greater scientific progress. Can we improve on the lottery in the second scenario? Yes! One concern is that many ideas are of low quality. We can filter these out by deciding that only proposals with a score above two qualify for the lottery. Each of the first three researchers will now have to spend some time writing, but just enough to ensure that they qualify. One of them will get funded, and science advances by an average of 6. This last idea is essentially what is being proposed: have panels classify proposals as meritorious or non-meritorious, and hold a lottery within the former group. The examples above illustrate how a lottery could outperform a ranking-based method, but also reveal that the reverse could be true. This isn't very helpful for policy guidance. Thankfully, the model above is equivalent to that studied by Hartline and Roughgarden.3 They find that a ranking outperforms the lottery if there are a few outlier ideas, which have far higher quality than the rest.4 In such cases, it will be relatively easy for the researchers behind these ideas to convince the panel that they deserve funding. Conversely, if there are many ideas of similar quality, then the ranking approach forces researchers to compete fiercely with each other, leaving little time for the actual research. Furthermore, this problem is exacerbated as the payline falls. The model above illustrates that as funding becomes more scarce, researchers spend ever more time chasing it. My gut says that grant panelists are often splitting hairs between similar proposals. If that is the case, then the analysis above suggests that moving to a lottery could dramatically reduce time spent writing grants, and actually increase the rate of scientific advances. To prevent funding truly bad proposals, the panel could start by throwing these out. Furthermore, if there are a few ideas that really stand out, we could ensure that they receive funding by having panelists sort proposals into three categories: definitely in, definitely out, and lottery. Although I don't have much experience with grants, I do have a fair amount of experience with refereed conferences. These conferences are generally seeing an increase in the number of submissions. While there are always some easy accepts and easy rejects, many submissions lie in the gray zone, and their fate can come down to the taste and mood of the reviewers. Perhaps a three-tiered lottery would lead to a fairer and less time-consuming review process!5 So readers, what do you think? Should grant funding (and conference acceptances) be partially dictated by lottery? Admittedly, these concerns undermine each other to some extent. For example, if the existing system is essentially a lottery, then why would researchers spend so much time polishing their proposals?↩ This model is of course extremely stylized. It assumes that researchers know exactly how their proposals will be scored, and that time spent writing the proposal does not improve the quality of the idea. Furthermore, it ignores the fact that researchers with good ideas may nevertheless be bad writers, and assumes that each researcher's utility is exactly their contribution to scientific progress. Despite these limitations, it has several interesting implications, which we explore below. The PLOS biology paper incorporates some of these complexities, although it does not provide a comprehensive analysis.↩ In fact, there is a large literature on contests, which I will not cite here. Matt Weinberg and I have a paper that uses this class of models to study bitcoin mining.↩ In a desire to maintain a conversational tone, I have abandoned mathematical precision. Formally, their results depend on whether the distribution of qualities has a heavy tail (decreasing hazard rate) or a light one (increasing hazard rate).↩ The model above suggests that it might also cause researchers to invest less time into writing. Anyone who has served as a program committee member will probably agree with me that this would not be desirable.↩Motivation
Model
High Payline
Low Payline
Lessons from This Example
A General Analysis
Conclusions