Examples may be a specific p-value or Bayes Factor in a hypothesis testing framework, or a confidence interval or highest density interval around a parameter estimate, from a ‘new statistics’ or parameter estimation framework. Your stopping criteria will depend on what statistical approach you are taking. These points may seem obvious, but we highlight them because we have found it is very easy to under-appreciate the chances of continued data collection, and underestimate the difficulties it may pose - especially at the start of a project when you are very excited about the possible findings. Of course, you can always opt to stop sooner than planned if circumstances change, but there is little use in considering a maximum size that you can anticipate not being able to test. You and your team should carefully consider what maximum sample size you are willing and able to support, and try to ensure that the maximum sample size would not be an excessive burden. In a similar vein, continuing to the maximum sample size is really a possibility, as estimates of power are difficult to make with high reliability.
We suggest some software below that can help guide you with this choice.įrom a practical perspective, it is good to keep in mind that reaching your stopping criterion at the minimum sample size is really a best-case scenario: you need to be prepared for continued data collection even if you hope you won’t need to! Assuming and planning for stopping at the first data check is likely to result in disappointment. On the other hand, setting your minimum sample size very high may result in wasted time and resources. If your minimum sample size is very small, you may raise the false positive rate (extreme values will have a large influence in a small sample size). When choosing a minimum or maximum sample size, as well as your batch size, a number of statistical and practical aspects should be considered.
Choosing your sequential testing parameters