Two-Sample Binomial Score Confidence Intervals in Small Samples: Evaluating Adjusted versus Unadjusted Score Statistics

Exact inference in small samples is difficult in 2×2 contingency tables due to discreteness, nuisance parameters, and the mean-variance relationship. Although large-sample theory has been well-developed, when “large” samples are not easily obtainable due to feasibility we still want to answer the scientific question using suitable small-sample methods. Typical approaches in small samples include using Fisher’s exact test or Yates’ correction; however, these tests are typically conservative due to discreteness. Another approach is to invert tests to obtain confidence intervals (CI). However, constructing CI based on the Wald test requires estimation of nuisance parameters to handle the mean-variance relationship and may be anti-conservative due to sample size. Alternatively, Mehta and Senchaudhuri use Barnard’s exact test for small samples because it “must maximize over all possible p-values [...] resulting in greater power” compared to Fisher’s [Mehta CR, Senchaudhuri P (2003). Ctyel Software Corporation Technical Paper]. We consider the two-sided hypothesis test for the difference in Binomial proportions. Our goal is to assess how inverting the score statistic along with Barnard’s unconditional adjustment compares to the unadjusted version by considering two key criteria in constructing CI—properties of coverage and expected length. Our adjustment method relies on finding the α*-level that provides better coverage than that obtained by the unadjusted score statistic relative to the pre-specified size α; lack of precision found from CI expected lengths is an acceptable tradeoff. We also compare our method to inversion of other tests, including Wald and likelihood ratio, which do not have ideal coverage.