Design and Analysis of Single-Observation vs. Comparison-Group Experiments

AbstractThe purpose of this work is to alert researchers that meaningful comparisons may be possible for two-condition experiments that have a control group (CG) and only a single treated observation (SO). These designs involve the comparison of a single outcome measurement (denoted as YT) from a single treated case obtained at one point in time vs. , where is a group-based estimate of what YT would have been had no treatment been applied (i.e., an estimate of the unobserved counterfactual parameter YCF). The subscript on YT denotes the treatment condition and the hat on denotes an estimate of the true counterfactual value YCF. The contrast (YT -YCF) defines the treatment effect and (YT - ) is the general notation used to denote an estimate of the treatment effect. The proposed designs may consist of four phases: (1) nc+1 participants are randomly selected from the population of interest, (2) one subject is randomly selected from the sample of nc+1 participants, (3) the properties of the selected subject are evaluated relative to the properties of the group of nc participants (when available data allow), and (4) re-randomization is carried out to identify an alternative participant to be treated if diagnostics from the third phase suggest it. There are three variants of the proposed designs; the first variant is the least satisfactory, and it omits the third and fourth design phases listed above. The three variants differ with respect to the data required, the counterfactual estimator, the method of statistical analysis, statistical power, and credibility of results.Key Words: statistical inference, small-N design, scientific method, randomized experiment