What is the proper procedure for reporting the significance of a signal in a simple counting experiment, when there is some uncertainty in the background underneath that signal?
Since the analysis is a simple counting experiment, the significance calculation is straightforward. All one has to do is calculate the probability for detecting at least as many events as were actually observed, given the background estimate. In principle this tail probability, also known as p value, is a Poisson probability, but a complication arises due to the uncertainty on the background. Often the background calculation is quite complex; a simplifying approximation is to consider the final background number and its uncertainty as the mean and width of a truncated Gaussian prior density in the Bayesian sense (“truncated”, because only positive background estimates are physically acceptable). One can then average the p value for the observation over all backgrounds, weighted by this truncated Gaussian prior. This is sometimes referred to as a “prior-predictive p value”. It is always useful to check the sensitivity of this calculation to the choice of prior, by using for example a g
