Sample size during testing

Two questions frequently arise when data collection is being planned:

• How many users should I observe/ get data from? Is the effect I have found statistically reliable? 
• The use of power analysis enables the researcher to get a handle on both these questions. 

The size of the sample, the statistical power of the measuring instrument, and the size of the effect being investigated are three inter-linked issues. Estimating the Cost of Quantitative Evaluation is paper on the HFRG resources page which gives an overview of the concepts involved, and suggests definitions of various effect sizes in HCI: known as small, medium, and large effects.