Practice Maths

← Statistical Inference › Type I and Type II Errors

Type I and Type II Errors

Key Terms

A Type I error occurs when we reject H0 when H0 is actually true (false positive)
A Type II error occurs when we fail to reject H0 when H0 is actually false (false negative)
The significance level α = P(Type I error) = P(reject H0 | H0 true)
β = P(Type II error) = P(fail to reject H0 | H0 false)
The power of a test = 1 − β = P(correctly reject H0 | H0 false)
Decreasing α increases β (and decreases power), for a fixed sample size
Decision table:

H0 is true H0 is false
Reject H0 Type I error (α) Correct (Power = 1−β)
Fail to reject H0 Correct (1−α) Type II error (β)

P(Type I error) = α (the significance level)
Power = 1 − β = P(reject H0 | H1 true)
Worked Example: A machine fills bottles and is set to dispense μ = 500 mL. A quality inspector tests H0: μ = 500 vs H1: μ ≠ 500 at α = 0.05. The machine is actually underfilling (μ = 494 mL). Describe any error that could be made.

Since H0 is false (the machine is underfilling), the inspector could make a Type II error: failing to reject H0 when the machine really is faulty. The inspector would conclude the machine is fine when it isn’t. The probability of this error is β.
Memory Aid: Type I = “cry wolf” (alarm when nothing is wrong). Type II = “miss the wolf” (no alarm when something is wrong). The significance level α directly controls Type I errors.

Understanding Errors in Hypothesis Testing

Every hypothesis test has the possibility of making a wrong decision. There are exactly two types of errors, and understanding them is essential for evaluating the reliability of statistical conclusions.

Type I Error

A Type I error is rejecting the null hypothesis H0 when H0 is actually true. This is a “false alarm” — we conclude there is an effect or difference when there really isn’t one.

Example: A pharmaceutical company tests H0: a new drug has no effect vs H1: the drug has an effect. A Type I error means concluding the drug works when it actually doesn’t. This could lead to an ineffective drug being approved.

The probability of a Type I error is equal to the significance level: P(Type I) = α. By choosing α = 0.05, we accept a 5% chance of falsely rejecting a true null hypothesis.

Type II Error

A Type II error is failing to reject H0 when H0 is actually false. This is a “missed detection” — we conclude there is no effect when there really is one.

Example: Testing whether a manufacturing process is faulty. A Type II error means concluding the process is fine when it is actually producing defective items. This could have safety consequences.

The probability of a Type II error is denoted β. Calculating β requires knowing the actual value of the parameter under H1.

Power of a Test

The power of a hypothesis test is the probability of correctly rejecting H0 when it is false:

Power = 1 − β = P(reject H0 | H0 is false)

A high-powered test (close to 1) is good at detecting a real effect. Power increases when:

  • Sample size n increases
  • Significance level α increases
  • The true effect size is larger
  • Population variance is smaller

The Trade-off Between α and β

For a fixed sample size, reducing α (making it harder to reject H0) increases β (making it more likely to miss a real effect), and vice versa. The only way to reduce both simultaneously is to increase the sample size.

In practice, the choice of α reflects how serious each type of error is:

  • If Type I errors are very costly (e.g., approving a dangerous drug), use a small α (0.01 or 0.001)
  • If Type II errors are very costly (e.g., missing a disease), maximise power (increase n or use larger α)
Strategy: When a question asks you to describe a Type I or Type II error in context, always phrase it in terms of the specific decision made (reject or not reject H0) and the actual truth of the situation.

Mastery Practice

See Answers ➔
  1. Fluency State the definition of a Type I error and give the probability of it occurring in terms of α.
  2. Fluency State the definition of a Type II error and give its probability in terms of β.
  3. Fluency A test is conducted at α = 0.05. What is the probability of making a Type I error? What is the probability of correctly not rejecting a true H0?
  4. Fluency Complete the decision table: copy and fill in “Type I error”, “Type II error”, “Correct decision” for the four possible outcomes of a hypothesis test.
  5. Understanding A medical test screens for a disease. H0: patient does not have the disease, H1: patient has the disease. Describe, in context, what a Type I error and a Type II error would mean. Which error has more serious consequences? Justify.
  6. Understanding A hypothesis test is conducted at α = 0.05. If H0 is actually false, and the test has power 0.82, find β and interpret it in context.
  7. Understanding A researcher reduces α from 0.05 to 0.01, keeping the sample size fixed. Describe how this affects: (a) the probability of a Type I error, (b) the probability of a Type II error, (c) the power of the test.
  8. Understanding A factory tests H0: μ = 50 g (correct fill) vs H1: μ ≠ 50 g. The significance level is 0.05. In 200 tests on correctly-filling machines, how many would you expect to result in a Type I error?
  9. Problem Solving A quality control test uses a sample of n = 25 items from a production line. The inspector tests H0: μ = 100 vs H1: μ < 100, where X ~ N(100, 36) under H0. The test rejects H0 when &x̄ < 97.
    • (a) Find α = P(Type I error) = P(&x̄ < 97 | μ = 100).
    • (b) Suppose the true mean is μ = 96. Find β = P(Type II error) = P(&x̄ ≥ 97 | μ = 96).
    • (c) Find the power of the test when μ = 96.
  10. Problem Solving In a clinical trial, researchers test H0: a new treatment has no effect at α = 0.01. The study has a power of 0.70 to detect a meaningful effect.
    • (a) What is the probability of concluding the treatment works when it actually doesn’t?
    • (b) What is the probability of failing to detect a real effect?
    • (c) A second study doubles the sample size. Without calculating, explain how this affects the power and β.
    • (d) Why might a researcher prefer α = 0.01 over α = 0.05 in a medical context?

View Full Worked Solutions →