The \(t\)-statistic is \(t\)-distributed
Abstract. We prove that the \(t\)-statistic of an OLS coefficient follows a Student \(t\)-distribution with \(n-p\) degrees of freedom, under the assumption of Gaussian errors. The proof uses three tools: the sampling distribution of \(\widehat{\beta}\), Cochran's theorem applied to the residual quadratic form, and the independence of \(\widehat{\beta}\) and \(\widehat{\sigma}^2\).
1 Introduction
Richard showed up to Ethan's NBA Finals watch party with a notebook and pen in order to continue working on some proofs he wanted to write for work. I was inspired because I haven't written proofs since school, so I thought I'd work on what he was working on Friday during slow work hours. It turns out, I misremembered, so this is a completely different proof to what he had.
When we fit a linear model and read off a coefficient \(\widehat{\beta}_j\), we want to know whether the estimate is distinguishable from a hypothesized value \(\beta_j\) given the noise in the data. The \(t\)-statistic answers this by standardizing the estimation error \(\widehat{\beta}_j - \beta_j\) by its estimated standard deviation. The price of estimating \(\sigma^2\) rather than knowing it is that the resulting ratio is not normal but \(t\)-distributed, with heavier tails.
The proof below generally follows dependency order. We first derive the sampling distribution of \(\widehat{\beta}\) (Section 3.1), which gives a standard normal numerator. We then take the residual-maker matrix \(M\) (Section 3.2) and use Cochran's theorem (Section 3.3) to show that the scaled residual sum of squares, is \(\chi^2_{n-p}\) (Section 3.4). After establishing that the numerator and denominator are independent (Section 3.5), the main result follows directly from the characterization of the \(t\)-distribution (Section 4).
2 Setup
2.1 The model and OLS estimator
Given a regression problem \(y = X\beta + \epsilon\) with unknown parameters \(\beta\) and error \(\epsilon \mid X \sim \mathcal{N}(0, \sigma^2 I)\), the ordinary least squares solution is \(\widehat{\beta} = (X^T X)^{-1} X^T y\). Here \(X\) is the \(n \times p\) design matrix and \(\beta\) is a \(p \times 1\) vector. All distributional statements below are conditional on \(X\).
2.2 The \(t\)-statistic
Definition 1. The \(t\)-statistic for coefficient \(\beta_j\) is defined as $$t_j = \frac{\widehat{\beta}_j - \beta_j}{\operatorname{S.E.}(\widehat{\beta}_j)} = \frac{\widehat{\beta}_j - \beta_j}{\sqrt{\widehat{\sigma}^2 \left[(X^T X)^{-1}\right]_{jj}}}$$
2.3 The error assumptions
For \(n\) observations, the error terms \(\epsilon\) are an \(n \times 1\) vector satisfying the conditions \(\mathbb{E}(\epsilon) = 0\) and \(\operatorname{Var}(\epsilon) = \sigma^2 I_n\), where \(\sigma^2\) is the error variance.
Our assumption is that a fitted model has uncorrelated and constant variance across all observations.
Standard error is the estimated standard deviation of a statistic. Since \(\sigma^2\) is unknown, we must estimate it from the residuals; the estimator \(\widehat{\sigma}^2\) is constructed in Section 3.4.
3 Building blocks
3.1 Sampling distribution of \(\widehat{\beta}\)
To find the standard error, we first compute \(\operatorname{Var}(\widehat{\beta} \mid X)\) and substitute the population variance \(\sigma^2\) for the sample variance \(\widehat{\sigma}^2\).
Lemma 2. \(\widehat{\beta} \,|\, X \sim \mathcal{N}(\beta, \sigma^2 (X^T X)^{-1})\)
Proof. Substituting \(y = X\beta + \epsilon\), $$\widehat{\beta} = (X^T X)^{-1} X^T (X\beta + \epsilon) = \beta + (X^T X)^{-1} X^T \epsilon.$$ So \(\widehat{\beta}\) is an affine transformation of the Gaussian vector \(\epsilon\) and is therefore Gaussian. Its mean is $$\mathbb{E}(\widehat{\beta} \mid X) = \beta + (X^T X)^{-1} X^T \, \mathbb{E}(\epsilon \mid X) = \beta.$$ For the variance, let \(A = (X^T X)^{-1} X^T\) and recall \(\operatorname{Var}(\epsilon \mid X) = \sigma^2 I\). Then, \begin{align*} \operatorname{Var}(\widehat{\beta} \mid X) &= \operatorname{Var}(A y \mid X) \\ &= A \operatorname{Var}(y \mid X) A^T \\ &= (X^T X)^{-1} X^T (\sigma^2 I) \big[ (X^T X)^{-1} X^T\big]^T \\ &= (X^T X)^{-1} X^T (\sigma^2 I) X (X^T X)^{-1} \\ &= \sigma^2 (X^T X)^{-1} \end{align*}
Thus, we have one building block for the \(t\)-distribution: for each coordinate \(j\),
$$ \widehat{\beta}_j - \beta_j \sim \mathcal{N}\big(0, \, \sigma^2 \left[(X^T X)^{-1}\right]_{jj}\big) \implies \frac{\widehat{\beta}_j - \beta_j}{\sqrt{\sigma^2 \left[(X^T X)^{-1}\right]_{jj}}} \sim \mathcal{N}(0, 1) $$
And, the standard error
$$\operatorname{S.E.}(\widehat{\beta}_j) = \sqrt{\widehat{\sigma}^2 \left[(X^T X)^{-1}\right]_{jj}}.$$
In order to develop a definition for \(\widehat{\sigma}^2\), we need to first define the residuals \(\widehat{\epsilon}\) or \(e\).
\begin{align*} e &= y - X\widehat{\beta} \\ &= y - X (X^T X)^{-1} X^T y \\ &= (I - X (X^T X)^{-1} X^T) y \end{align*}
3.2 The residual-maker matrix or annihilator matrix
Since we are estimating the linear model with OLS, our fitted values are $$\widehat{y} = X \widehat{\beta} = X (X^T X)^{-1} X^T y = \mathbf{P} y$$ Thus, our projection matrix is $$\mathbf{P} = X(X^T X)^{-1} X^T.$$
Lemma 3. \(M = I - \mathbf{P} = I - X(X^T X)^{-1} X^T\) is symmetric, idempotent, has rank \(n-p\), satisfies \(MX= 0\), and \(e = M\epsilon\). \(M\) is known as the residual-maker, or the annihilator matrix.
Proof. Symmetry. Since \((X^T X)^{-1}\) is symmetric, $$M^T = I - \big[X (X^T X)^{-1} X^T\big]^T = I - X (X^T X)^{-1} X^T = M.$$
Idempotency. First, \(\mathbf{P}\) is idempotent: $$\mathbf{P}^2 = X (X^T X)^{-1} \underbrace{X^T X (X^T X)^{-1}}_{= \, I} X^T = X (X^T X)^{-1} X^T = \mathbf{P}.$$ Therefore $$M^2 = (I - \mathbf{P})^2 = I - 2\mathbf{P} + \mathbf{P}^2 = I - \mathbf{P} = M.$$
Annihilation. $$MX = X - X (X^T X)^{-1} X^T X = X - X = 0.$$
Rank. For an idempotent matrix, rank equals trace. Using the cyclic property of the trace, $$\operatorname{rank}(M) = \operatorname{tr}(M) = \operatorname{tr}(I_n) - \operatorname{tr}\big(X (X^T X)^{-1} X^T\big) = n - \operatorname{tr}\big((X^T X)^{-1} X^T X\big) = n - \operatorname{tr}(I_p) = n - p.$$
Residuals. Substituting \(y = X\beta + \epsilon\) and using \(MX = 0\), $$e = My = M(X\beta + \epsilon) = MX\beta + M\epsilon = M\epsilon.$$
3.3 Cochran's theorem
Theorem 4 (Cochran's Theorem). Let \(z \sim \mathcal{N}(0, I_n)\) and suppose $$z^T z = \sum^m_{j=1} Q_j, \qquad Q_j = z^T A_j z,$$ where each \(A_j\) is symmetric and \(\sum^m_{j=1} \operatorname{rank}(A_j) = n\). Then the \(Q_j\) are mutually independent and \(Q_j \sim \chi^2_{\operatorname{rank}(A_j)}\) for each \(j\).
Proof. Sorry, we omit the proof, so see Keener, Theoretical Statistics (2010).
In our application we take \(z^T z = z^T \mathbf{P} z + z^T (I - \mathbf{P}) z\), with \(\operatorname{rank}(\mathbf{P}) + \operatorname{rank}(I - \mathbf{P}) = p + (n - p) = n\).
3.4 Distribution of \(\widehat{\sigma}^2\)
Definition 5 (Residual sum of squares). \(\mathrm{RSS} = e^T e = \left\| (I - \mathbf{P}) y \right\|^2\).
Definition 6 (Variance estimator). \(\displaystyle \widehat{\sigma}^2 = \frac{\mathrm{RSS}}{n - p}\).
We express \(\mathrm{RSS}\) as a quadratic form in the errors \(\epsilon\).
Lemma 7. \(\mathrm{RSS} = \epsilon^T (I - \mathbf{P})\, \epsilon\).
Proof. By Lemma 3, \(e = M\epsilon = (I - \mathbf{P})\,\epsilon\). Since \(M\) is symmetric (\(M^T = M\)) and idempotent (\(M^2 = M\)), \begin{align*} \mathrm{RSS} &= e^T e \\ &= (M\epsilon)^T (M\epsilon) \\ &= \epsilon^T M^T M \epsilon \\ &= \epsilon^T M^2 \epsilon \\ &= \epsilon^T M \epsilon \end{align*}
Proposition 8. \(\displaystyle \frac{\mathrm{RSS}}{\sigma^2} \sim \chi^2_{n-p}\).
Proof. Because Cochran's theorem applies to standard normal vectors, we scale \(\epsilon \sim \mathcal{N}(0, \sigma^2 I)\) by \(\epsilon / \sigma = z \sim \mathcal{N}(0, I_n)\). With Lemma 7, $$ z^T (I - \mathbf{P})\, z = \frac{\epsilon^T (I - \mathbf{P})\, \epsilon}{\sigma^2} = \frac{\mathrm{RSS}}{\sigma^2}. $$ By Lemma 3, \(\operatorname{rank}(I - \mathbf{P}) = n - p\), so Cochran's theorem gives us that the quadratic form \(z^T (I - \mathbf{P})\, z\) follows a \(\chi^2\) distribution with \(n - p\) degrees of freedom.
Corollary 9. \(\displaystyle \frac{(n-p)\,\widehat{\sigma}^2}{\sigma^2} \sim \chi^2_{n-p}\).
Proof. Follows from Proposition 8, since \((n-p)\,\widehat{\sigma}^2 = \mathrm{RSS}\).
Corollary 10 (Unbiasedness). \(\mathbb{E}\big(\widehat{\sigma}^2\big) = \sigma^2\).
Proof. A \(\chi^2_{n-p}\) random variable has mean \(n-p\). Taking expectations in Corollary 9, \(\mathbb{E}\big((n-p)\,\widehat{\sigma}^2/\sigma^2\big) = n-p\), and rearranging gives \(\mathbb{E}\big(\widehat{\sigma}^2\big) = \sigma^2\).
3.5 Independence of \(\widehat{\beta}\) and \(\widehat{\sigma}^2\)
The last building block is that the numerator and denominator of the \(t\)-statistic do not share information.
Lemma 11. Conditional on \(X\), \(\widehat{\beta}\) and \(\widehat{\sigma}^2\) are independent.
Proof. From the proof of Lemma 2, \(\widehat{\beta} - \beta = A\epsilon\) with \(A = (X^T X)^{-1} X^T\), and from Lemma 3, \(e = M\epsilon\). Both are linear in the same Gaussian vector \(\epsilon\), so \((\widehat{\beta} - \beta, \, e)\) is jointly Gaussian. Their cross-covariance is \begin{align*} \operatorname{Cov}(A\epsilon, M\epsilon) &= A \operatorname{Var}(\epsilon) M^T \\ &= \sigma^2 A M \\ &= \sigma^2 (X^T X)^{-1} X^T M \\ &= \sigma^2 (X^T X)^{-1} (M X)^T \\ &= 0 \end{align*} using symmetry of \(M\) and \(MX = 0\). Uncorrelated jointly Gaussian vectors are independent, so \(\widehat{\beta}\) is independent of \(e\). Since \(\widehat{\sigma}^2 = e^T e / (n-p)\) is a function of \(e\) alone, \(\widehat{\beta}\) is independent of \(\widehat{\sigma}^2\).
4 The Student \(t\)-distribution
Definition 12. Let \(Z \sim \mathcal{N}(0,1)\) and \(W \sim \chi^2_m\) be independent. Then $$t = \frac{Z}{\sqrt{W / m}}$$ follows the Student \(t\)-distribution with \(m\) degrees of freedom.
5 Main result
Theorem 13. Under the model of Section 2, \(t_j \sim t_{n-p}\).
Proof. Write \(c_{jj} = \left[(X^T X)^{-1}\right]_{jj}\). Dividing the numerator and denominator of \(t_j\) by \(\sqrt{\sigma^2 c_{jj}}\), \begin{align*} t_j &= \frac{\widehat{\beta}_j - \beta_j}{\sqrt{\widehat{\sigma}^2 \, c_{jj}}} \\ &= \frac{\big(\widehat{\beta}_j - \beta_j\big) \big/ \sqrt{\sigma^2 c_{jj}}}{\sqrt{\widehat{\sigma}^2 / \sigma^2}} \\ &= \frac{Z}{\sqrt{W / (n-p)}}, \end{align*} where $$Z = \frac{\widehat{\beta}_j - \beta_j}{\sqrt{\sigma^2 c_{jj}}} \sim \mathcal{N}(0,1) \qquad \text{and} \qquad W = \frac{(n-p)\,\widehat{\sigma}^2}{\sigma^2} \sim \chi^2_{n-p}$$ by Lemma 2 and Corollary 9 respectively. See that the unknown \(\sigma^2\) cancels in the ratio. By Lemma 11, \(Z\) (a function of \(\widehat{\beta}\)) and \(W\) (a function of \(\widehat{\sigma}^2\)) are independent. By Definition 12, \(t_j \sim t_{n-p}\).
6 Remarks
Remark 14. Statistical libraries test the null hypothesis \(H_0 \colon \beta_j = 0\), where the numerator then reduces to \(\widehat{\beta}_j\) and the reported statistic is \(t_j = \widehat{\beta}_j / \operatorname{S.E.}(\widehat{\beta}_j)\). Thus, a large \(|t_j|\) relative to the \(t_{n-p}\) distribution indicates that the coefficient is statistically significant.