ARMA Processes¶
\(ARMA(p,q)\) Process¶
Given white noise \(\{\varepsilon_t\}\), consider the process
\[\begin{split}Y_t & = c + \phi_1 Y_{t-1} + \ldots + \phi_p Y_{t-p} +
\varepsilon_t + \theta_1 \varepsilon_{t-1} + \ldots \theta_q
\varepsilon_{t-q},\end{split}\]
where \(c\), \(\{\phi_i\}_{i=1}^p\) and \(\{\theta_i\}_{i=1}^q\) are constants.
- This is an \(ARMA(p,q)\) process.
Expectation of \(ARMA(p,q)\)¶
Assume \(Y_t\) is weakly stationary.
\[\begin{split}E[Y_t] & = c + \phi_1 E[Y_{t-1}] + \ldots + \phi_p E[Y_{t-p}] \\
& \hspace{0.75in} + E[\varepsilon_t] + \theta_1 E[\varepsilon_{t-1}] + \ldots
+ \theta_q E[\varepsilon_{t-q}]\end{split}\]
\[= c + \phi_1 E[Y_t] + \ldots + \phi_p E[Y_t] \hspace{0.8in}\]
\[\begin{split}\Rightarrow E[Y_t] & = \frac{c}{1-\phi_1 - \ldots - \phi_p} =
\mu. \hspace{1.75in}\end{split}\]
- This is the same mean as an \(AR(p)\) process with parameters \(c\) and \(\{\phi_i\}_{i=1}^p\).
Autocovariances of \(ARMA(p,q)\)¶
Given that \(\mu = c/(1-\phi_1 - \ldots - \phi_p)\) for weakly stationary \(Y_t\):
\[\begin{split}Y_t & = \mu(1-\phi_1 - \ldots - \phi_p) + \phi_1
Y_{t-1} + \ldots + \phi_p Y_{t-p} \\
& \hspace{2in} + \varepsilon_t + \theta_1 \varepsilon_1 + \ldots \theta_q
\varepsilon_{t-q}\end{split}\]
\[\begin{split}\Rightarrow & (Y_t - \mu) = \phi_1(Y_{t-1} - \mu) +
\ldots + \phi_p(Y_{t-p} - \mu) \\
& \hspace{2in} + \varepsilon_t + \theta_1 \varepsilon_1 + \ldots \theta_q
\varepsilon_{t-q}.\end{split}\]
\[\gamma_j = E\left[(Y_t - \mu) (Y_{t-j} - \mu)\right] \hspace{2.7in}\]
\[\begin{split}& = \phi_1 E\left[(Y_{t-1} - \mu) (Y_{t-j} - \mu)\right] + \ldots \\
& \hspace{0.5in} + \phi_p E\left[(Y_{t-p} - \mu) (Y_{t-j} -
\mu)\right] \\
& \hspace{1in} + E\left[\varepsilon_t (Y_{t-j} - \mu)\right] +
\theta_1 E\left[\varepsilon_{t-1} (Y_{t-j} - \mu)\right] \\
& \hspace{2.25in} + \ldots +
\theta_q E\left[\varepsilon_{t-q} (Y_{t-j} - \mu)\right]\end{split}\]
Autocovariances of \(ARMA(p,q)\)¶
- For \(j > q\), \(\gamma_j\) will follow the same law of motion as for an \(AR(p)\) process:
\[\begin{split}\gamma_j & = \phi_1 \gamma_{j-1} + \ldots + \phi_p \gamma_{j-p}
\,\,\,\,\, \text{ for } j = q+1, \ldots\end{split}\]
- These values will not be the same as the \(AR(p)\) values for \(j = q+1, \ldots\), since the initial \(\gamma_0, \ldots, \gamma_q\) will differ.
- The first \(q\) autocovariances, \(\gamma_0, \ldots, \gamma_q\), of an \(ARMA(p,q)\) will be more complicated than those of an \(AR(p)\).
Estimating \(ARMA\) Models¶
Estimation of an \(ARMA\) model is done via maximum likelihood.
- For an \(ARMA(p,q)\) model, one would first specify a joint likelihood for the parameters \(\{\phi_1, \ldots, \phi_p, \theta_1, \ldots, \theta_q, \mu, \sigma^2\}\).
- Taking derivatives of the log likelihood with respect to each of the parameters would result in a system of equations that could be solved for the MLEs: \(\{\hat{\phi}_1, \ldots, \hat{\phi}_p, \hat{\theta}_1, \ldots, \hat{\theta}_q, \hat{\mu}, \hat{\sigma}^2\}\).
- The exact likelihood is cumbersome and maximization requires specialize numerical methods.
- The MLEs can be obtained with the \(\mathtt{arima}\) function in \(\mathtt{R}\).
Which \(ARMA\)?¶
How do we know if an \(ARMA\) model is appropriate and which \(ARMA\) model to fit?
- After fitting an \(ARMA\) model, we can examine the residuals.
- The \(\mathtt{acf}\) function in \(\mathtt{R}\) can be used to compute empirical autocorrelations of the residuals.
- If the residuals are autocorrelated, the model is not a good fit. Consider changing the parameters \(p\) and \(q\) of the \(ARMA\) or using another model.
Which \(ARMA\)?¶
The \(\mathtt{auto.arima}\) function in \(\mathtt{R}\) estimates a range of \(ARMA(p,q)\) models and selects the one with the best fit.
- \(\mathtt{auto.arima}\) uses the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to select the model.
- Minimizing AIC and BIC amounts to minimizing the sum of squared residuals, with a penalty term that is related to the number of model parameters.