Bayes Theorem¶

Sample Space¶

Consider a random variable \(X\).

The set of all possible outcomes of \(X\) is referred to as the sample space.

We will denote the sample space as \(\mathcal{S}\).

Each outcome, \(x\), of the random variable \(X\) is called a member of \(\mathcal{S}\).

In notation \(x \in \mathcal{S}\).

Subsets¶

A subset of \(\mathcal{S}\) is a collection of outcomes.

If \(A\) is a subset of \(\mathcal{S}\), we write \(A \subset \mathcal{S}\).

If \(B\) is a subset of \(\mathcal{S}\) and \(A\) is a subset of \(B\), we write \(A \subset B\).

We also say that \(A\) is contained in \(B\).

We often refer to subsets of the sample space as events.

The empty set is the subset with no elements and is denoted \(\emptyset\).

The empty set is an impossible event.

Example of Subsets¶

Let \(X\) be the result of a fair die roll.

The sample space is \(\{1,2,3,4,5,6\}\).

Let \(B\) be the event that \(X\) is even: \(B = \{2,4,6\}\).

Let \(A\) be the event that \(X\) is 2 or 4: \(A = \{2,4\}\).

Clearly, \(A \subset B \subset \mathcal{S}\).

Let \(C\) be the event that \(X\) is \(-1\).

Clearly, \(C = \emptyset\).

Union¶

The union of two sets is the set containing all outcomes that belong to \(A\) or \(B\).

We write the union of \(A\) and \(B\) as \(A \cup B\).

For the die roll example, if \(A = \{2,5\}\) and \(B = \{2,4,6\}\), then \(A \cup B = \{2,4,5,6\}\).

The union of many sets is written as

\[\bigcup_{i=1}^N A_i.\]

Union¶

\(\qquad\)

Union¶

\(\qquad\)

Union¶

\(\qquad\)

Intersection¶

The intersection of two sets is the set containing all outcomes that belong to \(A\) and \(B\).

We write the intersection of \(A\) and \(B\) as \(A \cap B\).

For the die roll example, if \(A = \{2,5\}\) and \(B = \{2,4,6\}\), then \(A \cap B = \{2\}\).

The intersection of many sets is written as

\[\bigcap_{i=1}^N A_i.\]

Intersection¶

\(\qquad\)

Intersection¶

\(\qquad\)

Intersection¶

\(\qquad\)

Complements¶

The complement of \(A \subset \mathcal{S}\) is the subset that contains all outcomes in \(\mathcal{S}\) that are not in \(A\).

We denote the complement by \(A^c\).

For the die roll example, if \(A = \{2,4,6\}\), then \(A^c = \{1,3,5\}\).

Complements¶

\(\qquad\)

Complements¶

\(\qquad\)

Complements¶

Some important properties:

\[(A^c)^c = A\]

\[\emptyset^c = \mathcal{S}\]

\[\mathcal{S^c} = \emptyset\]

\[A \cup A^c = \mathcal{S}\]

\[A \cap A^c = \emptyset.\]

Disjoint Events¶

Two events are disjoint or mutually exclusive if they have no outcomes in common.

\(A\) and \(B\) are disjoint if \(A \cap B = \emptyset\).

By definition, any event and its complement are disjoint: \(A \cap A^c = \emptyset\).

For the die roll example, if \(A = \{2,5\}\) and \(B = \{4,6\}\), then \(A \cap B = \emptyset\).

Probability¶

Given \(A \subset \mathcal{S}\), denote the probability \(P(X \subset A) = P(A)\).

In the Venn diagram, \(P(A)\) is the ratio of the area of \(A\) to the area of \(\mathcal{S}\).

Note that \(P(\mathcal{S}) = 1\).

Note that \(P(\emptyset) = 0\).

Probability of Intersections¶

The probability that \(X\) is in \(A\) and \(B\) is:

\[\begin{split}P(A \cap B) & = P((X \subset A) \cap (X \subset B)).\end{split}\]

For the die roll example, if \(A = \{2,5\}\) and \(B = \{2,4,6\}\), then

\[P(A \cap B) = P(X = 2) = \frac{1}{6}.\]

Probability of Unions¶

The probability that \(X\) is in \(A\) or \(B\) is:

\[P(A \cup B) = P((X \subset A) \cup (X \subset B)) \hspace{0.35in}\]

\[\hspace{0.5in} = P(A) + P(B) - P(A \cap B).\]

Probability of Unions¶

For the die roll example, if \(A = \{2,5\}\) and \(B = \{2,4,6\}\),

\[P(A) + P(B) - P(A \cap B) \hspace{1.5in}\]

\[\hspace{0.47in} = P\left(\{2,5\}\right) + P\left(\{2,4,6\}\right) - P\left(\{2\}\right)\]

\[= \frac{2}{6} + \frac{3}{6} - \frac{1}{6} \hspace{1.15in}\]

\[= \frac{4}{6} \hspace{1.85in}\]

\[= P\left(\{2,4,5,6\}\right) \hspace{0.95in}\]

\[= P(A \cup B). \hspace{1.3in}\]

Probability of Unions¶

If \(A \cap B = \emptyset\),

\[P(A \cup B) = P(A) + P(B),\]

since \(P(\emptyset) = 0\).

Probability of Unions¶

For the die roll example, if \(A = \{2,5\}\) and \(B = \{4,6\}\),

\[P(A) + P(B) - P(A \cap B) \hspace{1.5in}\]

\[\hspace{0.47in} = P\left(\{2,5\}\right) + P\left(\{4,6\}\right) - P\left(\emptyset\right)\]

\[= \frac{2}{6} + \frac{2}{6} - 0 \hspace{0.88in}\]

\[= \frac{4}{6} \hspace{1.53in}\]

\[= P\left(\{2,4,5,6\}\right) \hspace{0.63in}\]

\[= P(A \cup B). \hspace{0.97in}\]

Conditional Probability¶

Suppose we know that event \(B\) has occurred - that is, one of the outcomes in the subset \(B \subset \mathcal{S}\) has occurred.

How does this alter our view of the probability of event \(A\) occurring?

Denote the probability of \(A\), conditional on \(B\) occurring, as \(P(A|B)\).

If \(A \cap B = \emptyset\), we know \(P(A|B) = 0\). Why?

Conditional Probability¶

If \(A \cap B \neq \emptyset\)

\[\begin{split}P(A|B) & = \frac{P(A \cap B)}{P(B)}.\end{split}\]

\(P(A|B)\) is the ratio of the area of \(A \cap B\) to the area of \(B\).

That is, we reduce the sample space from \(\mathcal{S}\) to \(B\).

Conditional Probability¶

For the die roll example, if \(A = \{2,4\}\) and \(B = \{2,4,6\}\),

\[P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\;\; \frac{2}{6} \;\;}{\frac{3}{6}} = \frac{2}{3}.\]

Intuitively, if we know that 2,4 or 6 occurs, then the probability that a 2 or 4 occurs should be \(\frac{2}{3}\).

Bayes’ Theorem¶

From the definition of conditional probability,

(1)\[P(A|B) = \frac{P(A \cap B)}{P(B)} \quad \Rightarrow \quad P(A \cap B) = P(A|B) P(B).\]

Likewise

(2)\[P(B|A) = \frac{P(A \cap B)}{P(A)} \quad \Rightarrow \quad P(A \cap B) = P(B|A) P(A).\]

Bayes’ Theorem¶

Equating (1) and (2)

(3)\[P(A|B) P(B) = P(B|A) P(A).\]

Rearranging (3) gives Bayes’ Theorem:

\[P(B|A) = \frac{P(A|B) P(B)}{P(A)}.\]

Bayes’ Theorem¶

For the die roll example, if \(A = \{2,4\}\) and \(B = \{2,4,6\}\), we already know that

\(P(A) = \frac{2}{6}\).

\(P(B) = \frac{3}{6}\).

\(P(A|B) = \frac{2}{3}\).

Thus,

\[P(B|A) = \frac{P(A|B) P(B)}{P(A)} = \frac{\frac{2}{3} \times \frac{3}{6}}{\frac{2}{6}} = \frac{\;\; \frac{2}{6} \;\;}{\frac{2}{6}} = 1.\]

Partitions¶

\(\qquad\)

Partitions¶

\(\qquad\)

Partitions¶

\(\qquad\)

Partitions¶

\(\qquad\)

Partitions¶

\(\qquad\)

Partitions¶

Let \(B_1, B_2, \ldots, B_K\) be \(K\) subsets of \(\mathcal{S}\): \(B_i \subset \mathcal{S}\) for \(i = 1, \ldots, K\).

\(\{B_i\}_{i=1}^K\) is a partition of \(\mathcal{S}\) if

\[B_1 \cup B_2 \cup \cdots \cup B_K = \mathcal{S}\]

\[B_i \cap B_j = \emptyset \quad \text{for} i \neq j.\]

Note that \(A = (A \cap B_1) \cup \cdots \cup (A \cap B_K)\).

Partitions¶

Since \((A \cap B_i) \cap (A \cap B_j) = \emptyset\) for \(i \neq j\),

\[P(A) = P\left((A \cap B_1) \cup \cdots \cup (A \cap B_K)\right) \hspace{0.38in}\]

\[= P(A \cap B_1) + \cdots + P(A \cap B_K)\]

\[\hspace{0.55in} = P(A|B_1) P(B_1) + \cdots + P(A|B_K) P(B_K).\]

Bayes’ Theorem Extended¶

Given a partition \(B_1, \ldots, B_K\) of \(\mathcal{S}\), we can applied Bayes’ Theorem to each subset of the partition:

\[P(B_j|A) = \frac{P(A|B_j) P(B_j)}{P(A)} \hspace{1.98in}\]

\[\hspace{0.3in} = \frac{P(A|B_j) P(B_j)}{P(A|B_1) P(B_1) + \cdots + P(A|B_K) P(B_K)}.\]