The Halting Problem | CS 139: Theory of Computation

<h1 id="the-halting-problem">The Halting Problem</h1>

<hr />

<h2 id="administrivia">Administrivia</h2>
<ul>
  <li>Classes be online for <strong>for the whole semester</strong></li>
  <li class="fragment">Using <strong>Slack</strong> as a primary remote communication tool
    <ul>
      <li class="fragment">Please <strong>download the app</strong> on your computer</li>
      <li class="fragment">Makes communication seamless and easy to chat remotely—much quicker than email</li>
      <li class="fragment">Message me <strong>anytime</strong> with any questions you have</li>
    </ul>
  </li>
</ul>

<h1 id="decidability">DECIDABILITY</h1>

<h2 id="high-level-tm-for-a_textdfa">High-Level TM for <code class="language-plaintext highlighter-rouge">$A_\text{DFA}$</code></h2>

<p><code class="language-plaintext highlighter-rouge">$$A_\text{DFA} = \{\langle B,w\rangle \mid B\text{ is a DFA that accepts }w\}$$</code></p>

<hr />

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = On input &lt;B,w&gt;
  1. Simulate B on w
  2. If B accepts, then ACCEPT
  3. If B rejects, then REJECT
</code></pre></div></div>

<hr />

<ul>
  <li class="fragment">This is a reasonable Turing machine description</li>
  <li class="fragment">It abstracts away low-level details of simulating <code class="language-plaintext highlighter-rouge">$B$</code> on <code class="language-plaintext highlighter-rouge">$w$</code></li>
</ul>

<h2 id="another-example">Another Example</h2>
<p><code class="language-plaintext highlighter-rouge">$$ALL_\text{DFA} = \{\langle A\rangle \mid A\text{ is a DFA and }L(A)=\Sigma^\ast\}$$</code></p>

<hr />

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = On input &lt;A&gt; where A is a DFA
  1. Simulate the DFA A on all strings
  2. If it rejects any string, then REJECT
  3. If it accepts all strings, then ACCEPT
</code></pre></div></div>

<hr />

<ul>
  <li>Notice a problem here?</li>
  <li class="fragment">There are an infinite number of strings! Thus, it is possible for this machine to loop forever</li>
  <li class="fragment">If we want to <strong>decide</strong> this language, it must always halt</li>
</ul>

<h2 id="exercise-think-pair-share">Exercise: Think-Pair-Share</h2>
<p>Again, I find it useful to think about Python:</p>

<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">dfa_accepts_everything</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
    <span class="s">"""Returns True if the DFA A accepts every string"""</span>
    <span class="p">...</span>
</code></pre></div></div>

<ul>
  <li><strong>Question</strong>: How would you implement this in such a way that it will always halt?</li>
</ul>

<hr />

<ul>
  <li class="fragment"><strong>THINK:</strong> Take one minute and jot down your thoughts</li>
  <li class="fragment"><strong>PAIR:</strong> Share your thoughts in a breakout session</li>
  <li class="fragment"><strong>SHARE:</strong> Afterwards, we’ll discuss as a large group</li>
</ul>

<h2 id="idea-for-deciding-all_textdfa">Idea for Deciding <code class="language-plaintext highlighter-rouge">$ALL_\text{DFA}$</code></h2>
<ul>
  <li>
    <p>Check all the reachable states of <code class="language-plaintext highlighter-rouge">$A$</code> and make sure all of them are in <code class="language-plaintext highlighter-rouge">$F$</code>—if so, accept—otherwise reject</p>
  </li>
  <li class="fragment">
    <p>This is basically a <strong>breadth-first search</strong> through a graph data structure</p>
  </li>
</ul>

<h2 id="proof-sketch">Proof Sketch</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = On input &lt;A&gt; where A is a DFA
  1. Mark the starting state of A
  2. Repeat until no more states can be marked this way:
     a. Iterate over each marked state and mark the states that
        are immediately adjacent to it
  3. Check to see if every marked state is an accepting state,
     if so, ACCEPT, if not, REJECT
</code></pre></div></div>

</div>

<ul>
  <li class="fragment">This algorithm will always halt and accept if and only if all strings are accepted by <code class="language-plaintext highlighter-rouge">$A$</code>. Thus, it decides <code class="language-plaintext highlighter-rouge">$ALL_\text{DFA}$</code></li>
</ul>

<h2 id="countable-and-uncountable">Countable and Uncountable</h2>
<h1 id="sets">SETS</h1>

<h2 id="cardinality-and-counting">Cardinality and Counting</h2>
<ul>
  <li>Suppose I have a bag of <strong>apples</strong> and a bag of <strong>bananas</strong></li>
  <li>How would you determine whether there are <strong>more</strong> apples or bananas?</li>
  <li class="fragment">One approach is to <strong>count</strong> them by removing one apple and one banana from each bag at a time</li>
  <li class="fragment">This provides an <strong>ordering</strong> of the apples and the bananas</li>
</ul>

<h2 id="cardinality-and-counting-1">Cardinality and Counting</h2>
<ul>
  <li>We can represent the bags of fruit as <strong>sets</strong></li>
</ul>

<ul>
  <li class="fragment">We know that the <strong>cardinality</strong> of the sets is the same because we can pair each element of <code class="language-plaintext highlighter-rouge">$A$</code> with one from <code class="language-plaintext highlighter-rouge">$B$</code> such that none remain unpaired</li>
</ul>

<h2 id="cardinality-and-counting-2">Cardinality and Counting</h2>
<ul>
  <li>Now consider the following <strong>infinite sets</strong>:
<code class="language-plaintext highlighter-rouge">$$\mathbb{N} = \{1, 2, 3, 4, 5, \ldots\}$$</code>
<code class="language-plaintext highlighter-rouge">$$\mathbb{E} = \{2, 4, 6, 8, 10, \ldots\}$$</code></li>
</ul>

<hr />

<p><strong>3-2-1-GO Question</strong>: Which set has more elements?</p>

<hr />

<ul>
  <li class="fragment">Notice that <code class="language-plaintext highlighter-rouge">$\mathbb{E}\subset\mathbb{N}$</code>, so in one sense <code class="language-plaintext highlighter-rouge">$\mathbb{N}$</code> has <em>more</em> elements than <code class="language-plaintext highlighter-rouge">$\mathbb{E}$</code></li>
  <li class="fragment">However, they have the <strong>same cardinality</strong></li>
</ul>

<h2 id="cardinality-and-counting-3">Cardinality and Counting</h2>
<p><code class="language-plaintext highlighter-rouge">$$\mathbb{N} = \{1, 2, 3, 4, 5, \ldots\},\quad \mathbb{E} = \{2, 4, 6, 8, 10, \ldots\}$$</code></p>

<hr />

<ul>
  <li class="fragment">We can pair each element of <code class="language-plaintext highlighter-rouge">$n\in \mathbb{N}$</code> with the element <code class="language-plaintext highlighter-rouge">$f(n)\in \mathbb{E}$</code> using the function <code class="language-plaintext highlighter-rouge">$f(n) = 2n$</code></li>
  <li class="fragment"><code class="language-plaintext highlighter-rouge">$f$</code> is a <strong>one-to-one correspondence</strong> (aka a <strong>bijection</strong>) from the set <code class="language-plaintext highlighter-rouge">$\mathbb{N}$</code> to the set <code class="language-plaintext highlighter-rouge">$\mathbb{E}$</code></li>
  <li class="fragment">The existence of <code class="language-plaintext highlighter-rouge">$f$</code> by definition means that <code class="language-plaintext highlighter-rouge">$|\mathbb{N}| = |\mathbb{E}|$</code></li>
  <li class="fragment">An infinite set <code class="language-plaintext highlighter-rouge">$S$</code> is called <strong>countable</strong> if <code class="language-plaintext highlighter-rouge">$|S|=|\mathbb{N}|$</code></li>
</ul>

<h2 id="countable-languages">Countable Languages</h2>
<p><strong>Claim:</strong> <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast$</code> is countable</p>

<hr />

<ul>
  <li>Is the claim true?</li>
</ul>

<hr />

<h2 id="countable-languages-1">Countable Languages</h2>
<p><strong>Theorem:</strong> <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast$</code> is countable</p>

<hr />

<ul>
  <li>To prove this, we need a function <code class="language-plaintext highlighter-rouge">$f$</code> that maps strings to natural numbers</li>
  <li><strong>Proof Idea:</strong>
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast = \{\epsilon, 0, 1, 00, 01, 10, 11, 000, \ldots\}$</code></li>
      <li class="fragment">This is called the <strong>lexicographical order</strong> of <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast$</code></li>
      <li class="fragment">We also write <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast = \{s_1, s_2, s_3, \ldots \}$</code></li>
      <li class="fragment">Define <code class="language-plaintext highlighter-rouge">$f$</code> in such a way that <code class="language-plaintext highlighter-rouge">$f(s_n) = n$</code></li>
    </ul>
  </li>
</ul>

<h2 id="countable-languages-2">Countable Languages</h2>
<p><strong>Theorem:</strong> <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast$</code> is countable</p>

<hr />

<ul>
  <li>Define <code class="language-plaintext highlighter-rouge">$f:\{0,1\}^\ast\rightarrow\mathbb{N}$</code> by
<code class="language-plaintext highlighter-rouge">$$f(w) = 2^{|w|}+\text{val}(w)$$</code>
where <code class="language-plaintext highlighter-rouge">$\text{val}:\{0,1\}^\ast\rightarrow\mathbb{Z}_{\ge 0}$</code> is the decimal value of the string <code class="language-plaintext highlighter-rouge">$w\in\{0,1\}^\ast$</code>
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$\text{val}(011) = 3$</code> and <code class="language-plaintext highlighter-rouge">$\text{val}(1001) = 9$</code></li>
    </ul>
  </li>
  <li class="fragment"><code class="language-plaintext highlighter-rouge">$f$</code> is a <strong>bijection</strong> and so <code class="language-plaintext highlighter-rouge">$|\{0,1\}^\ast|=|\mathbb{N}|$</code></li>
</ul>

<h2 id="encoding-languages-as-sequences">Encoding Languages as Sequences</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$A\subseteq\{0,1\}^\ast$</code> be a language</li>
  <li class="fragment">The <strong>characteristic sequence</strong> of <code class="language-plaintext highlighter-rouge">$A$</code> is the infinite sequence <code class="language-plaintext highlighter-rouge">$\chi_A = b_1b_2b_3b_4\cdots$</code>, where
<code class="language-plaintext highlighter-rouge">$$
b_i =
\begin{cases}
1, &\text{if }s_i \in A\\
0, &\text{if }s_i \not\in A
\end{cases}
$$</code></li>
</ul>

<h2 id="example">Example</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$L = \{00, 1, 10\}$</code></li>
  <li><strong>3-2-1-GO Question</strong>: What is <code class="language-plaintext highlighter-rouge">$\chi_L$</code></li>
</ul>

<hr />

<ul>
  <li>The following may be helpful:
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast = \{\epsilon, 0, 1, 00, 01, 10, 11, 000, \ldots\}$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$\chi_L = 0011010000\cdots$</code></li>
    </ul>
  </li>
</ul>

<h2 id="encoding-languages-as-sequences-1">Encoding Languages as Sequences</h2>
<ul>
  <li>Every language <code class="language-plaintext highlighter-rouge">$A\subseteq\{0,1\}^\ast$</code> has a unique characteristic sequence <code class="language-plaintext highlighter-rouge">$\chi_A$</code></li>
  <li class="fragment">Thus the set <code class="language-plaintext highlighter-rouge">$\mathbb{B} = \{0,1\}^\infty$</code> of all infinite sequences of 0’s and 1’s represents the set of <strong>all languages</strong> over the alphabet <code class="language-plaintext highlighter-rouge">$\{0, 1\}$</code></li>
</ul>

<h1 id="theorem">Theorem</h1>
<p>The set of languages <code class="language-plaintext highlighter-rouge">$\mathbb{B}$</code> is <strong>uncountable</strong></p>

<h3 id="proof-by-contradiction">Proof (by contradiction)</h3>
<ul>
  <li class="fragment">Assume that <code class="language-plaintext highlighter-rouge">$\mathbb{B}$</code> is countable</li>
  <li class="fragment">Then there is a bijection <code class="language-plaintext highlighter-rouge">$f:\mathbb{N}\rightarrow\mathbb{B}$</code></li>
  <li class="fragment">Let <code class="language-plaintext highlighter-rouge">$s\in\mathbb{B}$</code> be the sequence <code class="language-plaintext highlighter-rouge">$s = b_1b_2b_3b_4\cdots$</code> where
<code class="language-plaintext highlighter-rouge">$$b_i = \begin{cases}
1, &\text{if the $i$th bit of $f(i)$ is $0$}\\
0, &\text{if the $i$th bit of $f(i)$ is $1$}
\end{cases}$$</code></li>
  <li class="fragment">Then for each <code class="language-plaintext highlighter-rouge">$n\in\mathbb{N}$</code>, <code class="language-plaintext highlighter-rouge">$s$</code> differs from <code class="language-plaintext highlighter-rouge">$f(n)$</code> by the <code class="language-plaintext highlighter-rouge">$n$</code>th bit</li>
  <li class="fragment">This means that no <code class="language-plaintext highlighter-rouge">$n\in\mathbb{N}$</code> exists such that <code class="language-plaintext highlighter-rouge">$f(n) = s$</code>, which means that <code class="language-plaintext highlighter-rouge">$f$</code> is not a bijection—a contradiction</li>
</ul>

<h2 id="most-languages-are-not-recognizable">Most Languages are Not Recognizable</h2>
<ul>
  <li><strong>Fact:</strong> The number of Turing machines is <strong>countable</strong>
    <ul>
      <li class="fragment"><strong>3-2-1-GO Question</strong>: Why is this true?</li>
      <li class="fragment">Every Turing machine has a finite description—which can be encoded as a binary string <code class="language-plaintext highlighter-rouge">$\langle M\rangle$</code></li>
      <li class="fragment">Since <code class="language-plaintext highlighter-rouge">$\{0,1\}^\ast$</code> is countable, so is set of Turing machines <code class="language-plaintext highlighter-rouge">$\{\langle M\rangle \mid M\text{ is a TM}\}\subseteq\{0,1\}^\ast$</code></li>
    </ul>
  </li>
  <li class="fragment"><strong>Corollary:</strong> Most languages are not Turing recognizable</li>
</ul>

<h1 id="universal">Universal</h1>
<h2 id="turing-machines">Turing Machines</h2>

<h2 id="universal-turing-machines">Universal Turing Machines</h2>
<p><strong>Theorem:</strong> The following language is Turing recognizable:
<code class="language-plaintext highlighter-rouge">$$A_\text{TM} = \{\langle M, w\rangle \mid M\text{ is a TM that accepts } w\}$$</code></p>

<div class="language-plaintext fragment highlighter-rouge"><div class="highlight"><pre class="highlight"><code>U = On input &lt;M,w&gt; where M is a Turing machine
  1. Simulate M on w
  2. If M halts and accepts, then ACCEPT
  3. If M halts rejects, then REJECT
</code></pre></div></div>

<h2 id="universal-turing-machines-1">Universal Turing Machines</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>U = On input &lt;M,w&gt; where M is a Turing machine
  1. Simulate M on w
  2. If M halts and accepts, then ACCEPT
  3. If M halts rejects, then REJECT
</code></pre></div></div>

<hr />

<ul>
  <li>Does this seem weird to you?</li>
  <li class="fragment"><strong>3-2-1-GO Question</strong>: What is a real-life example of a universal program that can simulate all other programs?</li>
  <li class="fragment">The <strong>Python interpreter</strong> is one such a program</li>
</ul>

<h2 id="universal-turing-machines-2">Universal Turing Machines</h2>
<ul>
  <li>Since <code class="language-plaintext highlighter-rouge">$U$</code> is a Turing machine, it’s possible to give it an input of <code class="language-plaintext highlighter-rouge">$\langle U, w\rangle$</code>—i.e., a description of <strong>itself</strong></li>
  <li class="fragment"><strong>3-2-1-GO Question:</strong> Can you think of a real-life example of running a program on itself?</li>
  <li class="fragment">Since the <code class="language-plaintext highlighter-rouge">gcc</code> compiler is written in C…
    <ul>
      <li>So it can be given its own source code as input and it will <strong>compile itself</strong> into machine code</li>
      <li class="fragment">This is called <strong>bootstrapping</strong> and is much more common than you think!</li>
    </ul>
  </li>
  <li class="fragment">A Python program <code class="language-plaintext highlighter-rouge">program.py</code> could also open itself:
    <div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">"program.py"</span><span class="p">)</span>
</code></pre></div>    </div>
  </li>
</ul>

<h1 id="the-halting-problem-1">The Halting Problem</h1>

<h2 id="the-halting-problem-2">The Halting Problem</h2>
<ul>
  <li>Alan Turing famously proved that the <strong>halting problem</strong> is undecidable:
<code class="language-plaintext highlighter-rouge">$$\text{HALT} = \{\langle M, w\rangle \mid M\text{ is a TM that halts on }w\}$$</code></li>
  <li class="fragment">It is possible to prove this in a similar way to how we proved that <code class="language-plaintext highlighter-rouge">$\mathbb{B}$</code> is uncountable</li>
</ul>

<h2 id="proof-idea-by-contradiction">Proof Idea: (by contradiction)</h2>
<ul>
  <li>Assume that <code class="language-plaintext highlighter-rouge">$\text{HALT}$</code> is decidable</li>
  <li class="fragment">Then there exists a Turing machine <code class="language-plaintext highlighter-rouge">$K$</code> that decides it</li>
  <li class="fragment">Therefore we know that for every input <code class="language-plaintext highlighter-rouge">$\langle M, w\rangle$</code>:
<code class="language-plaintext highlighter-rouge">$$K(M,w) = \begin{cases}
\text{ACCEPT}, &\text{if $M$ halts on $w$}\\
\text{REJECT}, &\text{if $M$ loops forever on $w$}
\end{cases}$$</code></li>
</ul>

<h2 id="proof-idea-continued">Proof Idea Continued…</h2>
<ul>
  <li>Since the number of TMs is countable it is possible to write them in-order <code class="language-plaintext highlighter-rouge">$M_1, M_2, M_3, \ldots$</code></li>
  <li class="fragment">Thus we can build a table like the following:</li>
</ul>

<table class="fragment">
  <tbody>
    <tr>
      <td> </td>
      <td><code class="language-plaintext highlighter-rouge">$\epsilon$</code></td>
      <td><code class="language-plaintext highlighter-rouge">$0$</code></td>
      <td><code class="language-plaintext highlighter-rouge">$1$</code></td>
      <td><code class="language-plaintext highlighter-rouge">$00$</code></td>
      <td><code class="language-plaintext highlighter-rouge">$01$</code></td>
      <td><code class="language-plaintext highlighter-rouge">$\cdots$</code></td>
    </tr>
    <tr>
      <td>M_1</td>
      <td>H</td>
      <td>H</td>
      <td>H</td>
      <td>L</td>
      <td>H</td>
      <td><code class="language-plaintext highlighter-rouge">$\cdots$</code></td>
    </tr>
    <tr>
      <td>M_2</td>
      <td>L</td>
      <td>H</td>
      <td>L</td>
      <td>L</td>
      <td>H</td>
      <td><code class="language-plaintext highlighter-rouge">$\cdots$</code></td>
    </tr>
    <tr>
      <td>M_3</td>
      <td>H</td>
      <td>H</td>
      <td>L</td>
      <td>L</td>
      <td>L</td>
      <td><code class="language-plaintext highlighter-rouge">$\cdots$</code></td>
    </tr>
    <tr>
      <td>M_4</td>
      <td>H</td>
      <td>H</td>
      <td>L</td>
      <td>L</td>
      <td>L</td>
      <td><code class="language-plaintext highlighter-rouge">$\cdots$</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">$\vdots$</code></td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
    </tr>
  </tbody>
</table>

<h2 id="proof-idea-continued-1">Proof Idea Continued…</h2>
<ul>
  <li>Note that both of the following are true
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$K(M,w)$</code> will always tell us if <code class="language-plaintext highlighter-rouge">$M$</code> halts on <code class="language-plaintext highlighter-rouge">$w$</code></li>
      <li class="fragment"><strong>Every</strong> Turing machine appears in this table</li>
    </ul>
  </li>
  <li class="fragment">To reach a contradiction, we will construct a TM that is different from <strong>every TM in the list</strong></li>
  <li class="fragment">Let <code class="language-plaintext highlighter-rouge">$N$</code> be the Turing machine defined by:</li>
</ul>

<div class="language-plaintext fragment highlighter-rouge"><div class="highlight"><pre class="highlight"><code>N = "On input string s_n,
  1. Run K on &lt;M_n, s_n&gt;
  2. If K accepts, then enter an infinite loop
  3. If K rejects, then ACCEPT"
</code></pre></div></div>

<h2 id="proof-idea-continued-2">Proof Idea Continued</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>N = "On input string s_n,
  1. Run K on &lt;M_n, s_n&gt;
  2. If K accepts, then enter an infinite loop
  3. If K rejects, then ACCEPT"
</code></pre></div></div>

<ul>
  <li>Notice that for each <code class="language-plaintext highlighter-rouge">$n\in\mathbb{N}$</code> we know the following:
    <ul>
      <li class="fragment">If <code class="language-plaintext highlighter-rouge">$K(M_n, s_n)$</code> accepts, then <code class="language-plaintext highlighter-rouge">$N(s_n)$</code> loops forever</li>
      <li class="fragment">If <code class="language-plaintext highlighter-rouge">$K(M_n, s_n)$</code> rejects, then <code class="language-plaintext highlighter-rouge">$N(s_n)$</code> halts</li>
    </ul>
  </li>
  <li class="fragment">This means that <code class="language-plaintext highlighter-rouge">$N \ne M_n$</code> for each <code class="language-plaintext highlighter-rouge">$n\in\mathbb{N}$</code></li>
  <li class="fragment">Thus <code class="language-plaintext highlighter-rouge">$N$</code> is not in the list—a contradiction</li>
</ul>

<h2 id="proof-debrief">Proof Debrief</h2>
<ul>
  <li>Note that this contradiction <strong>does NOT mean</strong> that the number of Turing machines is uncountable!</li>
  <li class="fragment">Since <code class="language-plaintext highlighter-rouge">$N$</code> is a valid Turing machine, it <strong>DOES appear</strong> in the list—so there exists an <code class="language-plaintext highlighter-rouge">$n\in\mathbb{N}$</code> such that <code class="language-plaintext highlighter-rouge">$N = M_n$</code></li>
  <li class="fragment">However, it means that <code class="language-plaintext highlighter-rouge">$K$</code>—the machine that is supposed to decide the halting problem—<strong>fails</strong></li>
  <li class="fragment">In particular, it will give the wrong output if it is given <code class="language-plaintext highlighter-rouge">$\langle N, s_n\rangle$</code> as a parameter</li>
</ul>