The Class P | CS 139: Theory of Computation

<h1 id="the-class-p">The Class P</h1>

<hr />

<h2 id="administrivia">Administrivia</h2>
<ul>
  <li>Assignment 8 is now released
    <ul>
      <li>Due next Tuesday at 11:59 PM</li>
    </ul>
  </li>
  <li class="fragment">It will be the last normal homework assignment of the semester</li>
  <li class="fragment">Next week I will release a “video presentation” assignment where you will record a short video about an NP complete problem of your choice
    <ul>
      <li>More details will follow next week</li>
    </ul>
  </li>
</ul>

<h1 id="time-complexity">Time Complexity</h1>

<h2 id="time-complexity-1">Time Complexity</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$M$</code> be a Turing machine that always halts</li>
  <li>The (<strong>worst-case</strong>) <strong>running time</strong> of <code class="language-plaintext highlighter-rouge">$M$</code> is the function <code class="language-plaintext highlighter-rouge">$f:\mathbb{N}\rightarrow\mathbb{N}$</code> where <code class="language-plaintext highlighter-rouge">$f(n)$</code> is the <em>maximum</em> number of steps <code class="language-plaintext highlighter-rouge">$M$</code> takes on an input of length <code class="language-plaintext highlighter-rouge">$n$</code></li>
  <li class="fragment">Adding extra features to a computational model might affect the time complexity of solving problems</li>
</ul>

<h1 id="asymptotic-analysis">Asymptotic Analysis</h1>

<h2 id="asymptotic-analysis-1">Asymptotic Analysis</h2>
<ul>
  <li>Remember that <code class="language-plaintext highlighter-rouge">is_palindrome</code> took approximately <code class="language-plaintext highlighter-rouge">4n+1</code> steps in Python?</li>
  <li>Calculating the number of steps <strong>exactly</strong> is really tedious—especially for more complicated algorithms</li>
  <li>For large inputs, the running time is always dominated by the <strong>highest order term</strong>, so we tend to worry about it the most (e.g. the “<code class="language-plaintext highlighter-rouge">n</code>” in the  <code class="language-plaintext highlighter-rouge">4n + 1</code>)</li>
</ul>

<h2 id="asymptotic-analysis-2">Asymptotic Analysis</h2>
<ul>
  <li>What is the highest-order term in each of the following expressions?
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$7n^2 + 100n + 17$</code>
        <ul>
          <li><code class="language-plaintext highlighter-rouge">$n^2$</code></li>
        </ul>
      </li>
      <li><code class="language-plaintext highlighter-rouge">$50\log_{10}(n) + n\log_{2}(n) + 400\log_{10}(n)^2$</code>
        <ul>
          <li><code class="language-plaintext highlighter-rouge">$n\log(n)$</code></li>
        </ul>
      </li>
      <li><code class="language-plaintext highlighter-rouge">$n^{1000} + 2^n + 10$</code>
        <ul>
          <li><code class="language-plaintext highlighter-rouge">$2^n$</code></li>
        </ul>
      </li>
    </ul>
  </li>
</ul>

<h2 id="big-o-notation">Big-O Notation</h2>
<ul>
  <li>For each of these functions, we often write:
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$7n^2 + 100n + 17 = O(n^2)$</code></li>
      <li><code class="language-plaintext highlighter-rouge">$n^{1000} + 2^n + 10 = O(2^n)$</code></li>
    </ul>
  </li>
</ul>

<hr />

<ul>
  <li><code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> for two functions <code class="language-plaintext highlighter-rouge">$f,g:\mathbb{N}\rightarrow\mathbb{R}^+$</code> if:
    <ul>
      <li class="fragment"><strong>THERE EXIST</strong> positive integers <code class="language-plaintext highlighter-rouge">$c$</code> and <code class="language-plaintext highlighter-rouge">$n_0$</code> such that:
        <ul>
          <li class="fragment"><strong>FOR EVERY</strong> integer <code class="language-plaintext highlighter-rouge">$n \ge n_0$</code> we have:
            <ul>
              <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) \le c\cdot g(n)$</code></li>
            </ul>
          </li>
        </ul>
      </li>
    </ul>
  </li>
</ul>

<h2 id="big-o-notation-1">Big-O Notation</h2>
<ul>
  <li>What does <code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> intuitively mean?</li>
  <li class="fragment">That <strong>eventually</strong> <code class="language-plaintext highlighter-rouge">$g(n)$</code> will grow <strong>at least as fast</strong> as <code class="language-plaintext highlighter-rouge">$f(n)$</code></li>
</ul>

<hr />

<p class="fragment">Image by Victor S.Adamchik, CMU, 2009 at <a href="cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html">source</a></p>

<h2 id="big-o-exercises">Big-O Exercises</h2>
<ul>
  <li><strong>3-2-1-GO</strong>: Is <code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> in the following cases?
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n^2 + 4n + 1$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n^2$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n\log(n)$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = 2^{2n}$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = 2^n$</code></li>
    </ul>
  </li>
</ul>

<h2 id="little-o-notation">Little-O Notation</h2>
<ul>
  <li>Another useful definition is “little-o”</li>
  <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))$</code> for two functions <code class="language-plaintext highlighter-rouge">$f,g:\mathbb{N}\rightarrow\mathbb{R}^+$</code> if:</li>
</ul>

<p class="fragment"><code class="language-plaintext highlighter-rouge">$$\lim_{n\rightarrow\infty}\frac{f(n)}{g(n)} = 0$$</code></p>

<ul>
  <li class="fragment">We say that “<code class="language-plaintext highlighter-rouge">$f$</code> is little-o of <code class="language-plaintext highlighter-rouge">$g$</code>” and that <code class="language-plaintext highlighter-rouge">$g$</code> <strong>dominates</strong> <code class="language-plaintext highlighter-rouge">$f$</code></li>
</ul>

<h2 id="little-o-exercises">Little-O Exercises</h2>
<ul>
  <li><strong>3-2-1-GO</strong>: Is <code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))$</code> in the following cases?
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n^2 + 4n + 1$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n^2$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n\log(n)$</code></li>
    </ul>
  </li>
</ul>

<h2 id="little-o-properties">Little-O Properties</h2>
<ul>
  <li>Notice that <code class="language-plaintext highlighter-rouge">$n^2 = O(n^2)$</code> but <code class="language-plaintext highlighter-rouge">$n \ne o(n)$</code></li>
  <li class="fragment">Big-O is similar to <code class="language-plaintext highlighter-rouge">$\le$</code>
    <ul>
      <li>Functions that grow at <strong>similar rates</strong> are Big-O of one-another</li>
    </ul>
  </li>
  <li class="fragment">Little-O is similar to <code class="language-plaintext highlighter-rouge">$<$</code>
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$g(n)$</code> must grow <strong>strictly faster</strong> than <code class="language-plaintext highlighter-rouge">$f(n)$</code></li>
    </ul>
  </li>
</ul>

<hr />

<ul>
  <li class="fragment"><strong>Fact:</strong> <code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))\implies f(n) = O(g(n))$</code></li>
</ul>

<h2 id="time-complexity-2">Time Complexity</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$t:\mathbb{N}\rightarrow\mathbb{R}^+$</code> be a function</li>
  <li class="fragment">Then <code class="language-plaintext highlighter-rouge">$\text{TIME}(t(n))$</code> is the collection of languages that are decidable by a single-tape deterministic Turing machine in <code class="language-plaintext highlighter-rouge">$O(t(n)$</code> time</li>
  <li class="fragment">We call <code class="language-plaintext highlighter-rouge">$\text{TIME}(t(n))$</code> a <strong>time complexity class</strong></li>
</ul>

<h2 id="time-complexity-3">Time Complexity</h2>
<ul>
  <li>Previously we showed that the language of palindromes is decidable in <code class="language-plaintext highlighter-rouge">$O(n^2)$</code> steps by a deterministic Turing machine; this means that it is in <code class="language-plaintext highlighter-rouge">$\text{TIME}(n^2)$</code></li>
</ul>

<h1 id="the-class-p-1">The Class P</h1>

<h2 id="polynomial-time-complexity">Polynomial Time Complexity</h2>
<ul>
  <li>An especially useful complexity class is the class <code class="language-plaintext highlighter-rouge">$\text{P}$</code> of languages decidable in <strong>polynomial time</strong></li>
</ul>

<p><code class="language-plaintext highlighter-rouge">$$\text{P} = \bigcup_{k\in\mathbb{N}} \text{TIME}(n^k)$$</code></p>

<h2 id="exponential-time-complexity">Exponential Time Complexity</h2>
<ul>
  <li>Another useful class is <code class="language-plaintext highlighter-rouge">$\text{EXPTIME}$</code> which are the languages decidable in <strong>exponential time</strong></li>
</ul>

<p><code class="language-plaintext highlighter-rouge">$$\text{EXPTIME} = \bigcup_{k\in\mathbb{N}} \text{TIME}(2^{n^k})$$</code></p>

<h2 id="easy-and-hard-problems">“Easy” and “Hard” Problems</h2>
<ul>
  <li>Polynomial-time algorithms are considered <strong>fast</strong></li>
  <li>Exponential-time algorithms are considered <strong>very slow</strong></li>
</ul>

<hr />

<ul>
  <li class="fragment">Suppose we have two algorithms <code class="language-plaintext highlighter-rouge">$M_A$</code> and <code class="language-plaintext highlighter-rouge">$M_B$</code> that run in <code class="language-plaintext highlighter-rouge">$O(n^3)$</code>-time and <code class="language-plaintext highlighter-rouge">$O(2^n)$</code>-time, respectively
    <ul>
      <li class="fragment">For an input size of <code class="language-plaintext highlighter-rouge">$n = 1000$</code>, we have that:</li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$M_A$</code> takes roughly <code class="language-plaintext highlighter-rouge">$(1000)^3 = 10^9$</code> steps</li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$M_B$</code> takes <code class="language-plaintext highlighter-rouge">$2^{1000} > 10^{300}$</code> steps</li>
    </ul>
  </li>
</ul>

<h2 id="sanity-check">Sanity Check</h2>
<ul>
  <li>Consider the following three algorithms that run in polynomial time:
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$M_A$</code> takes <code class="language-plaintext highlighter-rouge">$n$</code> steps</li>
      <li><code class="language-plaintext highlighter-rouge">$M_B$</code> takes <code class="language-plaintext highlighter-rouge">$100n$</code> steps</li>
      <li><code class="language-plaintext highlighter-rouge">$M_C$</code> takes <code class="language-plaintext highlighter-rouge">$n^{1000}$</code> steps</li>
    </ul>
  </li>
  <li class="fragment"><strong>3-2-1-GO:</strong> All four algorithms are poly-time, does that mean in practice their performance is unnoticeable?
    <ul>
      <li class="fragment">No! Clearly <code class="language-plaintext highlighter-rouge">$M_A$</code> is the most desirable!</li>
      <li class="fragment">Even though <code class="language-plaintext highlighter-rouge">$M_C$</code> is poly-time, it still takes an unpractical amount of steps</li>
      <li class="fragment">Even though <code class="language-plaintext highlighter-rouge">$M_A$</code> and <code class="language-plaintext highlighter-rouge">$M_B$</code> are both <code class="language-plaintext highlighter-rouge">$O(n)$</code>, we still care about the 100x speedup in practice</li>
    </ul>
  </li>
</ul>

<h2 id="why-is-textp-so-important-then">Why is <code class="language-plaintext highlighter-rouge">$\text{P}$</code> so important then?</h2>
<ul>
  <li>The difference of <code class="language-plaintext highlighter-rouge">$O(n)$</code> and <code class="language-plaintext highlighter-rouge">$O(n^{1000})$</code> is substantial</li>
  <li class="fragment">The difference of <code class="language-plaintext highlighter-rouge">$n^{100}$</code> and <code class="language-plaintext highlighter-rouge">$2^n$</code> is <strong>unbelievably enormous</strong></li>
  <li class="fragment">Historically when an algorithm is discovered for a problem that runs in <code class="language-plaintext highlighter-rouge">$O(n^k)$</code> for a large <code class="language-plaintext highlighter-rouge">$k$</code>, it is generally <strong>quickly optimized</strong> so that <code class="language-plaintext highlighter-rouge">$k$</code> is small</li>
  <li class="fragment">Thus <code class="language-plaintext highlighter-rouge">$\text{P}$</code> effectively captures the class of problems that can be solved practically by computers</li>
  <li class="fragment">Problems outside of <code class="language-plaintext highlighter-rouge">$\text{P}$</code> are called <strong>intractable</strong> because in practice, the computational resources required to solve them are too vast to be used in practice</li>
</ul>

<h2 id="showing-problems-are-in-p">Showing Problems are in P</h2>
<ul>
  <li>Consider the following algorithm for sorting a list:</li>
</ul>

<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">insertion_sort</span><span class="p">(</span><span class="n">lst</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">lst</span><span class="p">)):</span>
        <span class="n">insert_left</span><span class="p">(</span><span class="n">lst</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">insert_left</span><span class="p">(</span><span class="n">lst</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">i</span> <span class="o">></span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">lst</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o"><</span> <span class="n">lst</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
        <span class="n">swap</span><span class="p">(</span><span class="n">lst</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
        <span class="n">insert_left</span><span class="p">(</span><span class="n">lst</span><span class="p">,</span> <span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">swap</span><span class="p">(</span><span class="n">lst</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
    <span class="n">lst</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">lst</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">lst</span><span class="p">[</span><span class="n">j</span><span class="p">],</span> <span class="n">lst</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
</code></pre></div></div>
<ul>
  <li class="fragment"><strong>3-2-1-GO:</strong> What is the runtime of the algorithm?
    <ul>
      <li class="fragment">It is <code class="language-plaintext highlighter-rouge">$O(n^2)$</code></li>
    </ul>
  </li>
</ul>

<h2 id="strategy-for-showing-problems-are-in-p">Strategy for Showing Problems are in P</h2>
<ol>
  <li>Write an algorithm that solves the problem</li>
  <li>Identify the steps where <strong>repetition</strong> occurs</li>
  <li>Make sure each step of the algorithm runs in poly-time and repetition only occurs a poly numbers of times</li>
</ol>

<hr />

<ul>
  <li class="fragment">If these are true, then the entire algorithm will run in poly-time since the product/sum of polynomials is also a polynomial</li>
</ul>

<h2 id="example">Example</h2>
<p>Show that the following language is in <code class="language-plaintext highlighter-rouge">$\text{P}$</code>:</p>

<p><code class="language-plaintext highlighter-rouge">$$E_\text{DFA} = \{\langle D\rangle \mid D\text{ is a DFA and }L(D)=\emptyset\}$$</code></p>

<ul>
  <li class="fragment"><strong>3-2-1-GO:</strong> How can we check if a DFA never accepts a single input string?
    <ul>
      <li class="fragment">By checking if no accepting state is <strong>reachable</strong> from the start state</li>
    </ul>
  </li>
</ul>

<h2 id="example-1">Example</h2>
<ul>
  <li>Here is a <strong>high-level Turing machine</strong> that decides <code class="language-plaintext highlighter-rouge">$E_\text{DFA}$</code>:</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = "On input <D> where D is a DFA
  1. Add the start state of D to a set R of reachable states
  2. For each state of D not in R:
     a. For each state q1 in R:
        i. For each state q2 that is reachable from q1 on some
           symbol a, add it to R
     b. If no new states were added to R in step a., then break
  3. Iterate over the states in R
     a. If any of them are accepting states, REJECT
     b. If all of them are not accepting states, ACCEPT
</code></pre></div></div>

<ul>
  <li class="fragment">Runs in poly-time since every step runs in poly-time</li>
</ul>

<h2 id="proving-languages-arent-in-p">Proving Languages Aren’t in P</h2>
<ul>
  <li><strong>3-2-1-GO</strong>: How do you think we can prove that a language is <strong>not</strong> decidable in polynomial time?</li>
  <li class="fragment">Using a reductions!
    <ul>
      <li>We will be exploring techniques like this next week</li>
    </ul>
  </li>
  <li class="fragment"><strong>3-2-1-GO</strong>: Can you think of a practical problem that you don’t believe is in <code class="language-plaintext highlighter-rouge">$\text{P}$</code>?
    <ul>
      <li class="fragment">Optimal (generalized) Chess, Checkers, or Go algorithms</li>
    </ul>
  </li>
</ul>

<h2 id="one-last-exercise">One Last Exercise</h2>
<ul>
  <li>An undirected graph <code class="language-plaintext highlighter-rouge">$G = (V,E)$</code> <strong>contains a triangle</strong> if there is a fully connected subgraph of size 3</li>
  <li>Show that the following language is in <code class="language-plaintext highlighter-rouge">$\text{P}$</code>:</li>
</ul>

<p><code class="language-plaintext highlighter-rouge">$$\text{TRIANGLE} = \{\langle G\rangle \mid G\text{ contains a triangle}\}$$</code></p>