Measuring Complexity | CS 139: Theory of Computation

<h1 id="measuring-complexity">Measuring Complexity</h1>

<hr />

<h2 id="administrivia">Administrivia</h2>
<ul>
  <li>No assignment due next week</li>
  <li>Exam 2 is graded (will be returned after class)</li>
</ul>

<h2 id="exam-2-debrief">Exam 2: Debrief</h2>
<ul>
  <li>Average was 61.31%</li>
  <li class="fragment">Problem 5 and 6 scores were especially low
    <ul>
      <li>These problems were unintentionally tricky</li>
    </ul>
  </li>
  <li class="fragment">Your final grade for the exam will be calculated by taking the maximum of two grades:
    <ol>
      <li class="fragment">Original exam score (unaltered)</li>
      <li class="fragment">Exam score with problems 5 and 6 given <strong>half weight</strong></li>
    </ol>
  </li>
</ul>

<h2 id="course-themes">Course Themes</h2>
<h1 id="revisited">Revisited</h1>

<h2 id="1-automata-theory">1. Automata Theory</h2>
<blockquote>
  <p>The study of abstract <strong>machines</strong></p>
</blockquote>

<h2 id="2-computability-theory">2. Computability Theory</h2>
<blockquote>
  <p>The study of the <strong>limits</strong> of computation</p>
</blockquote>

<h2 id="2-computability-theory-1">2. Computability Theory</h2>
<ul>
  <li>What does it mean for a function to be <strong>computable</strong>?</li>
</ul>

<hr />

<ul>
  <li><em>Your</em> ideas from the first day of class:
    <ul>
      <li class="fragment">It can be computed in a finite number of operations</li>
      <li class="fragment">There is an “algorithm” that, when given <code class="language-plaintext highlighter-rouge">$n$</code>, outputs <code class="language-plaintext highlighter-rouge">$f(n)$</code> on all inputs
        <ul>
          <li>An algorithm: a series of steps / operations</li>
        </ul>
      </li>
      <li class="fragment">If a “machine” can compute the function</li>
    </ul>
  </li>
</ul>

<h2 id="2-computability-theory-2">2. Computability Theory</h2>

<ul>
  <li>A function <code class="language-plaintext highlighter-rouge">$f:\Sigma^\ast\rightarrow\Sigma^\ast$</code> is a <strong>computable</strong> if there exists a Turing machine <code class="language-plaintext highlighter-rouge">$M_f$</code> such that on every string <code class="language-plaintext highlighter-rouge">$w\in\Sigma^\ast$</code>, running <code class="language-plaintext highlighter-rouge">$M_f$</code> on <code class="language-plaintext highlighter-rouge">$w$</code> results in <code class="language-plaintext highlighter-rouge">$M_f$</code> halting with <code class="language-plaintext highlighter-rouge">$f(w)$</code> written on its tape.</li>
  <li class="fragment"><strong>Church-Turing Thesis:</strong> Every reasonable computational model is equivalent to the Turing machine</li>
</ul>

<h2 id="2-computability-theory-3">2. Computability Theory</h2>
<ul>
  <li>We’ve been encoding computational problems as <strong>languages</strong> for simplicity reasons
    <ul>
      <li>Yes/no questions are easier to formally study</li>
    </ul>
  </li>
  <li class="fragment">We’ve seen different <strong>types</strong> of computational problems
    <ul>
      <li class="fragment"><strong>Regular</strong>: solvable by a finite state machine
        <ul>
          <li>Pattern matching problems equivalent to regex</li>
        </ul>
      </li>
      <li class="fragment"><strong>Decidable</strong>: solvable by a Turing machine that always halts (this is the most practical class of problems)</li>
      <li class="fragment"><strong>Turing recognizable</strong>: solvable by a Turing machine that might sometimes get into an infinite loop if its input is not in the language (halting language)</li>
    </ul>
  </li>
</ul>

<h2 id="3-complexity-theory">3. Complexity Theory</h2>
<blockquote>
  <p>The study of the <strong>difficulty</strong> of computational problems</p>
</blockquote>

<hr />

<ul>
  <li class="fragment">How much <strong>CPU time</strong> does it take to solve?</li>
  <li class="fragment">How much <strong>memory</strong> does it take to solve?</li>
</ul>

<h1 id="measuring-complexity-1">Measuring Complexity</h1>

<h2 id="measuring-complexity-2">Measuring Complexity</h2>
<p>Consider the language:
<code class="language-plaintext highlighter-rouge">$$P = \{w\in\Sigma^\ast \mid w\text{ is a palindrome}\}$$</code></p>

<hr />

<ul>
  <li class="fragment">How much time does it take to <strong>decide</strong> <code class="language-plaintext highlighter-rouge">$P$</code>?
    <ul>
      <li>Think about how you would write this in <strong>Python</strong></li>
    </ul>
  </li>
</ul>

<h2 id="measuring-complexity-3">Measuring Complexity</h2>
<p><code class="language-plaintext highlighter-rouge">$$P = \{w\in\Sigma^\ast \mid w\text{ is a palindrome}\}$$</code></p>

<hr />

<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">is_palindrome</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">w</span><span class="p">)):</span>
        <span class="n">front</span><span class="p">,</span> <span class="n">back</span> <span class="o">=</span> <span class="n">w</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">w</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="o">-</span><span class="n">i</span><span class="p">]</span>
        <span class="k">if</span> <span class="n">front</span> <span class="o">!=</span> <span class="n">back</span><span class="p">:</span>
            <span class="k">return</span> <span class="bp">False</span>
    <span class="k">return</span> <span class="bp">True</span>
</code></pre></div></div>

<ul>
  <li class="fragment">If we were to write this in Sipser’s style:</li>
</ul>

<div class="language-plaintext fragment highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = "On input w,
  1. Continue the following until all characters are marked
     a. Compare the first and last unmarked characters in w
        by scanning across the tape from left to right
     b. If they are the same, mark them, move the tape head
        back to the left, and continue
     c. If they are different, REJECT
  2. After all characters have been marked, ACCEPT
</code></pre></div></div>

<h2 id="measuring-complexity-4">Measuring Complexity</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = "On input w,
  1. Continue the following until all characters are marked
     a. Compare the first and last unmarked characters in w
        by scanning across the tape from left to right
     b. If they are the same, mark them, move the tape head
        back to the left, and continue
     c. If they are different, REJECT
  2. After all characters have been marked, ACCEPT
</code></pre></div></div>

<ul>
  <li><strong>3-2-1-GO</strong>: How do we <strong>measure the efficiency</strong> of this algorithm?
    <ul>
      <li class="fragment">Count how many steps it takes on its inputs</li>
    </ul>
  </li>
</ul>

<h2 id="time-complexity">Time Complexity</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$M$</code> be a Turing machine that always halts</li>
  <li class="fragment">The (<strong>worst-case</strong>) <strong>running time</strong> of <code class="language-plaintext highlighter-rouge">$M$</code> is the function <code class="language-plaintext highlighter-rouge">$f:\mathbb{N}\rightarrow\mathbb{N}$</code> where <code class="language-plaintext highlighter-rouge">$f(n)$</code> is the <em>maximum</em> number of steps <code class="language-plaintext highlighter-rouge">$M$</code> takes on an input of length <code class="language-plaintext highlighter-rouge">$n$</code></li>
  <li class="fragment">It is also useful to analyze the <strong>average case running time</strong>, but for now we will focus on the worst-case</li>
</ul>

<h2 id="time-complexity-1">Time Complexity</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = "On input w,
  1. Continue the following until all characters are marked
     a. Compare the first and last unmarked characters in w
        by scanning across the tape from left to right
     b. If they are the same, mark them, move the tape head
        back to the left, and continue
     c. If they are different, REJECT
  2. After all characters have been marked, ACCEPT
</code></pre></div></div>

<hr />

<ul>
  <li><strong>3-2-1-GO:</strong> What is the <strong>best case</strong> running time of the above Turing machine?
    <ul>
      <li class="fragment">On an input of length <code class="language-plaintext highlighter-rouge">$n$</code>, it is possible for <code class="language-plaintext highlighter-rouge">$M$</code> to REJECT in roughly <code class="language-plaintext highlighter-rouge">$n$</code> steps if it finds the first and last characters do not match</li>
    </ul>
  </li>
</ul>

<h2 id="time-complexity-2">Time Complexity</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M = "On input w,
  1. Continue the following until all characters are marked
     a. Compare the first and last unmarked characters in w
        by scanning across the tape from left to right
     b. If they are the same, mark them, move the tape head
        back to the left, and continue
     c. If they are different, REJECT
  2. After all characters have been marked, ACCEPT
</code></pre></div></div>

<hr />

<ul>
  <li><strong>3-2-1-GO:</strong> What is the <strong>worst case</strong> running time of the above Turing machine?
    <ul>
      <li class="fragment">If the string is a palindrome, the first loop takes roughly <code class="language-plaintext highlighter-rouge">$2n$</code> steps, then <code class="language-plaintext highlighter-rouge">$2n-2$</code>, then <code class="language-plaintext highlighter-rouge">$2n-4$</code>, … which is approximately <code class="language-plaintext highlighter-rouge">$2\sum_{i=0}^n i = n(n+1)$</code></li>
    </ul>
  </li>
</ul>

<h2 id="time-complexity-3">Time Complexity</h2>
<ul>
  <li>Does that seem slow to you? Why is it so slow?</li>
  <li class="fragment">For comparison, how many steps does our original Python program take?</li>
</ul>

<div class="language-py fragment highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">is_palindrome</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">w</span><span class="p">)):</span>
        <span class="n">front</span><span class="p">,</span> <span class="n">back</span> <span class="o">=</span> <span class="n">w</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">w</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="o">-</span><span class="n">i</span><span class="p">]</span>
        <span class="k">if</span> <span class="n">front</span> <span class="o">!=</span> <span class="n">back</span><span class="p">:</span>
            <span class="k">return</span> <span class="bp">False</span>
    <span class="k">return</span> <span class="bp">True</span>
</code></pre></div></div>

<ul>
  <li class="fragment">Outer loop executes <code class="language-plaintext highlighter-rouge">$n$</code> times, inner loop roughly takes <code class="language-plaintext highlighter-rouge">$4$</code> steps, so approximately <code class="language-plaintext highlighter-rouge">$4n + 1$</code> steps total
    <ul>
      <li class="fragment">Above we are assuming that looking up the value <code class="language-plaintext highlighter-rouge">$w[i]$</code> takes a single step—but our Turing machine needs to do a linear scan until it finds the <code class="language-plaintext highlighter-rouge">$i$</code>th cell</li>
    </ul>
  </li>
</ul>

<h2 id="complexity-is-model-dependent">Complexity is Model-Dependent</h2>
<ul>
  <li>Recall that the <em>Church-Turing thesis</em> states that:
    <ul>
      <li>All reasonable models of computation are “equivalent” to the Turing machine</li>
    </ul>
  </li>
  <li class="fragment"><strong>3-2-1-GO</strong>: Did we just find a contradiction to the Church-Turing thesis?</li>
  <li class="fragment">No—the Church-Turing thesis is about <strong>computability</strong> (unbounded resources)</li>
</ul>

<h2 id="complexity-is-model-dependent-1">Complexity is Model-Dependent</h2>
<ul>
  <li><strong>Important Take-Away</strong>: Adding extra features to a computational model might affect the time complexity of solving problems</li>
  <li class="fragment"><strong>3-2-1-GO:</strong> If time-complexity depends on the features of the computational model, which model do we use?</li>
  <li class="fragment">It turns out to be useful to study <strong>multiple</strong> models and understand the relationships between them</li>
</ul>

<h1 id="asymptotic-analysis">Asymptotic Analysis</h1>

<h2 id="asymptotic-analysis-1">Asymptotic Analysis</h2>
<ul>
  <li>Remember that <code class="language-plaintext highlighter-rouge">is_palindrome</code> took approximately <code class="language-plaintext highlighter-rouge">4n+1</code> steps in Python?</li>
  <li class="fragment">Calculating the number of steps <strong>exactly</strong> is really tedious—especially for more complicated algorithms</li>
  <li class="fragment">For large inputs, the running time is always dominated by the <strong>highest order term</strong>, so we tend to worry about it the most (e.g. the “<code class="language-plaintext highlighter-rouge">n</code>” in the  <code class="language-plaintext highlighter-rouge">4n + 1</code>)</li>
</ul>

<h2 id="asymptotic-analysis-2">Asymptotic Analysis</h2>
<ul>
  <li><strong>3-2-1-GO:</strong> What is the highest-order term in each of the following expressions?
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$7n^2 + 100n + 17$</code>
        <ul>
          <li class="fragment"><code class="language-plaintext highlighter-rouge">$n^2$</code></li>
        </ul>
      </li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$50\log_{10}(n) + n\log_{2}(n) + 400\log_{10}(n)^2$</code>
        <ul>
          <li class="fragment"><code class="language-plaintext highlighter-rouge">$n\log(n)$</code></li>
        </ul>
      </li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$n^{1000} + 2^n + 10$</code>
        <ul>
          <li class="fragment"><code class="language-plaintext highlighter-rouge">$2^n$</code></li>
        </ul>
      </li>
    </ul>
  </li>
</ul>

<h2 id="big-o-notation">Big-O Notation</h2>
<ul>
  <li>For each of these functions, we often write:
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$7n^2 + 100n + 17 = O(n^2)$</code></li>
      <li><code class="language-plaintext highlighter-rouge">$n^{1000} + 2^n + 10 = O(2^n)$</code></li>
    </ul>
  </li>
</ul>

<hr />

<ul>
  <li><code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> for two functions <code class="language-plaintext highlighter-rouge">$f,g:\mathbb{N}\rightarrow\mathbb{R}^+$</code> if:
    <ul>
      <li class="fragment"><strong>THERE EXIST</strong> positive integers <code class="language-plaintext highlighter-rouge">$c$</code> and <code class="language-plaintext highlighter-rouge">$n_0$</code> such that:
        <ul>
          <li class="fragment"><strong>FOR EVERY</strong> integer <code class="language-plaintext highlighter-rouge">$n \ge n_0$</code> we have:
            <ul>
              <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) \le c\cdot g(n)$</code></li>
            </ul>
          </li>
        </ul>
      </li>
    </ul>
  </li>
</ul>

<h2 id="big-o-notation-1">Big-O Notation</h2>
<ul>
  <li>What does <code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> intuitively mean?</li>
  <li class="fragment">That <strong>eventually</strong> <code class="language-plaintext highlighter-rouge">$g(n)$</code> will grow <strong>at least as fast</strong> as <code class="language-plaintext highlighter-rouge">$f(n)$</code></li>
</ul>

<hr />

<p class="fragment">Image by Victor S.Adamchik, CMU, 2009 at <a href="cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html">source</a></p>

<h2 id="big-o-exercises">Big-O Exercises</h2>
<ul>
  <li><strong>3-2-1-GO</strong>: Is <code class="language-plaintext highlighter-rouge">$f(n) = O(g(n))$</code> in the following cases?
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n^2 + 4n + 1$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n^2$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n\log(n)$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = 2^{2n}$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = 2^n$</code></li>
    </ul>
  </li>
</ul>

<h2 id="little-o-notation">Little-O Notation</h2>
<ul>
  <li>Another useful definition is “little-o”</li>
  <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))$</code> for two functions <code class="language-plaintext highlighter-rouge">$f,g:\mathbb{N}\rightarrow\mathbb{R}^+$</code> if:</li>
</ul>

<p class="fragment"><code class="language-plaintext highlighter-rouge">$$\lim_{n\rightarrow\infty}\frac{f(n)}{g(n)} = 0$$</code></p>

<ul>
  <li class="fragment">We say that “<code class="language-plaintext highlighter-rouge">$f$</code> is little-o of <code class="language-plaintext highlighter-rouge">$g$</code>” and that <code class="language-plaintext highlighter-rouge">$g$</code> <strong>dominates</strong> <code class="language-plaintext highlighter-rouge">$f$</code></li>
</ul>

<h2 id="little-o-exercises">Little-O Exercises</h2>
<ul>
  <li><strong>3-2-1-GO</strong>: Is <code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))$</code> in the following cases?
    <ul>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n^2 + 4n + 1$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n^2$</code></li>
      <li class="fragment"><code class="language-plaintext highlighter-rouge">$f(n) = n$</code><br /><code class="language-plaintext highlighter-rouge">$g(n) = n\log(n)$</code></li>
    </ul>
  </li>
</ul>

<h2 id="little-o-properties">Little-O Properties</h2>
<ul>
  <li>Notice that <code class="language-plaintext highlighter-rouge">$n^2 = O(n^2)$</code> but <code class="language-plaintext highlighter-rouge">$n \ne o(n)$</code></li>
  <li class="fragment">Big-O is similar to <code class="language-plaintext highlighter-rouge">$\le$</code>
    <ul>
      <li>Functions that grow at <strong>similar rates</strong> are Big-O of one-another</li>
    </ul>
  </li>
  <li class="fragment">Little-O is similar to <code class="language-plaintext highlighter-rouge">$<$</code>
    <ul>
      <li><code class="language-plaintext highlighter-rouge">$g(n)$</code> must grow <strong>strictly faster</strong> than <code class="language-plaintext highlighter-rouge">$f(n)$</code></li>
    </ul>
  </li>
</ul>

<hr />

<ul>
  <li class="fragment"><strong>Fact:</strong> <code class="language-plaintext highlighter-rouge">$f(n) = o(g(n))\implies f(n) = O(g(n))$</code></li>
</ul>

<h1 id="time-complexity-4">Time Complexity</h1>

<h2 id="time-complexity-5">Time Complexity</h2>
<ul>
  <li>Let <code class="language-plaintext highlighter-rouge">$t:\mathbb{N}\rightarrow\mathbb{R}^+$</code> be a function</li>
  <li class="fragment">Then <code class="language-plaintext highlighter-rouge">$\text{TIME}(t(n))$</code> is the collection of languages that are decidable by a single-tape deterministic Turing machine in <code class="language-plaintext highlighter-rouge">$O(t(n)$</code> time</li>
  <li class="fragment">We call <code class="language-plaintext highlighter-rouge">$\text{TIME}(t(n))$</code> a <strong>time complexity class</strong></li>
</ul>

<h2 id="time-complexity-6">Time Complexity</h2>
<ul>
  <li>Previously we showed that the language of palindromes is decidable in <code class="language-plaintext highlighter-rouge">$O(n^2)</code> steps; this means that <code class="language-plaintext highlighter-rouge">$P\in\text{TIME}(n^2)$</code></li>
</ul>

<h2 id="polynomial-time-complexity">Polynomial Time Complexity</h2>
<ul>
  <li>An especially useful complexity class is the class <code class="language-plaintext highlighter-rouge">$P$</code> of languages decidable in <strong>polynomial time</strong></li>
</ul>

<p><code class="language-plaintext highlighter-rouge">$$P = \bigcup_{k\ge 1} \text{TIME}(n^k)$$</code></p>

<ul>
  <li class="fragment">The complexity class <code class="language-plaintext highlighter-rouge">$P$</code> is <strong>robust</strong> and is the same set of languages for all reasonable computational models</li>
</ul>