Lecture 02

Lecture Outline

2.1 Computational Tractability 2.2 Asymptotic Order of Growth 2.4 A Survey of Common Running Times Extra slides on Analysis of Algorithms

Polynomial-Time Brute force. For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution. Typically takes 2Ntime or worse for inputs of size N. Unacceptable in practice. n ! for stable matching But they are intuitive. with n men and n women „

„

„

Desirable scaling property. When the input size doubles, the algorithm should only slow down by some constant factor C. There exists constants c > 0 and d > 0 such that on every input of size N, its running time is bounded by c Nd steps.

Def. An algorithm is poly-time if the above scaling property holds. choose C = 2d observe the difference between c and C

2

Worst-Case Analysis Worst case running time. Obtain bound on largest possible running time of algorithm on input of a given size N. Generally captures efficiency in practice. Draconian view, but hard to find effective alternative. „

„

Too severe

Average case running time. Obtain bound on running time of algorithm on random input as a function of input size N. Hard (or impossible) to accurately model real instances by random distributions. Algorithm tuned for a certain distribution may perform poorly on other inputs. „

„

3

Worst-Case Polynomial-Time Def. An algorithm is theoretically efficient if its running time is polynomial. Justification: It really works in practice! Although 6.02 × 1023 × N20 is technically poly-time, it would be useless in practice. In practice, the poly-time algorithms that people develop almost always have low constants and low exponents. Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem. „

„

„

Exceptions. Some poly-time algorithms do have high constants and/or exponents, and are useless in practice. Some exponential-time (or worse) algorithms are widely used because the worst-case instances seem to be rare. „

„

simplex method Unix grep 4

Why It Matters

5

Asymptotic Order of Growth Upper bounds. T(n) is O(f(n)) if there exist constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 we have T(n) ≤ c · f(n). Lower bounds. T(n) is Ω(f(n)) if there exist constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 we have T(n) ≥ c · f(n). Tight bounds. T(n) is Θ(f(n)) if T(n) is both O(f(n)) and Ω(f(n)). Ex: T(n) = 32n2 + 17n + 32. T(n) is O(n2), O(n3), Ω(n2), Ω(n), and Θ(n2) . T(n) is not O(n), Ω(n3), Θ(n), or Θ(n3). „

„

6

Asymptotic Order of Growth in Plain English Big family. O(g(n)): class of functions f(n) that grow no faster than g(n). Θ (g(n)): class of functions f(n) that grow at same rate as g(n). Ω(g(n)): class of functions f(n) that grow at least as fast as g(n). Small family. o(g(n)): class of functions f(n) that grow slower than g(n). ω(g(n)): class of functions f(n) that grow faster than g(n).

7

Notation Slight abuse of notation. T(n) = O(f(n)). Not transitive: – f(n) = 5n3; g(n) = 3n2 – f(n) = O(n3) = g(n) – but f(n) ≠ g(n). Better notation: T(n) ∈ O(f(n)). „

„

Meaningless statement. Any comparison-based sorting algorithm requires at least O(n log n) comparisons. Statement doesn't "type-check." Use Ω for lower bounds. O notation is for upper bounds. „

„

8

Properties Transitivity. If f = O(g) and g = O(h) then f = O(h). If f = Ω(g) and g = Ω(h) then f = Ω(h). If f = Θ(g) and g = Θ(h) then f = Θ(h). „

„

„

Additivity. If f = O(h) and g = O(h) then f + g = O(h). If f = Ω(h) and g = Ω(h) then f + g = Ω(h). If f = Θ(h) and g = O(h) then f + g = Θ(h). „

„

„

Multiplicity If f = O(h) and g = O(i) then f × g = O(h × i) If f = Ω(h) and g = Ω(i) then f × g = Ω(h × i) If f = Θ(h) and g = Θ(i) then f × g = Θ(h × i) „

„

„

9

Asymptotic Bounds for Some Common Functions Polynomials. a0 + a1n + … + adnd is Θ(nd) if ad > 0. Polynomial time. Running time is O(nd) for some constant d independent of the input size n.

Logarithms. O(log a n) = O(log b n) for any constants a, b > 0. can avoid specifying the base

Logarithms. For every x > 0, log n = O(nx). log grows slower than every polynomial

Exponentials. For every r > 1 and every d > 0, nd = O(rn). every exponential grows faster than every polynomial

10

Linear Time: O(n) Linear time. Running time is at most a constant factor times the size of the input. Computing the maximum. Compute maximum of n numbers a1, …, an. max ← a1 for i = 2 to n { if (ai > max) max ← ai }

11

Linear Time: O(n) Merge. Combine two sorted lists A = a1,a2,…,an with B = b1,b2,…,bn into sorted whole.

i = 1, j = 1 while (both lists are nonempty) { if (ai ≤ bj) append ai to output list and increment i else(ai ≤ bj)append bj to output list and increment j } append remainder of nonempty list to output list

Claim. Merging two lists of size n takes O(n) time. Pf. After each comparison, the length of output list increases by 1. 12

O(n log n) Time O(n log n) time. Arises in divide-and-conquer algorithms. also referred to as linearithmic time

Sorting. Mergesort and heapsort are sorting algorithms that perform O(n log n) comparisons. Largest empty interval. Given n time-stamps x1, …, xn on which copies of a file arrive at a server, what is largest interval of time when no copies of the file arrive? O(n log n) solution. Sort the time-stamps. Scan the sorted list in order, identifying the maximum gap between successive time-stamps.

13

Quadratic Time: O(n2) Quadratic time. Enumerate all pairs of elements. Closest pair of points. Given a list of n points in the plane (x1, y1), …, (xn, yn), find the pair that is closest. O(n2) solution. Try all pairs of points. min ← (x1 - x2)2 + (y1 - y2)2 for i = 1 to n { for j = i+1 to n { d ← (xi - xj)2 + (yi - yj)2 if (d < min) min ← d } }

don't need to take square roots

Remark. Ω(n2) seems inevitable, but this is just an illusion.

see chapter 5

14

Cubic Time: O(n3) Cubic time. Enumerate all triples of elements. Set disjointness. Given n sets S1, …, Sn each of which is a subset of 1, 2, …, n, is there some pair of these which are disjoint? O(n3) solution. For each pairs of sets, determine if they are disjoint.

foreach set Si { foreach other set Sj { foreach element p of Si { determine whether p also belongs to Sj } if (no element of Si belongs to Sj) report that Si and Sj are disjoint } }

15

Polynomial Time: O(nk) Time Independent set of size k. Given a graph, are there k nodes such that no two are joined by an edge? k is a constant

O(nk) solution. Enumerate all subsets of k nodes. foreach subset S of k nodes { check whether S in an independent set if (S is an independent set) report S is an independent set } }

„

„

„

Check whether S is an independent set = O(k2). Number of k element subsets = ⎛ n ⎞ n(n − 1)(n − 2)L(n − k + 1) n k ⎜⎜ ⎟⎟= ≤ O(k2 nk / k!) = O(nk). k(k − 1)(k − 2)L(2)(1) k! ⎝k⎠ poly-time for k=17, but not practical

16

Exponential Time Independent set. Given a graph, what is maximum size of an independent set? O(n2 2n) solution. Enumerate all subsets. S* ← φ foreach subset S of nodes { check whether S in an independent set if (S is largest independent set seen so far) update S* ← S } }

17

Analysis of Algorithms Issues Correctness. Most important, algorithm must be correct otherwise it is useless. Time efficiency. How much time requires to run the algorithm. Space efficiency. The extra space the algorithm requires. – Both time and space were premium resources in the early days. At the present, the space issue is not of as much concern. The memory is cheap. Optimality. (In theory) no algorithm can be more efficient. „

„

„

„

Approaches Theoretical analysis. Mathematical analysis. Empirical analysis. Through experiments. „

„

18

Empirical Analysis of Time Efficiency Consider a program as a black block Select a specific (typical) sample of inputs Use physical unit of time (e.g., milliseconds) – Depend on other factors such as computer architecture, compiler optimization. „

„

OR Instrument a program with some counting mechanism Profiling Count actual number of basic operations (or sample number of basic ops) Analyze the empirical data „

„

„

19

Best-case, Average-case, Worst-case For some algorithms, the efficiency depends on type of input. Worst case. Worst(n) – maximum over inputs of size n Best case. Best(n) – minimum over inputs of size n Average case. Avg(n) – average over inputs of size n NOT the average of worst and best case, even they can be similar at times. Avg(n) ≠ (Worst(n)+Best(n))/2 Considered over the distribution of all possible inputs of size n. „

„

Amortized efficiency. Average over a sequence of operations performed on the same data structure. Same operations along the sequence require a different amount of time. „

20

Example: Sequential Search Problem. Given a list of n elements and a search key K, find an element equal to K, if any. Algorithm. Scan the list and compare its successive elements with K until either a matching element is found (successful search) or the list is exhausted (unsuccessful search). Worst case? Search all elements, no match found. Best case? Match at the first comparison. Average case? Depends on how often the search key matches the list.

21

Recurrence Relations Running time can be expressed as a function of n. x(n) = 2n for n ≥ 0 The generic terms (n) can be expressed as one or more of other terms (with lower index number) of the function, with one or more explicit values of the first terms. x(n) = x(n-1) + n x(0) = 0

for n > 0

initial condition

This type of definition is called recurrence equation, recurrence relation.

22

Methods for Solving Recurrences Method of forward substitutions. Start with the initial term(s), use recurrence equation to generate few more terms and analyze for the pattern, then prove it using mathematical induction. Method of backward substitutions. Start with the generic term, substitute x(n-1) on the right side of recurrence equation with equivalent x(n-2) term. Repeat the substitution and observe for the pattern. Rewrite the recurrence so that x(n) is a function of x(n-i) for any i, then select i to make n-i reach the initial condition. Guess and substitute. Guess the solution of the recurrence, substitute the guessing term. Then show that it is correct.

23

Example1: Recursive Evaluation of n ! Def. n ! = 1*2*…*(n-1)*n recursive definition of n!

Factorial(n) { if (n=0) fac = 1; else fac = Factorial(n-1) * n return fac; }

Recurrence for number of multiplications, M(), to compute n!. Using method of backward substitutions: (for n > 0) M(n) = M(n-1) + 1 substitute M(n-1) with M(n-2)+1 = [M(n-2) + 1] +1 = M(n-2) + 2 substitute M(n-2) with M(n-3)+1 = [M(n-3) + 1] +1 +1 = M(n-3) + 3 = M(n-i) + i = … = M(n-n) + n = n. 24

Example 2: Tower of Hanoi Tower of Hanoi Def. There are n disks of different sizes and 3 pegs Initially, all disks are on the first peg in order of size, the largest on the bottom, the smallest on top. Goal. Move all disks to the third peg using second one as an auxiliary, if needed. Constraints. One disk can be move at a time A larger disk cannot be on top of a smaller one at any time. „

„

25

Recurrence of Tower of Hanoi Let M(n) denote number of moves M(n) = M(n-1) + 1 + M(n-1) M(1) = 1.

for n > 1.

Using method of backward substitutions M(n) = 2M(n-1) + 1 substitute M(n-1) with 2M(n - 2) + 1 = 2[2M(n-2) + 1] + 1 substitute M(n-2) with 2M(n - 3) + 1 = 22M(n-2) + 2 + 1 = 22[2M(n-3) + 1] + 2 + 1 = 23M(n-3) + 22 + 2 + 1 = 2iM(n-i) + 2i-1 + … + 22 + 2 + 1 = 2iM(n-i) + 2i – 1 = 2n-1M(1) + 2n-1 – 1 = 2n-1 + 2n-1 – 1 = 2n -1.

26

Master Theorem Let T(n) be an eventually nondecreasing function. T(n) = aT(n/b) + f(n) for n= bk, k = 1, 2, … T(1) = c, where a ≥ 1, b ≥ 2, c ≥ 0. If f(n) ∈ Θ(nd), d ≥ 0 then T(n) ∈ Θ(nd) T(n) ∈ Θ(nd lg n ) T(n) ∈ Θ(nlog b a)

Case 1. a < bd Case 2. a = bd Case 3. a > bd

The same results hold for O, Ω too.

27

Idea of Master Theorem

a

f(n/b)

Recursion tree

Work for a level

f(n)

f(n)

f(n/b)

a ...

f(n/b)

a f(n/b)

h =logb(n) f(n/b2)

T(1)

f(n/b2)

...

f(n/b2)

# leaves = ah = alog_b n = nlog_b a

a2 f(n/b2)

# leaves * T(1)

28

Idea of Master Theorem Case 1. The weight decreases geometrically from the root to leaves. The root holds a constant fraction of the total weight.

Θ(nd ) Case 2. The weight is approximately the same on each of the logbn levels.

Θ(nd lg n) Case 3. The weight increases geometrically from the root to leaves. The leaves hold a constant fraction of the total weight.

Θ(nlogb a )

29

Master Theorem Examples Example 1. T(n) = T(n/2) + n T(1) = 1 Î

a = 1, b = 2, c = 1, f(n) = n, d = 1.

Match case 1, therefore T(n) = Θ(n1) = Θ(n) = O(n) = Ω(n).

Example 2. T(n) = 2T(n/2) + n T(1) = 1 Î

a = 2, b = 2, c = 1, f(n) = n, d = 1.

Match case 2, therefore T(n) = Θ(n lg(n)).

Example 3. T(n) = 7T(n/4) + n T(1) = 1 Î

a = 7, b = 4, c = 1, f(n) = n, d = 1.

Match case 3, therefore T(n) = Θ(nlog4 7)

30

Recursion as Mathematical Induction Recursion is mathematical induction Example.

n

∑i = i =1

n(n + 1) 2

for all n ≥ 1

We need to prove.

1

∑i = i =1

1(1+ 1) ; 2

2

∑i = i =1

2(2 + 1) ; 2

3

∑i = i =1

n 3(3 + 1) n(n + 1) ; L;∑i = 2 2 i =1

31

Mathematical Induction 1

∑i =

Basis step, prove.

i=1

1(1+ 1) ⇒1= 1 2

Inductive step (weak induction) Assume the nth term is true and show the (n+1)th term is true. „

n

∑i =

Assume.

i =1

n+1

∑i =

Show.

i=1

n(n + 1) 2

(n + 1)(n + 2) 2 ⎛ n ⎞ i = ⎜ ∑i ⎟ + (n + 1) ∑ i =1 ⎝ i =1 ⎠ n+1

Rewrite

(n+1)th

term in form of

nth

term

Use the inductive assumption n+1

∑i = i =1

n(n + 1) (n + 1)(n + 2) (n + 1)((n + 1) + 1) + (n + 1) = = 2 2 2 32

Recursion Tree Visualize a recurrence T(n) = 2T(n/2) + n2

T(n)

n2

T(n/2)

T(n/2)

33

Recursion Tree (cont.) Recursion tree

Work for a level n2

n2

(n/2)2

T(n/4)

T(n/4) T(n/4)

Values decrease geometrically

(1/2)n2

(n/2)2

T(n/4)

Total work

(1/4)n2

Θ(n2)

The top level dominates the run time. Total run time is a constant factor of the top term. 34

Recursion Tree (cont.) Recursion tree

Work for a level n2

n2

(n/4)2

T(n/16)

(n/2)2

T(n/8) T(n/8)

T(n/4)

(5/16)n2

(25/256)n2

Total work = n2(1+5/16+(5/16)2+…) = Θ(n2) T(n) = T(n/4) + T(n/2) + n2. Different recurrence but the same complexity. 35

Recursion Tree (cont.)

T(n) = T(n/3) + T(2n/3) + n

Recursion tree

Work for a level

n

n

n/3

2n/3

n

36

Recursion Tree (cont.)

Recursion tree

Work for a level n

n

n/3

log3/2(n)

T(n/9)

n

2n/3

T(2n/9) T(2n/9)

T(4n/9)

Total work

n

O(n lg n) 37