Big-O, Big-Omega, Big-Theta
CS205 Data Structures
Use arrow keys or buttons to navigate | Press S to reveal steps
The same problem can be solved in vastly different amounts of time depending on the algorithm chosen.
Consider searching for a name in a phone book:
Imagine sorting 1 million exam papers. A bad algorithm could take years. A good one finishes in seconds.
| Complexity | Operations | At 1 GHz |
|---|---|---|
| O(n) | 1,000 | 1 μs |
| O(n log n) | ~10,000 | 10 μs |
| O(n²) | 1,000,000 | 1 ms |
| O(n³) | 10&sup9; | 1 sec |
| O(2ⁿ) | ~10³&sup0;¹ | Heat death of universe |
Algorithm analysis lets us predict performance before we run the code, and compare algorithms independently of hardware.
An algorithm is a step-by-step procedure for solving a problem in a finite number of steps.
We use pseudocode (not any specific language) so the analysis is language-independent.
We analyze the algorithm, not the program. The algorithm is the idea; the program is just one implementation of it.
"My laptop ran it in 2 seconds" tells us almost nothing about the algorithm itself.
Instead of timing, we count primitive operations as a function of input size n.
Each primitive operation takes constant time — we call it O(1).
Let's carefully count every operation in findMax:
| Operation | Count |
|---|---|
| Initialization | 2 |
| Loop control | 1 + 2(n-1) |
| Comparison in loop | 2(n-1) |
| Assignment in loop (worst) | 2(n-1) |
| Return | 1 |
| Best case total | 2 + 3(n-1) + 1 = 3n |
| Worst case total | 2 + 5(n-1) + 1 = 5n - 2 |
Both best and worst case are proportional to n. The exact constants (3 vs 5) don't matter for classification — it's O(n) either way.
We care about how running time grows as input size n increases, not the exact count.
If you're driving 1000 miles, it doesn't matter if you start 5 feet ahead. At large scale, the dominant term determines everything.
Growth rate analysis lets us focus on the big picture — how algorithms scale, not how they perform on tiny inputs.
f(n) is O(g(n)) if there exist positive constants c and n0 such that:
f(n) ≤ c · g(n) for all n ≥ n0
Big-O is like saying "this trip will take at most 3 hours." It might take less, but never more (past a certain point).
Therefore, 3n + 5 is O(n) with c=4, n0=5.
Therefore, n² + 3n is O(n²) with c=2, n0=3.
The choice of c and n0 is not unique — many valid pairs exist. You only need to find one pair that works. Also, 3n + 5 is technically O(n²) too, but that's a loose (less useful) bound.
f(n) is Ω(g(n)) if there exist positive constants c and n0 such that:
f(n) ≥ c · g(n) for all n ≥ n0
3n + 5 is Ω(n): choose c=3, n0=1.
3n + 5 ≥ 3n for all n ≥ 1. ✓
Big-Omega is like saying "this trip takes at least 1 hour." It might take more, but never less.
f(n) is Θ(g(n)) if there exist positive constants c1, c2, and n0 such that:
c1·g(n) ≤ f(n) ≤ c2·g(n)
for all n ≥ n0
f(n) is Θ(g(n)) if and only if:
3n + 5 is Θ(n):
Θ gives the exact growth rate. When someone says "this algorithm is n log n," they really mean Θ(n log n). It's the most precise characterization.
Listed from fastest to slowest growing (best to worst for runtime):
| Name | Big-O | n = 10 | n = 100 | n = 1,000 | Example |
|---|---|---|---|---|---|
| Constant | O(1) | 1 | 1 | 1 | Array index |
| Logarithmic | O(log n) | 3 | 7 | 10 | Binary search |
| Linear | O(n) | 10 | 100 | 1,000 | Linear search |
| Linearithmic | O(n log n) | 30 | 700 | 10,000 | Merge sort |
| Quadratic | O(n²) | 100 | 10,000 | 1,000,000 | Bubble sort |
| Cubic | O(n³) | 1,000 | 1,000,000 | 10&sup9; | Matrix multiply |
| Exponential | O(2ⁿ) | 1,024 | ~10³&sup0; | ~10³&sup0;¹ | Subsets |
| Factorial | O(n!) | 3,628,800 | ~10¹&sup5;&sup8; | LOL | Permutations |
Anything beyond O(n²) becomes impractical very quickly. O(2ⁿ) and O(n!) are essentially unsolvable for n > 30–50.
Multiply for nested loops. Halving or doubling the loop variable means logarithmic. A loop running n times with a O(log n) body gives O(n log n).
Recursive algorithms are analyzed using recurrence relations.
Write the recurrence → expand it → find the pattern → determine the total.
| Case | When? | Comparisons |
|---|---|---|
| Best | Target is first | 1 = O(1) |
| Worst | Target is last / absent | n = O(n) |
| Average | Target equally likely anywhere | n/2 = O(n) |
We usually analyze the worst case because:
Don't confuse Big-O (a bound on a function) with worst case (a type of input). You can talk about the Big-O of the best case, worst case, or average case.
Sometimes a single operation is expensive, but averaged over many operations, the cost is low.
When the array is full, we double its capacity and copy everything.
Individual insert can be O(n) in the worst case. But the amortized cost per insert is O(1). "Expensive operations are rare enough that they average out."
Think of it like paying rent. You save a little each day, and once a month you pay a big lump sum. On average, your daily spending is still constant.
Algorithm analysis isn't just about time — memory usage matters too.
Use only O(1) extra space — they modify the input directly.
| Algorithm | Time | Space (aux) |
|---|---|---|
| Linear search | O(n) | O(1) |
| Binary search (iterative) | O(log n) | O(1) |
| Binary search (recursive) | O(log n) | O(log n)* |
| Merge sort | O(n log n) | O(n) |
| Insertion sort | O(n²) | O(1) |
| Copy array | O(n) | O(n) |
* Recursive calls use stack space
Recursion uses stack space! Each recursive call adds a frame. A recursion depth of n means O(n) space, even if no arrays are allocated.
| Algorithm | Best | Average | Worst |
|---|---|---|---|
| Bubble Sort | O(n) | O(n²) | O(n²) |
| Insertion Sort | O(n) | O(n²) | O(n²) |
| Selection Sort | O(n²) | O(n²) | O(n²) |
| Merge Sort | O(n log n) | O(n log n) | O(n log n) |
| Quick Sort | O(n log n) | O(n log n) | O(n²) |
Comparison-based sorting has a proven lower bound of Ω(n log n). No comparison sort can do better in the worst case.
| Algorithm | Time | Requires |
|---|---|---|
| Linear Search | O(n) | Nothing |
| Binary Search | O(log n) | Sorted array |
| Hash Table Lookup | O(1) avg | Hash table |
| Algorithm | Time |
|---|---|
| BFS / DFS | O(V + E) |
| Dijkstra's | O((V+E) log V) |
V = vertices, E = edges. You'll learn these later in the course!
Saying "linear search is O(n²)" is technically true (O is an upper bound), but misleading. The tight bound is Θ(n). Always give the tightest bound you can.
Big-O ignores constants, but in the real world, an O(n) algorithm with a huge constant can be slower than O(n²) for reasonable inputs.
In Big-O, log base is irrelevant because:
Best/worst/average case = which input.
O / Ω / Θ = type of bound on the function.
You can say: "The worst case is Θ(n²)" or "The best case is O(n)." These are separate concepts!
Rules of thumb for fast analysis:
Single loop over n? → O(n)
Nested loop? → O(n²)
Halving? → O(log n)
Divide and conquer? → Probably O(n log n)
| Notation | Meaning | Bound |
|---|---|---|
| O(g(n)) | f(n) ≤ c·g(n) | Upper |
| Ω(g(n)) | f(n) ≥ c·g(n) | Lower |
| Θ(g(n)) | c&sub1;·g(n) ≤ f(n) ≤ c&sub2;·g(n) | Tight |
Algorithm analysis is about understanding scalability. A good algorithm on slow hardware will always eventually beat a bad algorithm on fast hardware — you just need a large enough input.