Pipelining — Why Modern CPUs Cheat at Fetch-Decode-Execute

You learned the fetch-decode-execute cycle as a clean, sequential loop: fetch one instruction, decode it, execute it, then start the next. That model is true, but it's also the slow, honest version of the story — every real high-performance CPU since the 1980s has been quietly cheating on it, running several instructions' worth of fetch-decode-execute simultaneously, like an assembly line instead of one craftsman building a whole car from scratch before starting the next. This closing lesson is your forward-looking hook into that cheat — pipelining — and the strange new problems it creates.

Learning Objectives

By the end of this lesson, you should be able to:

• explain why pipelining improves instruction throughput but not the latency of one instruction;
• calculate the ideal cycle count for N instructions in a S-stage pipeline;
• identify data, control, and structural hazards;
• explain stalls, bubbles, forwarding, and flushes at a beginner-friendly level.

The Honest, Slow Version

If each stage takes one clock cycle, instruction I2 doesn't even start being fetched until I1 has completely finished. Three instructions cost nine cycles. This is exactly the model from lesson 5 — and it leaves most of the CPU's hardware idle most of the time. While the ALU is busy executing I1, the instruction-fetch circuitry is sitting there, doing nothing, perfectly capable of fetching I2 right now.

The Cheat: Overlap the Stages

A pipeline splits instruction processing into discrete stages — classically five — and lets different instructions occupy different stages in the same clock cycle. Nobody waits for anybody else to fully finish.

The classic 5-stage RISC pipeline:

With five independent stages acting like five workstations on an assembly line, five different instructions can be in flight at once — one being fetched while another is decoded, while a third executes, while a fourth touches memory, while a fifth writes back its result, all in the very same clock cycle.

Watching It Fill Up

title "Pipeline fill, illustrative stage activity"
time start=0 end=6 unit=cycles divisions=6
I1: pulse label="I1 active F-D-E-M-W" low=0 high=1 at=0 width=5 color=#2563eb
I2: pulse label="I2 starts one cycle later" low=0 high=1 at=1 width=5 color=#16a34a
I3: pulse label="I3 starts two cycles later" low=0 high=1 at=2 width=4 color=#9333ea
I4: pulse label="I4 starts three cycles later" low=0 high=1 at=3 width=3 color=#ea580c
marker FULL at=4 label="steady overlap"

The waveform is illustrative rather than a transistor-level simulation. Each high region means the instruction is somewhere in the five-stage pipeline; the stage labels are explained in the table above.

After the pipeline fills (a brief startup delay), the CPU completes roughly one instruction per clock cycle in steady state, instead of one instruction per five cycles. That's the entire point of pipelining: it doesn't make any single instruction finish faster — I1 still takes 5 cycles start to finish — but it dramatically increases throughput, the rate at which instructions complete.

For large N, that's nearly a 5x speedup, purely from keeping idle hardware busy — no faster clock, no smarter ALU, just better scheduling of work that was already there.

Worked Example: Ideal Pipeline Timing

If a pipeline has S stages and runs N independent instructions, the ideal cycle count is:

cycles = S + (N - 1)

For a 5-stage pipeline running 20 independent instructions:

cycles = 5 + (20 - 1) = 24 cycles

A non-pipelined design that takes 5 cycles per instruction would need:

cycles = 5 x 20 = 100 cycles

Ideal speedup:

speedup = 100 / 24 = 4.17x

It is not exactly 5x because the pipeline needs fill time at the beginning and drain time at the end. Hazards reduce the real speedup further.

The Catch: Hazards

Overlapping instructions only works cleanly if they're truly independent. The moment one instruction depends on another, the assembly-line metaphor breaks down, and the pipeline must either stall (insert a bubble — a wasted cycle) or get clever with forwarding. There are three classic categories.

Data hazards

If SUB enters Decode while ADD's result for R1 hasn't reached Writeback yet, SUB would read a stale value of R1. Real pipelines solve much of this with forwarding (routing the ALU's output directly back into the next instruction's Execute stage, bypassing the register file entirely) but some cases still force a stall.

Control hazards (branches)

The pipeline must fetch something for the cycle right after a branch, but it doesn't yet know whether the branch will be taken — that's only resolved once the branch reaches Execute. Guess wrong, and every instruction fetched on the wrong path must be discarded (a "pipeline flush"), costing several wasted cycles. This is exactly why branch prediction exists in real CPUs — a topic for further study beyond this course.

Structural hazards

If instruction and data memory share the same physical bus (recall the tri-state bus arbitration from lesson 9), a load/store instruction in the Memory stage can collide with a Fetch needing the same bus in the same cycle. Harvard-architecture designs sidestep exactly this by giving instruction and data memory separate buses — one of the deepest payoffs of a decision made all the way back in your very first Fundamentals lessons.

Stalls in Action

title "Load-use stall, illustrative bubble"
time start=0 end=6 unit=cycles divisions=6
LDR: pulse label="I1 load in pipeline" low=0 high=1 at=0 width=5 color=#2563eb
USE: pulse label="I2 waits one cycle" low=0 high=1 at=1 width=2 color=#16a34a
USE2: pulse label="I2 resumes after bubble" low=0 high=1 at=4 width=2 color=#16a34a
BUBBLE: pulse label="stall bubble" low=0 high=1 at=3 width=1 color=#dc2626
marker WAIT at=3 label="value not ready"

That flat, repeated -- segment is a bubble — a cycle where the pipeline stage does nothing useful, deliberately, because going forward would compute the wrong answer. Bubbles are the price pipelining pays for hazards; clever CPU design is largely about minimizing how often they're needed.

Why This Is the Right Place to Stop — For Now

Pipelining only makes sense once you've internalized two things this course already gave you: that the fetch-decode-execute cycle is a sequence of distinct hardware stages (lesson 5), and that hardwired control logic is what drives each stage without "thinking" (lesson 4). Pipelining is simply running several copies of that hardwired control logic, staggered in time, against several instructions at once. Everything beyond this — branch prediction, out-of-order execution, superscalar issue, speculative execution, multi-level caching — is built directly on top of the five-stage idea you just saw, and is exactly where a follow-up course on computer architecture would begin.

Key Takeaway

A modern CPU doesn't execute instructions one at a time any more than a factory builds one car at a time on an assembly line — it overlaps stages to turn "one instruction every five cycles" into "nearly one instruction every cycle," at the cost of carefully managing data, control, and structural hazards. You now understand, end to end, the journey from a reset vector forcing a known starting address, through opcode decoding, addressing modes, interrupts, shared buses, and finally the pipelined overlap that makes all of it fast — the complete internal life of a CPU, from power-on to full speed.

Common Mistakes

Further Reading

• Patterson and Hennessy, Computer Organization and Design, pipeline datapath and hazards chapters.
• ARM Cortex-M technical reference manuals, for practical examples of simple embedded pipelines.
• RISC-V unprivileged ISA and educational pipeline materials, for clean examples of five-stage pipeline behavior.

Mind Map