Firmware Hardening — Writing Code That Survives the Real World

Firmware lockup is the most embarrassing certification failure because it has nothing to do with the
hardware. A device that passes every electrical test, sails through radiated emissions, survives the
ESD gun, and then silently hangs after 72 hours of operation — or resets the moment a nearby motor
starts — has a firmware problem, not a hardware problem. And firmware problems are invisible until
deployment, when a customer calls at 2 AM to say the unit is frozen and the conveyor has stopped.

Watchdog Timers — The Last Line of Defense

A watchdog timer is a hardware counter that resets the CPU if firmware does not periodically reset
it (the "kick"). The idea is simple: if the main loop is running correctly, it kicks the watchdog.
If the main loop hangs — deadlock, infinite loop, bus hang, stack corruption — the watchdog
expires and forces a reset. The system comes back alive.

Hardware Watchdog vs Software Watchdog

Hardware watchdog (IWDG on STM32, WDT on AVR, WDOG on Kinetis): clocked by an independent
oscillator (LSI on STM32, typically 32 kHz). Continues running even if the main clock fails. This
is what you want. Enable it. Always.

Software watchdog: a timer interrupt that checks a set of flags. If all flags are set, it
resets them. Each software task sets its own flag. The interrupt can detect that a specific task
has stopped — more granular than a hardware watchdog alone. Use both. The hardware watchdog catches
total lockups; the software watchdog catches a single stuck task.

Window Watchdog (WWDG on STM32)

The standard watchdog only catches "too late" — the kick never arrived. The window watchdog also
catches "too early" — the kick arrived before a minimum time had elapsed. This matters because a
runaway firmware loop can accidentally kick the watchdog on every iteration and the standard
watchdog never fires.

With WWDG, the kick window is, for example, 50–100 ms. A kick at 30 ms fails. A kick at 110 ms
fails. Only a kick between 50 and 100 ms succeeds. This forces the main loop to run at
approximately the correct rate.

Rules for Watchdog Use

NEVER kick the watchdog inside an interrupt service routine.

If you kick inside an ISR, the watchdog is kicked even when the main loop is completely frozen, as
long as any interrupt keeps firing. The timer tick ISR fires every 1 ms — and it will keep firing
while your main loop is stuck in a deadlock. Your watchdog becomes useless.

Enable the watchdog LAST during initialization.

Init sequences are the most likely place for a hang — waiting for a sensor to respond, configuring
a peripheral, loading calibration data from EEPROM. If the watchdog is running during init and the
I²C sensor does not respond within the watchdog period, you get a reset loop. Perform all init
first, then enable the watchdog at the end of main() before the main loop begins.

Safe State Design

Every embedded system must define — explicitly, in a design document — what state it enters when a
fault is detected. This is not optional in any safety-relevant application.

Motor controller: safe state = motor off (PWM outputs driven low, not floating).

Proportional valve controller: safe state = valve closed (or open — depends on the process
fail-safe direction; spring-return valves default to one position, know which).

HVAC damper controller: safe state = economizer position (fresh air) to avoid CO₂ buildup.

Medical infusion pump: safe state = pump stopped, alarm active.

The safe state must be reachable from a watchdog reset, a brown-out reset, or an assertion failure.
It must not require the CPU to execute complex logic — the hardware must be designed so that the
output drivers default to the safe state when the CPU tristates or drives a known level on reset.
This is a hardware-firmware co-design requirement. IEC 61508 functional safety mandates explicit
safe state definition at every SIL level.

Power-On Reset and Brown-Out Detection (BOD)

When a motor starts or a solenoid fires on the same power rail, the supply voltage can dip by
hundreds of millivolts in tens of microseconds. At 3.3 V nominal, a 600 mV dip brings the rail to
2.7 V. The STM32F4 minimum operating voltage is 1.8 V, so the CPU keeps running — but the flash
memory read timing and the peripheral bus timings were calibrated for 3.3 V. The CPU is now
running on corrupted data without knowing it.

Brown-out detection (BOD) solves this by monitoring the supply voltage and asserting a reset when
it falls below a threshold. The CPU gets a clean reset instead of executing garbage instructions.

STM32 BOD thresholds (Power Voltage Detector, PVD):
Programmable from 2.0 V to 3.0 V in 0.1 V steps. For a 3.3 V system, set PVD threshold to 2.7 V.
When voltage drops below 2.7 V, the PVD interrupt fires. In the PVD ISR: save critical state to
backup SRAM (if available), drive outputs to safe state, and wait for reset.

AVR BOD: fuse-programmable at 1.8 V, 2.7 V, or 4.3 V. For a 3.3 V AVR, use 2.7 V BOD.
Note: BOD fuses are one-time per programming cycle — get them right before mass programming.

Stack Overflow Protection — MPU Guard Regions

Stack overflow is silent on microcontrollers without an MPU. The stack pointer decrements past the
bottom of the stack allocation and starts overwriting heap, BSS, or data segment variables. The
corruption is invisible — variables change value for no apparent reason, then the firmware
eventually crashes in a completely unrelated location. Debugging it without MPU support is painful.

The Cortex-M3, M4, M7, and M33 all include a Memory Protection Unit (MPU). Configure a 32-byte
"guard region" at the bottom of the stack with no read/write permission:

/* Example: STM32 HAL MPU configuration for stack guard */
MPU_Region_InitTypeDef mpu = {
    .Enable           = MPU_REGION_ENABLE,
    .Number           = MPU_REGION_NUMBER0,
    .BaseAddress      = STACK_BOTTOM_ADDRESS,  /* bottom of stack, aligned to region size */
    .Size             = MPU_REGION_SIZE_32B,
    .AccessPermission = MPU_REGION_NO_ACCESS,
    .IsBufferable     = MPU_ACCESS_NOT_BUFFERABLE,
    .IsCacheable      = MPU_ACCESS_NOT_CACHEABLE,
    .IsShareable      = MPU_ACCESS_NOT_SHAREABLE,
    .DisableExec      = MPU_INSTRUCTION_ACCESS_DISABLE,
};
HAL_MPU_ConfigRegion(&mpu);
HAL_MPU_Enable(MPU_HFNMI_PRIVDEF);

When the stack pointer enters the guard region, the MPU fires a MemManage fault immediately —
before any variables are corrupted. The fault handler can log the stack pointer, the LR, and the
task name, then reset to safe state. This converts a mysterious corruption bug into a clean,
debuggable fault.

Communication Failure Handling

Every external communication must have a timeout. "Wait forever for a response" is not acceptable
in production firmware.

I²C bus hang recovery: If an I²C slave is reset mid-transfer, it may be holding the SDA line
low, causing a permanent bus hang. The 9-clock recovery procedure: release the bus, manually
clock SCL 9 times (bit-banging via GPIO), then issue a STOP condition. If SDA goes high after any
of the 9 clocks, the slave has released the bus. If it does not, the slave is damaged — flag the
fault and enter safe state.

CAN bus-off: On overload or persistent errors, a CAN controller enters bus-off state and
silences itself. This is automatic in hardware (compliant with ISO 11898). Firmware must detect
bus-off via the error interrupt, wait for the mandatory 128 × 11 recessive bits bus-off recovery
sequence, then attempt to recover. Do not auto-recover more than N times before flagging a
persistent fault.

Modbus RTU timeout: The standard defines a frame timeout of 3.5 character times. For 9600
baud: 1 character = 1.04 ms, so 3.5 × 1.04 = 3.64 ms. If no response arrives within
150 ms (a generous application-level timeout), the master must time out, log the event, and retry.
After 3 consecutive failures, declare the device absent and raise an alarm.

Retry with backoff: For network communication (Ethernet, Wi-Fi, MQTT), use exponential backoff:
first retry after 1 s, then 2 s, then 4 s, cap at 60 s. Never retry in a tight loop — a stuck
firmware that hammers a server causes network problems and can trigger rate-limiting or bans.

Nonvolatile Storage — Wear and Corruption

EEPROM and flash have finite write cycles. STM32 internal flash: 10,000 erase cycles. AT24C series
EEPROM: 1,000,000 write cycles. If firmware writes a counter to the same address every second, it
will wear that location out in hours (flash) or days (EEPROM).

Wear levelling: for frequently updated values, distribute writes across a circular buffer of N
locations. The current value is whichever location has the highest sequence number.

CRC on all stored configuration: calculate a CRC-16 or CRC-32 over the configuration struct
before writing, store it alongside. On power-on, recalculate and compare. Mismatch → load factory
defaults and flag the event. Never boot with corrupted configuration silently applied.

Backup copy: for critical parameters, maintain two copies in different flash pages. Write the
new value to the backup page first, verify, then update the primary page. On CRC failure of the
primary, fall back to the backup.

IEC 62443 — Cybersecurity for Industrial Systems (Brief)

IEC 62443 defines security levels (SL 1–4) for industrial automation and control systems. The
firmware-relevant requirements:

• Secure boot: firmware image is signed with a private key. The bootloader verifies the
  signature using the manufacturer's public key stored in OTP (one-time programmable) memory before
  executing the image.
• Encrypted OTA updates: firmware updates transmitted over the network must be encrypted
  (AES-128 minimum) and signed. Never accept unsigned firmware images over any interface.
• Input validation: every value received over a communication interface must be range-checked
  before use. A Modbus register value of 65535 for a setpoint that should be 0–100 must be
  rejected, not applied.
• Privilege separation: configuration changes require authentication. Runtime process data does
  not. Separate these access levels.

Full cybersecurity implementation is a project in itself. The minimum viable position for an
industrial product: secure boot + signed OTA + input validation on all external interfaces.

IEC 61508 and Functional Safety — SIL Levels

IEC 61508 is the base functional safety standard for electrical/electronic/programmable systems.
IEC 62061 applies it specifically to machinery. SIL = Safety Integrity Level.

What SIL demands from firmware:

• Redundant calculations: compute safety-critical values twice (ideally using different
  algorithms or different data types) and compare results. Discrepancy → safe state.
• Diverse implementation: SIL 3/4 requires software diversity — two independent software
  implementations, ideally by different teams, compiled with different toolchains.
• Diagnostic coverage (DC): what fraction of failures are detected by self-tests. SIL 2
  requires DC ≥ 60%, SIL 3 requires DC ≥ 90%.
• Proof-test interval: how often the safety function is tested end-to-end in service. IEC 61508
  uses this interval in the PFH calculation.
• No dynamic memory allocation: heap allocation is non-deterministic and non-verifiable in
  safety firmware. Use static allocation only.
• MISRA C compliance: the majority of automotive and industrial safety projects require MISRA C
  compliance as a minimum coding standard.

Watchdog Kick Sequence — Failure Scenario

Common Firmware Failure Modes

Key Takeaway

The firmware that ships to production is almost never the firmware that ran in the lab. In the lab,
no cables are 2 metres long, no one walks across carpet and touches the enclosure, and the power
supply is clean. Design firmware assuming the worst-case real-world environment: voltage dips,
communication timeouts, stuck sensors, and ESD events. Every failure mode in the table above has
killed a real product in production — usually discovered by a customer, not by the engineering team.

→ Next: Electronics Hardening