Error Handling, Watchdogs, and Recovery
Production firmware must expect faults. Sensors disconnect, communication times out, memory corrupts, and code can hang. Robust systems detect faults, move outputs to safe states, record useful evidence, and recover when possible.
Learning Objectives
By the end of this lesson, you should be able to define fault classes, design error codes, use watchdogs safely, log reset reasons, and choose recovery actions for embedded products.
Fault Classes
- Transient fault: one CRC error. Retry and count it.
- Recoverable fault: sensor timeout. Reinitialize or enter degraded mode.
- Critical fault: overtemperature. Move outputs to a safe shutdown state.
- Programming fault: assertion failure. Log context and reset if needed.
Do not treat every error as a reset. Resets can hide root causes and create unsafe output glitches.
Error Codes
typedef enum {
ERR_NONE = 0,
ERR_SENSOR_TIMEOUT,
ERR_SENSOR_CRC,
ERR_FLASH_COMMIT,
ERR_SUPPLY_LOW,
ERR_WATCHDOG_RESET
} error_code_t;
Include context such as timestamp, subsystem, state, and measured value when RAM or storage allows.
Watchdogs
A watchdog that is kicked from a timer ISR can miss a hung main loop and gives false confidence.
Watchdog Timeout Selection
Timeout must be longer than worst-case legitimate blocking time but shorter than the maximum safe hang time.
$$
T_{wdg} > T_{max\ normal\ loop}+margin
$$
Also verify startup, flash erase, radio joins, and firmware update operations.
Reset Reason and Crash Data
On boot, read reset reason flags before clearing them. Store reset reason, program counter or fault registers, active state, last error code, build version, and uptime when possible.
Worked Example: I2C Sensor Lockup
If an I2C sensor stops responding, abort after timeout, pulse SCL if the bus is stuck, reinitialize the peripheral, retry sensor initialization, enter degraded mode after repeated failure, and log ERR_SENSOR_TIMEOUT with retry count.
Common Mistakes
- Refreshing the watchdog unconditionally.
- Resetting immediately on every minor error.
- No reset reason log.
- Infinite retry loops with no backoff or fault state.
- Leaving actuators energized during fault handling.
Summary
Fault handling is a product feature. Use specific errors, bounded retries, safe outputs, watchdogs tied to health checks, reset reason logging, and clear recovery policies.
Further Reading
- MCU vendor independent watchdog and reset controller documentation.
- Memfault, "Watchdog Best Practices" and "Fault Debugging" guides.
- IEC 60730 and functional safety literature for safety-related appliances.