Abstract Abstract This article presents a practical perspective on designing and implementing fault management in embedded systems, particularly within automotive and industrial control contexts. It discusses the architecture of a layered fault-handling approach—comprising the Application Layer, Diagnostic Event Management (DEM) Wrapper, Low-Level Software (LLSW) Logic, and Hardware Monitoring. Through real-world observations and project-based experience, the article highlights challenges in fault detection, such as debounce logic, timing sensitivity, and noisy analog readings. Recommendations and lessons are drawn from specific case studies, including fault suppression through retry filters and handling sensor inaccuracies using median filters. The objective is to provide system engineers with actionable strategies for building fault-resilient embedded applications. 1. Introduction 1. Introduction Fault handling is a critical component in the reliability and safety of embedded systems. In safety-critical domains such as automotive, aerospace, and industrial automation, the ability to detect, log, and respond to faults in a timely and traceable manner can be the difference between product stability and costly field failures. This article discusses an architecture that we have used in production systems, built on four conceptual layers: Application Layer
DEM Wrapper
LLSW Fault Log
Hardware Monitoring Layer Application Layer DEM Wrapper LLSW Fault Log Hardware Monitoring Layer While these components are common across many embedded frameworks (including AUTOSAR-based systems), their practical implementation requires careful balancing of timing constraints, hardware limitations, and system-level behavior. Drawing on lessons from real-world field issues and internal testing, this article aims to bridge the gap between abstract fault frameworks and implementation reality. Figure 1: Fault Handling Architecture Diagram 2. Architectural Layers of Fault Handling 2. Architectural Layers of Fault Handling 2.1 Application Layer 2.1 Application Layer The Application Layer interfaces with both runtime logic and diagnostic components. Its primary role in fault management is to: Query fault states using Get_Fault() APIs
Trigger test simulations using Set_Fault() APIs during validation
Modify system behavior in response to active faults (e.g., initiate safe mode) Query fault states using Get_Fault() APIs Get_Fault() Trigger test simulations using Set_Fault() APIs during validation Set_Fault() Modify system behavior in response to active faults (e.g., initiate safe mode) In one case involving a drivetrain control module, a recurring early-morning fault warning was traced back to the application layer correctly reporting a fault that was falsely triggered in LLSW. This highlighted the importance of validating the logic behind fault generation—not just the retrieval. Design Insight: Application code should treat diagnostic queries as "truth," but diagnostic inputs must be traceable and rigorously validated. Design Insight: Application code should treat diagnostic queries as "truth," but diagnostic inputs must be traceable and rigorously validated. 2.2 DEM Wrapper 2.2 DEM Wrapper The DEM Wrapper abstracts the logging and state management of faults. Key functionalities include: Storing fault metadata (e.g., timestamps, frequency)
Managing fault maturity and dematurity windows
Providing consistent fault access APIs to the upper layers Storing fault metadata (e.g., timestamps, frequency) Managing fault maturity and dematurity windows Providing consistent fault access APIs to the upper layers For example, a heater fault would only be marked as “matured” if the fault condition persisted for a defined period—initially set to 50ms, later adjusted to 250ms after detecting false positives due to voltage sags during preconditioning. Best Practice: Configure fault maturing/dematuring thresholds based on empirical testing. Excessively tight windows lead to noise; loose thresholds may suppress genuine failures. Best Practice: Configure fault maturing/dematuring thresholds based on empirical testing. Excessively tight windows lead to noise; loose thresholds may suppress genuine failures. Best Practice 2.3 LLSW Fault Management Logic 2.3 LLSW Fault Management Logic LLSW (Low-Level Software) is responsible for translating raw data into meaningful fault conditions. It typically: Applies retry logic (e.g., three consecutive failure samples before fault)
Implements debounce filters to remove transient spikes
Manages fault enablement based on runtime configuration Applies retry logic (e.g., three consecutive failure samples before fault) Implements debounce filters to remove transient spikes Manages fault enablement based on runtime configuration One project used a three-strike model with 10ms sampling intervals to determine a PWM signal drop. This was necessary to distinguish between true hardware failures and shutdown-induced signal tapering. Tip: Use runtime flags to disable fault detection in transitional states (e.g., boot-up, self-test) to reduce false positives. Tip: Use runtime flags to disable fault detection in transitional states (e.g., boot-up, self-test) to reduce false positives. Tip 2.4 Hardware Monitoring Layer 2.4 Hardware Monitoring Layer At the lowest layer, the system interacts with physical components via sensors, ADCs, and digital inputs. This layer is where signal noise, EMI, and analog drift often undermine ideal fault detection. An example involved a TI INA219 current sensor used for bus monitoring. EMI during high-power charging events caused voltage spikes, falsely triggering fault logic. A simple rolling median filter stabilized readings: Lesson: Filtering at the hardware abstraction level can significantly improve fault signal integrity. Lesson: Filtering at the hardware abstraction level can significantly improve fault signal integrity. Lesson Figure 2: Sample Debounce Timing Flow 3. Case Study: Heater Fault During Preconditioning 3. Case Study: Heater Fault During Preconditioning Scenario: Fault was reported daily at 6:10 AM during scheduled battery preconditioning. The DEM logged an overcurrent event, but all signals normalized within seconds. Scenario Diagnosis: Diagnosis Voltage dip during cold-start heater ramp-up
LLSW triggered fault after a single sample (no retry)
DEM maturing window too short (50ms) Voltage dip during cold-start heater ramp-up LLSW triggered fault after a single sample (no retry) DEM maturing window too short (50ms) Resolution: Resolution Implemented three-sample retry with 10ms spacing
Extended maturing window to 250ms
Added conditional skip for initial startup phase Implemented three-sample retry with 10ms spacing Extended maturing window to 250ms Added conditional skip for initial startup phase Outcome: Fault no longer triggered falsely; system behavior stabilized without disabling the check entirely. Outcome 4. Recommendations 4. Recommendations Aspect

Recommendation



Maturing Logic

Start with conservative timing, adjust based on test data



Retry Implementation

Use sample-count thresholds (e.g., 3 out of 5) to validate faults



Debouncing

Include both time-based and event-based debounce logic



Visual Logging

Pair logs with system state snapshots for traceability



Environmental Testing

Validate fault detection at low temperature, high EMI, etc. Aspect

Recommendation



Maturing Logic

Start with conservative timing, adjust based on test data



Retry Implementation

Use sample-count thresholds (e.g., 3 out of 5) to validate faults



Debouncing

Include both time-based and event-based debounce logic



Visual Logging

Pair logs with system state snapshots for traceability



Environmental Testing

Validate fault detection at low temperature, high EMI, etc. Aspect

Recommendation Aspect Aspect Recommendation Recommendation Maturing Logic

Start with conservative timing, adjust based on test data Maturing Logic Maturing Logic Start with conservative timing, adjust based on test data Start with conservative timing, adjust based on test data Retry Implementation

Use sample-count thresholds (e.g., 3 out of 5) to validate faults Retry Implementation Retry Implementation Use sample-count thresholds (e.g., 3 out of 5) to validate faults Use sample-count thresholds (e.g., 3 out of 5) to validate faults Debouncing

Include both time-based and event-based debounce logic Debouncing Debouncing Include both time-based and event-based debounce logic Include both time-based and event-based debounce logic Visual Logging

Pair logs with system state snapshots for traceability Visual Logging Visual Logging Pair logs with system state snapshots for traceability Pair logs with system state snapshots for traceability Environmental Testing

Validate fault detection at low temperature, high EMI, etc. Environmental Testing Environmental Testing Validate fault detection at low temperature, high EMI, etc. Validate fault detection at low temperature, high EMI, etc. 5. Conclusion 5. Conclusion Fault management in embedded systems is rarely plug-and-play. While architectural patterns like DEM-LLSW separation are valuable, their real-world application requires careful tuning, environmental validation, and resilience against noisy hardware signals. This article outlined a layered approach supported by field-tested observations and practical code snippets. A robust fault management design balances responsiveness, stability, and traceability—qualities essential for any safety-critical embedded system.

This story contains new, firsthand information uncovered by the writer.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Arweave

Low-Level Diagnostics, Fault Management in Embedded Systems: Design, Implementation, Lessons Learned

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Building a Factory Test Framework for ECU Low-Level APIs in CAN-Based HIL Systems

Adaptive AUTOSAR in a nutshell

A Brief Guide to Rust for Embedded Development

A Closer Look at the Top 3 Embedded Databases: SQLite, RocksDB, and DuckDB

A Guide to Network Gadgets that Contain Ping, NTP, TFTP, and Iperf

Building a Factory Test Framework for ECU Low-Level APIs in CAN-Based HIL Systems

Adaptive AUTOSAR in a nutshell

A Brief Guide to Rust for Embedded Development

A Closer Look at the Top 3 Embedded Databases: SQLite, RocksDB, and DuckDB

A Guide to Network Gadgets that Contain Ping, NTP, TFTP, and Iperf

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps