Analyzing real-time operating systems using trace techniques

Code and data trace have absolutely no influence on the real-time behaviour of a system. This advantage can be used extensively when analyzing the runtime behaviour of real-time operating systems.


By Jens Braunes, PLS                                                Download PDF version of this article


From experience, system observation is among the most critical aspects of testing and analyzing embedded software, because many factors may adversely affect runtime behaviour. One of them arises already during the build process: do we compile the application for debugging and test purposes or for production enabling additional optimizations for code size or speed? Profiling, which is frequently applied in order to optimize performance, also has a considerable influence. Quite often, the application needs to be instrumented for measuring the runtimes of functions and tasks. The added test code will have a small, but measurable influence on the application runtime behaviour. Even the memory layout may change due to the additional code, so that the influence cannot be neglected anymore. Among other variants of high-level system observation, monitoring is frequently used especially in the context of real-time operating systems. For this purpose, the monitor checks for RTOS state modifications that are typically mapped to specific memory locations. Triggered by interrupts which occur on task switches (among others), the monitor records the event for later analysis.

Those engineers who want to go without any instrumentation of the program code and without any monitors for analyzing the runtime behaviour of a real-time operating system (RTOS), should select a microcontroller providing appropriate trace hardware including a high-bandwidth trace interface. MCU families offering suitable features are now available from almost all semiconductor vendors. Today, tracing capabilities are often a show-stopper for the selection of a platform. Tracing can be used to observe modifications of system states without influencing the system real-time behaviour. Depending on the controller manufacturer and the implemented trace architecture, there will be several trace modes which might be useful to analyze the runtime behaviour of an operating system. However, before going deeper into that, it is necessary to point out what information is required actually in the specific case.

In RTOS-controlled applications, the runtime of each specific task plays a significant role for evaluating execution performance. For instance, the task execution time directly indicates whether the system is running at a reasonable workload or in an overload situation. As another important factor, it is interesting to know how often tasks are interrupted by other tasks and how long tasks can operate without being interrupted. Due to the overhead incurred by task switches, this often bears significant potential for optimization. The same applies for the interrupt load which gives an indication of how often and how long task execution is halted by interrupts.

Obviously, code tracing is the method of choice for obtaining the required runtime information for all necessary measurements as simply and quickly as possible. Basically, tracing consists of recording the addresses of executed branches (more specifically of the branch targets) as well as any deviations from the regular, sequential execution flow as a result of calls, returns and interrupts. For task analysis, code tracing can be used to detect task switches that manifest themselves by multiple function calls and returns (including the scheduler), thus causing disruptions of the sequential execution flow. However, the code trace is much too large for this purpose because trace data are also recorded for the execution of in-function code, including if-then-else constructs or loops. This increases the size of the trace and represents an unnecessary load for the on-chip or external trace memory. Infineon uses an interesting approach to avoid this dilemma. The so-called Compact Function Trace (CFT) of this company records only function calls and returns. The program flow within functions is not captured by the trace at all. Of course, that reduces dramatically the amount of trace data and saves trace memory.

Data trace is another option complementing code tracing. Typically, RTOSs have their own management structures located in MCU memory reflecting the current system state and the task scheduling. This includes active tasks and information about the interrupt processing. Changes of the system state, including task switches, thus always trigger a write access to these management structures. Some trace implementations allow data trace that can be used to record these write accesses. That enables a precise tracking and analysis of state changes of the operating system. However, data tracing is quite expensive in terms of memory space. Without additional measures, all memory accesses of the application would be recorded. Similar to code tracing, this is undesirable here as well. Appropriate filters must therefore be applied to ensure that recording is exclusively restricted to relevant accesses targeting the management structures. Regardless of which trace mode is used for obtaining runtime information, unique, uniform timestamps are required here. Otherwise, it would be virtually impossible to get precise measurements and useable results.

How an analysis using trace data will work in the field can be demonstrated with an example of an OSEK-compliant operating system. OSEK is the abbreviation for “Offene Systeme und deren Schnittstellen für die Elektronik in Kraftfahrzeugen” (open systems and their interfaces for automotive electronics). It designates a standard for implementing real-time operating systems particularly in the automotive environment. The well-known Autosar approach represents an advanced development and reuse of the OSEK specification. The OSEK also defines an interface format called ORTI (OSEK Runtime Interface) for the communication of the operating system with analysis tools and debuggers. In a text file (the so-called ORTI file) this format describes all relevant internal operating system data, facilitating their use by tools and their visualization for users.

The debugger - the Universal Debug Engine (UDE) from PLS in this case - extracts the data structure from the ORTI file containing for example the current task (figure 1). This information is used for configuring the trace filter, ensuring that only data trace is recorded actually relevant for task analyses (figure 2). In the next step, the trace is preprocessed to allow a proper visualization of the sequences of executed operating-system tasks including their precise timing. Specialized tools like Eclipse Trace Compass (www.tracecompass.org) can be used afterwards for sophisticated visualizations and additional analyses (figure 3). For this purpose, the UDE provides an export function based on the BTF format (Best Trace Format) defined by Timing Architects, Inc. BTF, which was specifically developed as an exchange format for event traces, is used in many simulation, profiling and trace analysis tools.

As demonstrated by this example, a combined tool solution consisting of a debugger and hardware tracing is now a genuine alternative to collecting runtime information using instrumentation or special monitors. Even though hardware trace will often operate too finely-grained - it will typically record the entire control flow including in-function instructions and will thus be quite memory-hungry - developments like CFT or the use of ORTI for data tracing can quickly compensate for this disadvantage. All in all, tracing provides a method for collecting runtime information in a fully reactionless and highly precise manner. And not least, it can also be used for examinations at the task and operating-system level.


Related


Hardware-based AES Encrypted Storage Solution

Secure data encryption is essential for a wide variety of mission-critical applications pertaining to both civilian matters and national security. These sectors both require comprehensive safeguards t...

Give Your Product a Voice with Alexa

Join us for a deep dive into the system architecture for voice-enabled products with Alexa Built-In. Device makers can use the Alexa Voice Service (AVS) to add conversational AI to a variety of produc...

The two big traps of code coverage

Code coverage is important, and improving coverage is a worthy goal. But simply chasing the percentage is not nearly so valuable as writing stable, maintainable, meaningful tests. By Arthur Hick...

Securing the smart and connected home

With the Internet of Things and Smart Home technologies, more and more devices are becoming connected and therefore can potentially become entry points for attackers to break into the system to steal,...

 

nVent Schroff at Embedded World 2019

The theme of the nVent Schroff booth at Embedded World 2019 was “Experience Expertise – Modularity, Performance, Protection and Design”. Join us as our experts give an overview of th...


Garz & Fricke Interview at Embedded World 2019 with Dr. Arne Dethlefs: We are strengthening our presence in North America

Through its US subsidiary, located in Minnesota, Garz & Fricke is providing support for its growing HMI and Panel-PC business in the USA and Canada while also strengthening its presence in North A...


SECO's innovations at embedded world 2019

In a much larger stand than in previous years, at embedded world 2019 SECO showcases its wide range of solutions and services for the industrial domain and IoT. Among the main innovations, in this vid...


Design and Manufacturing Services at Portwell

Since about two years Portwell is part of the Posiflex Group. Together with KIOSK, the US market leader in KIOSK systems, the Posiflex Group is a strong player in the Retail, KIOSK and Embedded market...


Arrow capabilities in design support

Florian Freund, Engineering Director DACH at Arrow Electronics talks us through Arrow’s transformation from distributor to Technology Platform Provider and how Arrow is positioned in both, Custo...


Arm launches PSA Certified to improve trust in IoT security

Arm’s Platform Security Architecture (PSA) has taken a step forward with the launch of PSA Certified, a scheme where independent labs will verify that IoT devices have the right level of securit...


DIN-Rail Embedded Computers from MEN Mikro

The DIN-Rail system from MEN is a selection of individual pre-fabricated modules that can variably combine features as required for a range of embedded Rail Onboard and Rail Wayside applications. The ...


Embedded Graphics Accelerates AI at the Edge

The adoption of graphics in embedded and AI applications are growing exponentially. While graphics are widely available in the market, product lifecycle, custom change and harsh operating environments...


ADLINK Optimizes Edge AI with Heterogeneous Computing Platforms

With increasing complexity of applications, no single type of computing core can fulfill all application requirements. To optimize AI performance at the edge, an optimized solution will often employ a...


Synchronized Debugging of Multi-Target Systems

The UDE Multi-Target Debug Solution from PLS provides synchronous debugging of AURIX multi-chip systems. A special adapter handles the communication between two MCUs and the UAD3+ access device and pr...


Smart Panel Fulfills Application Needs with Flexibility

To meet all requirement of vertical applications, ADLINK’s Smart Panel is engineered for flexible configuration and expansion to reduce R&D time and effort and accelerate time to market. The...


Artificial Intelligence

Morten Kreiberg-Block, Director of Supplier & Technology Marketing EMEA at Arrow Electronics talks about the power of AI and enabling platforms. Morten shares some examples of traditional designin...


Arrow’s IoT Technology Platform – Sensor to Sunset

Andrew Bickley, Director IoT EMEA at Arrow Electronics talks about challenges in the IoT world and how Arrow is facing those through the Sensor to Sunset approach. Over the lifecycle of the connected ...


AAEON – Spreading Intelligence in the connected World

AAEON is moving from creating the simple hardware to creating the great solutions within Artificial Intelligence and IoT. AAEON is offering the new solutions for emerging markets, like robotics, drone...


Arrow as a Technology Provider drive Solutions selling approach

Amir Sherman, Director of Engineering Solutions & Embedded Technology at Arrow Electronics talks about the transition started couple of years ago from a components’ distributor to Technology...


Riding the Technology wave

David Spragg, VP, Engineering – EMEA at Arrow Electronics talks about improvements in software and hardware enabling to utilize the AI capabilities. David shares how Arrow with its solutions is ...


ASIC Design Services explains their Core Deep Learning framework for FPGA design

In this video Robert Green from ASIC Design Services describes their Core Deep Learning (CDL) framework for FPGA design at electronica 2018 in Munich, Germany. CDL technology accelerates Convolutional...


Microchip explains some of their latest smart home and facility solutions

In this video Caesar from Microchip talks about the company's latest smart home solutions at electronica 2018 in Munich, Germany. One demonstrator shown highlights the convenience and functionalit...


Infineon explains their latest CoolGaN devices at electronica 2018

In this video Infineon talks about their new CoolGaN 600 V e-mode HEMTs and GaN EiceDRIVER ICs, offering a higher power density enabling smaller and lighter designs, lower overall system cost. The nor...


Analog Devices demonstrates a novel high-efficiency charge pump with hybrid tech

In this video Frederik Dostal from Analog Devices explains a very high-efficiency charge-pump demonstration at their boot at electronica 2018 in Munich, Germany. Able to achieve an operating efficienc...