Measuring Real-Time Operating System Performance – Part I: Finding and Porting a Benchmark Suite

When working on real-time systems, it is often necessary that certain operations meet a specific deadline. Efficient operating system services, like task switching or inter-task synchronization, for example, play a crucial part in meeting these deadlines. Most of these services come at a cost: control has to be transferred from the user tasks to the operating system and, therefore, latencies outside of our control may occur.

In this series of two blog posts, I want to show you how benchmarks can be used to measure these latencies. This provides us with an estimate of the suitability of certain operating systems for a specific real-time application. We will measure the performance of two open-source real-time operating systems in the second blog post: FreeRTOS and Zephyr. In this first blog post, I want to focus on finding a benchmark suite designed for testing operating system services and porting it to our two operating systems.

Finding a Benchmark Suite

Before we can measure anything, we need a benchmark suite fitted to our needs. Many real-time benchmark suites are concerned with typical real-time applications, like video encoding for example. They provide benchmarks in the form of algorithms often used in these kinds of applications. Examining the services provided by the platform’s underlying operating system requires a different set of test cases. During the research, I was pointed to RTOSBench, a benchmark suite specifically designed for that purpose.

RTOSBench is the result of a master thesis from the University of Montréal. It includes test cases for services present in most operating system kernels: (cooperative) context switches, mutexes, semaphores, message queues and interrupts. Unfortunately, the original project seems to be no longer maintained and has smaller bugs in it. I forked the project to fix them and you can find my version here.

Porting RTOSBench to FreeRTOS and Zephyr

One key idea behind RTOSBench is portability, meaning it can be adapted to previously unsupported operating systems. This is done by encapsulating the respective kernel’s API inside a thin porting layer. In order to run RTOSBench on a new operating system, we have to provide two things: a configuration header (named config.h) containing macros and data type definitions and an implementation of the porting layer’s function API (in e.g. zephyr_porting_layer.c). The graphic below shows a schematic overview of RTOSBench and its porting layer.

Providing Type Definitions

In order to get a better understanding of the porting process, let us implement semaphore support for FreeRTOS and Zephyr. First, we have to define RTOSBench’s data type no_sem_t in such a way that it uses the respective operating system’s semaphore. FreeRTOS uses the data type SemaphoreHandle_t. Therefore, we have to add the following type definition to our config.h for FreeRTOS:

  typedef SemaphoreHandle_T no_sem_t;

Zephyr’s semaphore data type is called struct k_sem, i.e. we have to add the following type definition to the Zephyr port’s config.h:

  typedef struct k_sem no_sem_t;

Implementing API Functions

After providing RTOSBench with a definition of no_sem_t, we can move on to implementing the semaphore API functions. Overall, we have to implement three semaphore-related functions in our porting_layer.c: no_sem_create, no_sem_wait and no_sem_signal. The function no_sem_create() is used to initialise a semaphore sem with an initial value. Zephyr provides the same service via the function k_sem_init(). It takes a pointer to a semaphore and initialises it accordingly. We can implement no_sem_create() as follows:

  void
  no_sem_create(no_sem_t *sem, int value)
  {
          k_sem_init(sem, (unsigned int)value, K_SEM_MAX_LIMIT);
  }

FreeRTOS does this is a bit differently. We can use the function xSemaphoreCreateCounting() to create a new semaphore. Unlike Zephyr’s k_sem_init(), this function does not receive a pointer to the semaphore, but allocates a semaphore from FreeRTOS’s heap memory and returns it. Therefore, the FreeRTOS implementation of no_sem_create() looks like this:

  void
  no_sem_create(no_sem_t *sem, int value)
  {
          /*
           * The possible maximum value depends on your platform. On the
           * example platform UBaseType_T is a typedef for unsigned long.
           */
          *sem = xSemaphoreCreateCounting(ULONG_MAX, value);
          /* Was the semaphore allocation successful? */
          if (*sem == NULL) {
                  no_serial_write("sem_create: error");
                  Error_Handler();
          }
  }

The two remaining semaphore functions can be ported in the same manner and after proceeding in a similar way with all other kernel services, we should be able to execute RTOSBench on both operating systems. If you are interested in more details, you can find my Zephyr port on GitHub. Please note that it only supports ARM Cortex chipsets with a Nested Vector Interrupt Controller (NVIC).

Conclusion

The information provided here should get you started in porting RTOSBench to new operating systems. In conclusion, porting RTOSBench to Zephyr turned out to be relatively simple. I did not encounter any larger obstacles, since Zephyr’s toolchain makes it easy to create new applications and its code base provides good abstractions of the target platform’s low-level details. Porting RTOSBench to FreeRTOS, however, was a bit more complicated. The main reason is that FreeRTOS is more minimalist and does neither provide a common hardware abstraction layer nor a toolchain like Zephyr’s. Therefore, hardware manufacturers have to provide these instead. This means that parts of RTOSBench’s porting layer might have to be re-implemented for each platform that RTOSBench should be executed on when benchmarking FreeRTOS.

In the second blog post of this series, we will put our newly ported benchmarks to work and compare the kernel service performance of FreeRTOS and Zephyr.