Performance Evaluation of Real-Time Operating Systems

1. Performance Evaluation of Real-Time Operating Systems

RTEMS, RTLinux, eCos

2. Real-Time Systems

Systems that interact predictably with
events in the outside world
Examples:
Flight control systems
Collision avoidance systems
Satellite guidance systems
Patient Monitoring System
Control Systems for synchrotrons

3. Real-Time Operating Systems

Free the applications programmer from writing
code for task scheduling and dispatching.
Manage time-sharing of a processor for a number
of different tasks and interrupt sources, while
adhering to strict time constraints.
Widely used in all kinds of applications both on
PCs and in embedded systems.
Generally operate unattended -- consequences of
failure can be catastrophic.

4. What is available?

Many RTOSs have been developed:
Commercial
Open source
RTOS vendors and developers publish
performance metrics to showcase their
products. However, their evaluations are:
not performed on a common platform
not comprehensive

5. Evaluation of an RTOS

How would you choose an appropriate RTOS for
your application?
Need for an impartial evaluation.
Several such projects have been conducted.
Only evaluated some performance characteristics
Only looked at a few commercial operating systems
Many are now out of date
‘Real-Time Consult’ introduced a more
comprehensive project.
Developed a thorough test methodology
Evaluated several commercial RTOSs
Have not evaluated open source RTOSs

6. Research Objective

Performance evaluation, on a common
platform, of three open source real-time
operating systems:
RTEMS
RTLinux
eCos

7. Real-Time Operating Systems

Overview
RTOS
Task
Scheduling
Task
Synchronization
Task
Dispatching

8. Task Scheduling

Time Slicing
Last
Arrival
Order
Task 3
Task 3
Task 2
First Task 1
Task 2
Task 1
Time

9. Task Scheduling

Preemptive Priority
High
Task 3
Priority
Task 2
Low Task 1
Task 2
Task 1
Time

10. Task Dispatching

Task-control blocks is the
most popular method of
identifying and managing
tasks.
Task-Control Block (TCB)
contains:
a context (e.g. program counter
and register contents)
an identification string
a status, such as ready,
executing or blocked
a priority (if applicable)
A Typical
Task Control Block
Program Counter
Task Status
Task ID Number
Register Contents
Pointer to next TCB
:
Priority
Other Context

11. Task Synchronization

Events
Messages
Semaphores
Mutexes

12. Desired Performance Characteristics

Tasks should run with:
minimal event latency (the time between the
triggering of a task and its start)
minimal jitter (the variation in running times of
a task that is supposed to run at a fixed period).

13. Performance Evaluation

Throughput – speed at which the system
executes instructions
Responsiveness – how fast it starts to
handle in interrupt request
Determinism – how it reacts under load
how long does it take to finish what it is doing
and start handling the interrupt request

14. Performance Metrics

Low Level Tests
Context switching
Interrupt latency
Exclusion objects
Semaphores
Mutexes
Synchronization events

15. Context Switch Time

Switch to Task B
Task A
Save context
of Task A
Load context
of Task B
PC
PC
Register 0
Register 0
:
:
:
:
Other Context
Other Context
Context Switch Time
Start
Task B
Time

16. Other Considerations

Not an isolated event
Affected by number of tasks pending
Depends on priorities of pending tasks
Amount of context to be saved

17. Interrupt Latency

Interrupt and Interrupt Dispatch
Latencies
Interrupt
Occurs
First Instruction in
Interrupt Service Routine
(ISR) is executed
Task
resumes
ISR ends
ISR
Task
Interrupt
Latency
Task
Interrupt
Dispatch Latency
Time

18. Multiple Interrupt Latencies

Interrupt-to-Task Run
Interrupt
Task Run
Interrupt Dispatch Time
Interrupt Service Routine
Other Interrupt
Pre-emption Disabled
Scheduling
Context Switch
Return from System Call

19. Interrupt Task Response Time

Real response to an interrupt often occurs in
a task synchronized by, but outside of, the
actual ISR.
Task response also depends on whether the
kernel is preemptable. If not, then kernel
call must be completed before ISR.

20. Performance Metrics

High Level Tests
Network throughput
Stress tests

21. Real-Time Consult Project

Evaluation and comparison of RTOS performance
Commercial systems only, to date
Evaluations available for:
Intime 1.20
RTX 4.2
Hyperkernel 4.3
VxWorks/x86 5.3.1
pSOSystem/x86 2.2.6
QNX 6.1
CE 3.0

22. Real-Time Consult Project

Test Method: timing is measured using an
external PCI bus analyzer.
Two types of tests:
Performance Tests
Stress Tests
Qualitative Evaluation (API richness, etc.)

23. Real-Time Consult Project

Performance Tests (context switch &
latencies):
Thread switch latency
Interrupt latency
preempt current thread and start interrupt handler
Interrupt dispatch latency
time required to preempt current thread
switch from interrupt context to context of
interrupted thread or next thread in queue
Thread creation and deletion

24. Task Switch Latency Test

Task start
TSK0
Each Task
Create tasks
TSK1, …, TSKN
End of Test?
No
Write trace on PCI
Bus
TRC_TSK0_TS0
Yield
Processor
Task start
TSKN
Yes
Delete tasks
TSK1, …, TSKN
Task end
TSK0
Write trace on PCI
Bus
TRC_TSKN_TSN
Yield
Processor

25. Real-Time Consult Project

Performance Tests (objects, file, network):
Synchronization objects
Exclusion objects
time to create and delete an exclusion object
File system operations
time to create and delete a synchronization object
creating and deleting files, reading from files and
writing to files in synchronous mode
Network stack
performance of TCP/IP stack
bandwidth for various packet sizes, and CPU usage

26. Real-Time Consult Project

Stress Tests:
Two simultaneous interrupts
Interrupt nesting
maximum sustainable interrupt frequency
maximum number of objects
memory leaks

27. Focus of my Research

Evaluation of three Open Source RealTime Operating Systems:
RTEMS
Real Time Executive for Multiprocessor Systems
RTLinux
Real Time Linux
eCos
Embedded Configurable Operating System

28. RTEMS

Developed by the U.S. military as alternative to
using commercial RTOS
Small, easy to port
High level of user configurability
Kernel is preemptible
Application Dependent Software
Standard Application Components
Device Drivers
RTEMS
Target Hardware

29. RTLinux

Abstraction layer between the hardware and
the standard Linux kernel
Appears as actual hardware to standard Linux
kernel
Lowest priority is assigned to standard Linux
kernel, which then runs as a independent task.
RTLinux executive is nonpreemptible
Leaves Linux kernel essentially untouched
so it doesn’t hinder future Linux
development

30. ECOS

Targeted at high-volume applications:
consumer electronics, telecommunications, automotive
other deeply embedded applications.
Configurable:
lets developer configure a system that best matches the
needs of the application.
Typical configuration options:
type of scheduler
number of priority levels.
current release of the system has over 200 options
Portable
Fully preemptable

31. Typical Application

Control system for the Canadian Light
Source (CLS) uses an open source real-time
operating system and control software:
RTOS RTEMS
Control Software EPICS
(Experimental Physics and Industrial Control System)