RTLinux versus RTAI

Two approaches to real-time Linux

1. Improving the Linux kernel preemption.
2. Adding a new software layer beneath Linux kernel with full control of interrupts and processor key features.

TimeSys and Linux kernel preemption project (supported by Montavista) took the first approach, both groups are working in parallel to produce a similar result.

The second approach is supported by RTLinux and RTAI.

A detailed analysis of the pros and cons of the two alternatives can be found in the paper Comparing two approaches to real-time Linux by Tim Bir.

How Real-Time performance is achieved in both, RTLinux and RTAI

"The real-time Linux scheduler treats the Linux operating system kernel as the idle task. Linux only executes when there are no real time tasks to run, and the real time kernel is inactive. The Linux task can never block interrupts or prevent itself from being preempted. The mechanism that makes this possible is the software emulation of interrupt control hardware."

There are some intrinsic real-time features that are achieved by just executing real-time tasks in kernel space:

• Real-Time tasks (threads) are executed inside kernel memory space, which prevents threads to be swapped-out and also also the number of TLB misses is reduced.
• Threads are executed in processor supervisor mode (i.e. ring level 0 in i386 arch), have full access to the underlying hardware.
• Since the RTOS and the application are linked together in a "single" execution space, system call mechanism is implemented by mean of a simple function call (not using a software interrupt which produces higher overhead).

RTLinux:

RTLinux is released under two different licenses:

1. Open RTLinux Patent License . This license allow you to use RTLinux without fee in two situations:
   1. By software licensed under the GPL; or
   2. By software that executes within an Open RTLinux Execution Environment - whether that software is licensed under the GPL or not.
   where "Open RTLinux Execution Environment" means: A computer hardware system where the interrupt control hardware of processors and system boards is under the direct control of unmodified Open RTLinux Software in binary form.

   Any modification of the code covered by the Open RTLinux License must by GPL 2 (fully GPL 2, i.e. the code can not be closed). It must be open and be available on the web. Since it is NOT LGPL it is not legal to link with non-GPL code.

   
   
2. Non-free¹ License: FSM Labs require you to buy a commercial license if you want to impose proprietary restrictions on the use of your products³, that is, distribute your derived work (based on RTLinux) using a different licensed than the GPL. The comercial license is simmilar to the one used by QT Library done by TrollTech. The main idea of these licenses is that you can do whatever you want, but if you make money (money change hands, between you and your customer) then you have to buy a license.

   FSM Labs has patented (U.S. Patent No. 5,995,745) the process of capturing the interrupts and provide a virtualized hardware to Linux so that RTLinux can take full control of the machine to provide hard real-time response. It is a software patent which is not applicable to Europe, it is only valid in USA.

RTAI:

Mostly LGPL 2. Which means that: users can modify the code (Like any GPL program) but also it is possible to use proprietary modules that can be linked or used in conjunction with RTAI code.

The RATI core is GPL since that work came from RTLinux original code. The situation of the RTAI license is not clear. It is even possible (where software patents are enforced, no in EU) that you have to buy a license from FSM Labs to use RTAI.

Comments

From the academic point of view, RTLinux and RTAI are the same, you can use and modify the code as you like.

The RTAI team tries to allow proprietary development for zero prize² (by usign LGPL license) while RTLinux do not.

RTAI license (LGPL) seems to be not too legal, in the sense that RTAI is based on GPL code and it is not possible to change the license without the permission of all the code copyright holders. In other words, derived work of GPL code can not be released under a different license.

[Note]The use of the word "seems" in the previous paragraph, is seen as a way to spread FUD. It is not my intention, but I'm not a layer and it is difficult to give a more detailed interpretation of the reality.

RTLinux:

Web:fsmlabs.com
Mail-list: Active, posted questions got answered.
Main back-company: FSMLabs (fsmlabs.com)

RTAI:

Web: www.rtai.org
Mail-List: Active, posted questions got answered.
Suporting-companies: DENX Software Engineering (www.denx.de), Pengutronix (www.pengutronix.de) and Sysgo Realtime Solutions (www.sysgo.de)

RTLinux:

Implemented as a POSIX 1003.13 "minimal realtime operating system". The internal design was driven by the POSIX requirements.

RTAI:

RTAI supports it's own API derived from the RTLinux V1 API. The new features (message queues, mailboxes, etc) do not follow a coherent API and are incompatible between them. The same feature is implemented in several ways with different systems calls.

RTAI maintains compatibility with the V1 RTLinux API.

There is a POSIX module which provides POSIX 1003.1c. (PThreads) and 1003.1b (Pqueues).

RTLinux:

i386, PPC, ARM.

RTAI:

i386, MIPS, PPC, ARM, m68k-nommu.

RTLinux:

Development stalled at version "3.1" (May-2001). There are known bugs.
Linux kernel supported: 2.2.19, 2.4.4 (unofficially 2.4.18, no SMP support).

RTAI:

Actively developed. Linux kernel supported: 2.4.18

Comments:

Current Linux kernel is 2.4.18 (06-03-2003).

RTLinux:

1. Source-level debugging using GDB (DDD), with SMP support. From within the same target machine (no cross debugging required)
2. Tracer. It can trace kernel and application events.
3. POSIX trace.

RTAI:

1. kgdb: Source-level debugging from a host linked by a serial line.
2. Linux Trace Toolkit (LTT). It is a full-featured tracing system for the Linux kernel. It includes both the kernel components required for tracing and the user-level tools required to view the traces.

RTLinux:

Dynamic memory

By default, memory must be reserved before the real time thread has started. No dynamic memory allocation (malloc and free functions) are available.

Shared memory (RTLinux<->Linux)

MBUFF module (mbuff_alloc, mbuff_free) (Non-POSIX)

RTAI:

Dynamic memory

Support for dynamic memory allocation (rt_malloc, rt_free). This functions do not have a hard real time behavior. (Non-POSIX)

Shared memory (RTAI<->Linux)

1. MBUFF module (mbuff_alloc, mbuff_free) (Non-POSIX)
2. SHMEM module (rtai_malloc, rtai_free, rtai_kmalloc, rtai_kfree). Shmem is the RTAI version, developed by Paolo Mantagazza, which operates in much the same way but is dependant upon RTAI (Non-POSIX)

RTLinux:

FIFO

UNIX PIPE's like communication mechanism that can be used to communicate real-time process between them and also with normal linux user processes. (non-POSIX)

Mailboxes

Jerry Epplin IPC implementation (rt_mq_open, rt_mq_send, etc. ) (obsoleted) (Non-POSIX).

RTAI:

FIFO

UNIX PIPE's like communication mechanism. Same than in RTLInux. (non-POSIX)
Also provides a mechanism to create fifos by name.

Mailboxes

RTAI provides mailboxes. Messages are ordered in FIFO order. Different size messages of are allowed. Multiple senders and receivers can read and write messages to the same mailbox. There are several sending an receiving functions that provides a lot of flexibility: blocking, non-blocking, timed and whole/partial message.

Message queues

Provides four different (incompatible) intertask messages facilities

• Message queues compatible with POSIX 1003.b queues. The functionality is provided by the POSIX compatibility module.
• Small (4 bytes) synchronous messages. Messages can be priority ordered (determined by the compile time option MSG_PRIORD). Also there are timed sending and receiving primitives.
• Extended intertask messaging. Allows to use intertaks messages of any size.
• Remote Procedure Calls (RPCs) do the same thing that synchronous messages but the tasks are coupled awaiting a reply from the receiver. RPCs operate like complementary, send and receive message pairs. (Non-POSIX).

net_rpc

RTAI has extended its API to allow remote (other host) procedure call RPC. New API functions has been added with the following syntax: replace the first two letters of the function name (for example: given the rt_mbx_send(), the new function RT_mbx_send() has been added); and the new function has two new parameters: node and port. This feature do not comply with any communication standard.

Comments:

Too many incompatible, redundant and non standard features is not desirable. A RTOS should provide a simple API.

RTLinux:

Mutex

POSIX thread mutex variables. Provides PRIORITY_PROTECT protocol to deal with the priority inversion problem.

Condition variables

POSIX conditional variables.

Semaphores

POSIX semaphores.

RTAI:

Mutex

POSIX thread mutex variables. Provides PRIORITY_INHERIT protocol to cope with the priority inversion problem.

Conditional variables

POSIX conditional variables.

Semaphores

1. POSIX semaphores.
2. RTAI semaphores (rtf_sem_init, rtf_sem_wait, ...). The underling mechanism used to implement this semaphores is the FIFO's blocking features.

RTLinux:

User space real-time signals. Hardware interrupts and timed signals can be handled at user space signal handlers. It is not possible to call any Linux system calls nor RTLinux services from these handlers.

New features in the commercial distribution.

RTAI:

Well developed and tested. Caller LXRT. Same API than in kernel-space real-time, except the POSIX threads API.

There are three different implementations:

LXRT services (Soft-hard real time in user space)

Soft-hard real time in user space API
rt_task_init rt_sem_init rt_mbx_init rt_get_adr rt_register rt_get_name rt_drg_on_adr rt_drg_on_name rt_allow_nonroot_hrt

LXRT is a module that allows to use RTAI services in user space.

It is implemented by using RTAI process (being executed in the kernel space) for every LXRT process to perform the actual function call. Every time a LXRT process calls a RTAI API function, the parameters are copied to the RTAI peer process which is the one that to the real job.

It is possible to use any Linux System calls from a LXRT process, as well RTAI services.

LXRT user space tasks can interact with kernel space RTAI tasks by using the same API (share memory, send messages, use semaphores, etc.).

Extended LXRT (Hard-hard real time in user space)

Extended hard-hard user space API
rt_make hard_real_time rt_make_soft_real_time

This is an extension to the original LXRT facility. By calling the function rt_make hard_real_time(), a LXRT process gets even higher "priority". The process is scheduled by the Linux kernel but from a bottom-half (a bottom-half is an internal Linux kernel utility user to run interrupt services).

From: RTAI Programming Guide 1.0, page 139 (See references):
"The call to rt_make_hard_real_time allows to take a normal process out from the Linux running queue by calling schedule() after having queued the task to a bottom handler. When the bh runs, the task is scheduled as a hard real time module by lxrt_schedule(), and Linux will not know of it, while having it still in its process list."

These kind of processes should not call any Linux System call that can lead to a task switch, otherwise a deadlock may occur.

Mini RTAI LXRT

Mini RTAI functions
rt_task_init rt_task_delete rt_insert_tasklet rt_remove_tasklet rt_tasklet_exec rt_timer_init rt_remove_timer rt_set_timer_priority rt_set_timer_period rt_set_timer_handler rt_set_timer_data

This module provides a way to execute functions in a similar way the Linux Kernel 2.4 tasklets are executed. These taskslets are of two kinds: normal and timed (timer).

They can be used to implement Programmable Logic Controllers (PLC), and other simple real time sub-systems.

Blocking system calls (Linux or RTAI) should not be called from tasklets.

Currently, (RTAI 24.1.9) Mini_RTAI_LXRT and LXRT processes can not be jointly used.

Comments

There are two main reasons to run real-time applications in user space:

1. A task crash due to a programming bug do not affect the system stability. The crashing task is terminated, but the system remains safe. Which yields to faster development.
2. It is easier than inserting modules into the kernel. The real-time application is coded and started-up like a normal UNIX process.

By using POSIX API, it is possible to use the same API both in user and kernel space. Therefore, it is worthless to "port" (bring RTAI or RTLinux API) to user space process. In fact, using Xavier Leroy PThreads [Xavi] it is possible to implement soft RT applications in user space. Also, the new preemptible kernel reduces system latency, so it is possible to have soft real time in normal linux user processed.

By using user space processes all the advantages of running inside the kernel space are lost: more TLB misses and heavy system call mechanism.

In a recent discussion in the linux-kernel mailing list, Linus Torvalds talked about user space real-time (see marc.theaimsgroup.com)

...
With RTLinux, you have to split the app up into the "hard realtime" part (which ends up being in kernel space) and the "rest".

Which is, in my opinion, the only sane way to handle hard realtime. No confusion about priority inversions, no crap. Clear borders between what is "has to happen _now_" and "this can do with the regular soft realtime".
...

RTLinux:

• Design Whitepaper
• FSMLabs Lean POSIX for RTLinux
• Getting Started with RTLinux
• RTLinux API tutorial (Spanish)
• Installing RTLinux
• RTLinux Man Pages
• Most POSIX Threads books can be used to learn RTLinux.

RTAI:

• DIAPM RTAI - Beginner's Guide
• Dissecting DIAPM RTHAL-RTAI
• Hardware Floating Point Support in Interrupt Handlers
• The RTAI Manual Manual pages compendium
• RTAI Programming Guide

Although RTAI documentation is well written, it is not maintained. New features are not well documented while deprecated and not supported functionality is still in the documentation.

RTAI:

• Octave, MatLab & Simulink interface: generate code for RTAI from Simulink Workshop.
• Watchdog module: the higest priority periodic task that takes protects the systems against tasks errors. Available policies:
  • Nothing.
  • Resynchronise the task's frame.
  • Debug.
  • Stretch the period of the offending task.
  • Slip the offending task by forcibly suspending it for a percentage of its period.
  • Suspend the offending task.
  • Kill the offending task and remove all trace of it.
• System information through proc interface.
• Scripting language bindings: PERL.
• Xenomai: a project is to help creating emulators of traditional RTOS APIs that ease the migration from these systems to a GNU/Linux-based real-time environment. The following API's are available
  • pSOS+ emulator (complete)
  • VRTXsa emulator (complete)
  • VxWorks emulator (complete)
  • uITRON implementation (underway)

Both:

• RT-NET: UDP-IP.
• Comedi: linux control and measurement device interface.
• RT-Lab: Real-Time Linux Experiment Interface System.

The following table summarizes the differences. Features that are almost the same in both systems are not included:

1. Comparing real-time Linux alternatives
2. Comparing two approaches to real-time Linux
3. The RTAI Manual
4. RTAI Programming Guide 1.0
5. Tutorial del API de RTLinux (Spanish)
6. Xavier Leroy implementation of POSIX threads.
7. Real-time Linux sub-kernels, benchmarks... discussion

(1) non-free

The term "non-free" is preferred to "commercial" in this sentence (See FSF recommendations Commercial)

(2) for zero prize

The term "for zero prize" is preferred to "for free" in this sentence (See FSF recommendations For Free)

(3) sell

The term "imposing proprietary restrictions" is preferred to "sell" in this sentence (See FSF recommendations Sell Software)

This research was supported by the IST 2001-35102 OCERA project.