LibeRTOS: A Flexibly Configurable and Highly Capable Platform for Industrial Automation

1. LibeRTOS: A Flexibly Configurable and Highly Capable Platform for Industrial Automation Information and Telecommunication Technology Center Industrial Advisory Board Meeting April 7, 2005 Douglas Niehaus niehaus@ittc.ku.edu

2. Overview • Linux for real-time and embedded systems • KURT-Linux Origins • Time keeping • Event scheduling • Performance Evaluation • LibeRTOS • Unified computation control • Configurability • Tool support • Conclusions

3. Real-Time and Embedded Linux • Linux is strongly increasing in popularity for real-time and embedded systems for several reasons • Open source ensures all parts of the system are accessible to the system developers • Local archive of entire system • Long term stability • Simple licensing restrictions: GPL and LGPL • Application software unrestricted • Low cost • Mature programming tools: GCC, GDB, binutils, busybox

4. Real-Time and Linux • Two basic methods for real-time computation using Linux • Two OS method • Single OS method • Two OS method establishes a RT OS on top of the hardware and then runs Linux under the RTOS control • Linux is generally a non-real-time best effort computation • RTLinux and RTAI use this approach • Relatively simple way to provide accurate scheduling support for simple RT computations • Drawback is that access to Linux services is more complex and RTOS services must all be implemented

5. Real-Time and Linux • Single OS approach modifies Linux internals to make it appropriate for real-time • Harder to do but it permits use of all Linux system services much easier • KURT-Linux was the first system to use this approach • Monta Vista and Timesys are commercial companies using this architectural approach • Several components of Linux need to consider RT • Interrupt handling, thread scheduling, concurrency control, preemption, kernel computations • Single OS solutions now approach two OS performance

6. KURT-Linux • Project started roughly eight years ago • Initial areas of concentration were: • Time keeping • Event scheduling • Performance evaluation • Simple programming model with explicit plan scheduling • Execution intervals for computations were explicitly specified on a cyclically repeating time line • Adequate for demonstrating microsecond resolution soft-real-time scheduling of fairly simple computations

7. KURT-Linux:Time Keeping • Vanilla Linux uses a periodic interrupt to keep time and to control the occurrence of explicitly scheduled events • KURT-Linux separated time keeping and event interrupts • Uses x86 Time Stamp Counter (TSC) as time standard • TSC ticks with CPU clock frequency • Can be calibrated against an Network Time Protocol standard within 30-40 parts per billion • On a 1 GHz Pentium machine • Provides fine grain local clock to applications and performance evaluation framework

8. KURT-Linux: Event Scheduling • Explicitly schedules each event interrupt using microsecond (x86) resolution PC timer chip • Simple and portable internal API: ARM, PPC • Resolution varies with the platform • Explicit plan model is a simple way to ensure that execution of periodic real-time computations is controlled accurately • Within microseconds to tens of microseconds depending on configuration options and CPU clock speed • However 102 outlier events still occurred per 107 events • Interrupt blocking and lack of kernel preemption

9. KURT-Linux:Performance Evaluation • Basic principle of real-time and embedded system development is that the accuracy of the performance evaluation methods must keep pace with the accuracy of the behavior control desired • KURT-Linux thus required performance evaluation methods much more accurate than vanilla Linux provided • Data Streams sub-system addresses this • Pseudo-device driver for kernel event recording • Utility library for application instrumentation • Unified application/OS performance evaluation and analysis methods permits end-to-end computation view

10. Data Stream Overview

11. Data Streams • Basic data type is the event • Time stamp (TSC), unique ID, context information • Instrumentation points in OS or application code produce an event when a thread of control cross them • Aggregate data types have lower overhead • Histograms and counters • Data added to aggregate by instrumentation points • Administrative events can include snapshots of aggregate data in the output event stream during an experiment as well as at the end • Instrumentation points can be individually enabled

12. Data Streams • Powerful post processing is key to analysis of often huge data sets gathered from and experiment • Post-processing filters are easy to use and to write • Python based implementation • GUI tool support for filter configuration • Constraint checking and scenario specific filters useful • Visualization is also quite useful for understanding higher level patterns of computation behavior • Detection of unusual situations • Data streaming across sockets especially useful for embedded systems

13. Pipeline Processing Visualization

14. LibeRTOS • Collaborative project with: Linutronix, several industrial automation customers in Germany, and Washington U. • Industrial-use version of KURT-Linux • KURT-Linux should remain research version and venue • LibeRTOS will receive updates from research efforts • LibeRTOS industrial driving problems enrich KURT-Linux research and development • LibeRTOS motivated feature development • Unified Computation Control: Group Scheduling Model • Configurability of many aspects of system semantics • Tool support

15. LibeRTOS: Unified Computation Control • Motivation: many application computations are implemented by groups of computational components • Emphasizes flexible representation and control of entire computations, including internal Linux components • Internal Linux computation types are “off-the-books” • Linux internal types: hard-IRQ, soft-IRQ, tasklets • Execution manifests as noise in thread scheduling models • Key Group Scheduling issues • Customizable scheduling of applications • End-to-end scheduling of all computation components • Precise fine-grain resource control and partitioning

16. Group Scheduling (GS) Model • Simple form of hierarchical scheduling permits composition of a wide variety of scheduling semantics • Middleware version for portability, OS for complete control • GS controls arbitrary groups of computational components • Threads: OS and Middleware versions • Hard-IRQ, soft-IRQ, tasklet, bottom-halves only under OS • Groups nest to form scheduling hierarchy • Scheduling Decision Function (SDF) associated with each group selects among members when asked • Application specific semantics can be created by implementing an SDF embodying those semantics

17. Group Scheduling Model • Generally we use decision trees but could be DAGs • Scheduling Decision Tree (SDT) is any graph subset • System Scheduling Decision function (SSDF) is SDT controlling system as a whole • This SSDF illustrates the “first refusal” semantics for GS under KURT-Linux/LibeRTOS • Only desired computations are under GS others default to Linux vanilla scheduler control for convenience

18. Group Scheduling Model • Emphasizes modularity and configurability • Controls selected computations • Does not require specification for all computations • SSDF can clearly represent desired semantics • Users can define SDFs • Functions in MW, Kernel modules in OS • Application centric and state-aware SDF use was the driving problem for this paper • SSDF integrates several scheduling constraints

19. Group Scheduling Target Examples • Many computations have multiplecomponents and it is the computations developers really wish to schedule • Ethernet Time Division Multiplexing on a LAN • Data Streams instrumentation effect reduction • Rodent behavioral experiment data collection & control • Real-time graphics calculation and display • Coordinate execution of threads doing calculation with execution of the X-server thread doing the display • Application-aware scheduling • Balanced progress of multiple multi-threaded pipelines for video processing, sensor fusion, or multi-track audio

20. Application Aware Driving Problem • Based on DARPA PCES driving problem • Critical and non-critical streams of data frames • Each stream is processed by a pipeline of threads • Processing time of each frame varies • By frame and pipeline stage • Balanced progress by many streams is desired • Multiple stream video monitoring • Sensor fusion and control loops in more general terms • Difficult to implement without an algorithm using application state information • System where progress of a set of computations matters

21. Application Aware Driving Problem

22. Scheduling Scenario 1 • Preference to Critical Streams • Frames move through pipelines as fast as possible • Receiving frames more important than processing • Non-critical streams under default Linux scheduling

23. Scheduling Scenario 2 • Preference to Critical Streams • Frames move through pipelines as fast as possible • Receiving frames more important than processing • Equal execution opportunity for non-critical streams

24. Is Round Robin Enough? • As implemented in the RR SDF, each non-critical stream received opportunity to use equal CPU share • We could have made it equal consumption • Under either policy, the variable processing time of each pipeline stage for each frame means that frame progress is not balanced • Such variable execution behavior is common to many groups of computations • Need for behavior to meet specific and unchanging constraints despite variable behavior also common • Application aware scheduling addresses this

25. Scheduling Scenario 3 • Preference to Critical Streams • Frames move through pipelines as fast as possible • Receiving frames more important than processing • Balanced Progress of non-critical Streams

26. Scenario 1 – Linux Scheduler

27. Linux Scheduler Conclusions • Linux scheduler is oblivious to non-critical stream • Existence • Pipeline component relationships • Frame progress over various streams thus varies widely • Complete failure to produce desired application behavior

28. Scenario 2 – Round Robin SDF

29. Round Robin Scheduling Conclusions • Non-critical streams under GS control • Control computations as groups of threads • Gives much more rational execution behavior • RR gives streams opportunities to use roughly equal portions of the left over CPU • RR is, however, unaware of progress by the application so progress is not balanced across streams • Complete failure to produce desired application behavior

30. Scenario 3 – Frame Progress SDF

31. Frame Progress Scheduling Conclusions • Substitution of the FP SDF makes the SSDF aware of the non-critical application state • Extremely tight balance between progress of all streams since the SDF has access to published application state • Application aware SDF is a simple and effective way to produce the desired application semantics

32. LibeRTOS: Configurability • Group scheduling provides flexible scheduling semantics configurability for precise computation component control • Interrupt enable/disable software semaphore control • Reduces interrupt response latency • Permits configurability of IRQ handling semantics and integration under the group scheduling model • Similar treatment of soft-IRQ and tasklet control permits configuration and integration under group scheduling • Loadable module can completely replace default vanilla Linux semantics with the desired programming model • Concurrency control in OS is a remaining issue

33. LibeRTOS: Tool Support • Data Streams • Event name space GUI editor • Post processing filter configuration GUI editor • Used as EECS 448 Software Engineering projects (Sp04) • Experiment data collection configuration GUI editor • Group Scheduling • Hierarchy configuration language, GUI editor, and system software interface to OS and MW group schedulers • GUI is EECS 448 (SPR05) driving problem • Eclipse Data Streams and Group Scheduling plugins • Under development

34. Project Support • Portions of the work described here have been supported by a variety of sources and projects including: • Sprint • NSF E&HS program • DARPA PCES program • DARPA ANTS program • Linutronix (Eclipse plugin prototypes) • NIH (Behavior Experiment support)

35. Conclusions • Linux is an increasingly popular platform for real-time and embedded systems for a variety of strong reasons • KURT-Linux/LibeRTOS addresses many of the existing constraints keeping Linux from addressing an even wider range of distributed real-time and embedded applications • Configurability of the system makes it applicable to a variety of application areas beyond real-time • Real-time scheduling and behavior semantics is now simply one configuration choice among many • Real-time embedded systems will remain a focus but will not be the only one

36. Future Work • Unification of concurrency control interface in Linux • Will simplify code, making analysis and study easier • Will provide clean structure within which to integrate OS concurrency control within LibeRTOS flexibly configurable framework • Will permit much cleaner structure within which to integrate OS concurrency control and group scheduling • Proxy execution • Simple accounting will track semaphore holders • Integration with group scheduling will prevent deadlock when preemption of threads holding locks is permitted

37. Future Work • Consolidation of end-to-end scheduling for computation components on real-time and embedded endsystems • Extension of end-to-end scheduling to sets of distributed computation components crossing endsystem boundaries • Integration of formal modeling of distributed real-time and embedded computation components to OS components • Extend current Washington University PhD thesis • Explore use of LibeRTOS component oriented configurability for HW/SW co-design of OS and application support and integration with group scheduling

38. Future Work • Apply KURT-Linux/LibeRTOS to various target applications areas • Behavioral science experiment control and data collection • Dr. Fowler at KU and Dr. Snyder at Washington University • Video processing: Dr Gauch (KU) and Dr. Fowler • Performance evaluation of GRID computations • CHARMM distributed molecular dynamics simulation • Drs. Clark , Kuczera, Houdonougbo • Apply group scheduling to improve performance