Can Real-Time Systems be built with Off the Shelf Components?

Real-Time Systems Laboratory University of Massachusetts, Amherst & Indian Institute of Technology, Bombay Can Real-Time Systems be built with Off the Shelf Components? Krithi Ramamritham

Talk Outline • Using off-the-shelf components for Real-Time applications? • Future application characteristics • Problems and challenges • Characteristics and limitations of OTS OSs • OTS component based solutions for distributed Real-Time applications • User-level scheduling of communicating RT tasks • Conclusions, recommendations, future

Industrial Control Environment Server PLC Operator Commands Device monitoring Field Network Video/audio Background Field Devices EWS Operator Stations NT M. Services M. Services M. Services M. Services M. Services Network LynxOS VxWorks PLC M. Services M.Services Field Network Field Devices

Future Application Characteristics • Interactions and communication • Human/device to human/device • Co-existence of multiple types of media • Small control and signal data • Periodic updates • Bursty file/page/image access and transfer • Continuous media • Computer and communication technology advances

What Are the New Challenges? • Communication centered • Operating System is no-longer standalone • Support for real-time and non-real-time co-existence • Integrated, end-to-end solutions required • Application-level control of system resources • Need to support new applications and technologies

System Research to Meet the Challenges Middleware Services Network Architecture Rate Control Network Interface Design Traffic Management Resource Manager Scheduling Algorithms Scheduler

Real-Time Spectrum Hard Soft

Real-Time OS Spectrum Hard Real-Time Operating System VxWorks, Lynx, QNX, ... Intime, HyperKernel, RTLinux (Windows NT, Linux) General-Purpose Operating System (Windows NT, Linux) Soft

Using General Purpose Operating Systems • GPOS offer some capabilities useful for real-time system builders • RT applications can obtain leverage from existing development tools and applications • Some GPOSs accepted as de-facto standards for industrial applications

Windows NT -- for RT applications? • Scheduling and priorities • Preemptive, priority-based scheduling non-degradable priorities priority adjustment • No priority inheritance • No priority tracking • Limited number of priorities • No explicit support for guaranteeing timing constraints

Windows NT -- for RT applications? (contd.) • Quick recognition of external events • Priority inversion due to Deferred Procedure Calls (DPC) • I/O management • Timers granularity and accuracy • High resolution counter with resolution of 0.8 sec. • Periodic and one shot timers with resolution of 1 msec. • Rich set of synchronization objects and communication mechanisms. • Object queues are FIFO

Goals - I • Evaluate the real-time capabilities of NT. • Identify areas where NT is (not) suitable for real-time applications. • Determine to what extent the unpredictable parts of NT can be “masked”. • Offer recommendations to designers using NT.

Priority Model Time-critical 31 Highest Real-time class 26 Above Normal Thread Level 25 Normal 24 Below Normal 23 Lowest 22 Idle 16 15 Time-critical Dynamic classes 15 High class 14 13 12 11 11 10 Normal class 9 8 7 6 5 Idle class 4 3 1 Idle 2

Thread Priority = Process class + level Time-critical 31 Highest Real-time class 26 Above Normal Thread Level 25 Normal 24 Below Normal 23 Lowest 22 Idle 16 15 Time-critical Dynamic classes 15 High class 14 13 12 11 11 10 Normal class 9 8 7 6 5 Idle class 4 3 1 Idle 2

Scheduling Interrupts Deferred Procedure Calls (DPC) System and user-level threads • Threads scheduled by executive. • Priority based preemptive scheduling.

Servicing an interrupt 1 Interrupt Service Routine DPC Routine Device interrupts DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. Interrupt Dispatch Table DPC DPC Interrupts DPC FIFO Queue DPC user-level threads

Servicing an interrupt 2 Interrupt Service Routine DPC Routine Transfer control to ISR DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. Interrupt Dispatch Table DPC DPC Interrupts DPC FIFO Queue DPC user-level threads 1 Device interrupts

Servicing an interrupt 3 Interrupt Service Routine DPC Routine Stop int.. and queue DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. Interrupt Dispatch Table DPC DPC Interrupts DPC FIFO Queue DPC user-level threads 1 2 Transfer control to ISR Device interrupts DPC

Servicing an interrupt Interrupt Service Routine DPC Routine DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. Interrupt Dispatch Table DPC DPC Interrupts DPC FIFO Queue DPC user-level threads Task level drops and DPC can execute 4 1 2 3 Transfer control to ISR Device interrupts Stop int.. and queue DPC

Servicing an interrupt Interrupt Service Routine DPC Routine DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. 5 Interrupt Dispatch Table Transfer control to driver’s DPC DPC DPC Interrupts DPC FIFO Queue DPC user-level threads 1 2 3 Transfer control to ISR Device interrupts Stop int.. and queue DPC Task level drops and DPC can execute 4

Servicing an interrupt Interrupt Service Routine DPC Routine DPC Power Failure . . . DPC DPC Device Driver Device X DPC FIFO Queue . . . Dispatch/DPC APC Normal Exec. Interrupt Dispatch Table DPC DPC Interrupts DPC FIFO Queue DPC user-level threads 1 2 3 Transfer control to ISR Device interrupts Stop int.. and queue DPC 5 6 Transfer control to driver’s DPC Execution of DPC routine Task level drops and DPC can execute 4

I/O Handling • I/O request is sent to device driver. • Device completes operation and interrupts. • Complete I/O request. Buffered I/O Direct I/O User’s space User’s space System System APC Device Device (disk, network) (Keyboard, mouse)

Prototype Software Architecture Highest Priority Normal Priority Operator Station Acquisition &Control Equipment Consumer Buffer Receiver Producer Heartbeat Timer Heartbeat Timer Command Real Video Ack. Real-Time Class

Performance Metrics • Round Trip Time (RTT) as seen by the operator input. • Rate of execution of sensor data processing entities. • Quality of the video output.

Workload • Number of sensor data streams (2-20). • Period of new sensor values (10-1000 ms). • Period of control messages (10-100 ms). • Amount of work done in processing data. • One Video and audio.

Operator Command Performance • Two 1KB Sensor data streams • 1 second update rate • 30 ms period for control messages • No work

Operator command Performance • NT Scheduler - same RT priority • 16 Sensor data streams (1KB) • Update rate (100, 200, 500, 1000 ms) • 90 ms period for control messages • work

Operator command Performance(user-level scheduling) • Rate Monotonic with limited levels • 16 Sensor data streams (1Kb) • Update rate (100, 200, 500, 1000 ms) • 50 ms period for control messages • work • Time-cognizant dispatcher + plan • 8 Sensor data streams (1Kb) • Update rate (100, 200, 500, 1000 ms) • 100 ms period for control messages • work

Design principles and recommendations: • Do not depend on NT scheduler to accomplish timing behavior in interactive applications. • Utilize user-level scheduling to achieve higher predictability. • If possible, characterize duration of I/O activity and its frequency. • Lock pages in memory for real-time threads. • Manage and control utilization of systems resources.

Use a General Purpose OS for RT? • It is possible to improve the predictability of real-time tasks. • It is not possible to “mask” all sources of unpredictability within NT “as is”. • Designer needs to be aware of the effects DPC queue on any user thread. • I/O handling. • Prototype application demonstrates the uses of the recommendations. K. Ramamritham, C. Shen, O. González, S. Sen, S.B. Shirgurkar. Using Windows NT for Real-Time Applications. In Proc. of the 4th IEEE Real-Time Technology and Applications Symposium, Denver, CO, June 1998.

Challenges in SupportingCommunicating Real-Time Tasks Middleware Services Rate Control Resource Manager Network Interface Design Network Architecture Traffic Management Scheduling Algorithms Scheduler

Communications Middleware • Transporting multiple media types, control, data, visual, image, alarms, video and audio • Integrating control with visual monitoring • Enabling plug-and-play • QoS management

Real-Time Channel-based Reflective Memory (RT-CRM) Node 1 Node 2 DPA_1 Writer Network ReMA ReMA DRA_n DPA_n Reader_i DPA = Data Push Agent DRA = Data Receive Agent ReMA = Reflective Memory Area

RT-CRM Operation Modes Node 1 Node 2 DPA_1 Writer Network ReMA ReMA DPA_n Reader_i Synchronous/Asynchronous Blocking / Non-blocking

Using MidART services Node 1 Node 2 Write Cmd DPA Process Cmd Result DPA Network • Synchronous mode

End-to-End QoS Provisions Memory-to-Memory Application-to-Application Node 1 Node 2 DPA_1 Writer Network ReMA ReMA DPA_n Reader_i

GPOS Problems • No priority inheritance • Among user processes/threads • No Priority tracking • From user to system/network threads • Limited number of priorities • No explicit support for guaranteeing timing constraints

Server-based User Level Scheduling Problem alleviated: No priority inheritance and limited priorities No priority tracking Hidden protocol stack Priority inversion Solution: Modified Dual Priority Scheduling

Dual Priority Scheduling (York) Real-time tasks after promotion time High Band Three Priority Bands Non-real-time tasks Middle Band Real-time tasks before promotion time Low Band

Dual Priority in MidART Time Critical Band Critical real-time tasks Real-time tasks after promotion time High Band Four Priority Bands Non-real-time tasks Middle Band Real-time tasks before promotion time Low Band

Communication Server Rate Controllers Queues ReMA • Request = actual messages • Queues limited priorities High Calculation of promotion time DPA Promoted ReMA Data Pusher To network Middle DPA ReMA Low @ promotion time DPA

Admission control • To calculate the schedulabilityT_i: Promotion Timei = Deadlinei - worst case response timei worst case response timei = worst case delay (wi) + release jitteri wi m+1 = Ci + So + [S wi m + Jj Cj ] j e hp(i) Tj Interference of high priority tasks Computation time Blocking time

Rate Control with Uniform Message Size • Bounding the priority inversion time due to non-preemptive message transmission. • The “optimal” message size is a platform dependent parameter. • Receiving node capability is the key - the known live-lock problem. • Match value between the system timer granularity and the data push latency. (E.g., in our case, 8KB)

Communication Server Rate Controllers Queues ReMA Uniform message size to allow transmission with fix preemption points High DPA Promoted ReMA STOP To network Middle DPA Data Pusher ReMA Low @ promotion time DPA

Communication Server Rate Controllers Queues ReMA Uniform message size to allow transmission with fix preemption points High DPA Promoted ReMA To network Middle DPA Data Pusher ReMA Low DPA

Can Real-Time Systems be built with Off the Shelf Components?