Real-Time Database Systems and Data Services: Issues and Challenges

1. Real-Time Database Systems and Data Services: Issues and Challenges Sang H. Son Department of Computer Science University of Virginia Charlottesville, Virginia 22903 son@cs.virginia.edu

2. Outline • Introduction: real-time database systems and real-time data services • Why real-time databases? • Misconceptions about real-time DBS • Paradigm comparison • Characteristics of data and transactions in real-time DBS • Origins of time constraints • Temporal consistency and data freshness • Time constraints of transactions • Real-time transaction processing • Priority assignment • Scheduling and concurrency control • Overload management and recovery

3. Outline (cont’d) • Advanced real-time applications • Active, object-oriented, main-memory databases • Flexible security paradigm for real-time databases • Embedded databases • Real-world applications and examples • Real-time database projects and research prototypes • BeeHive system • Research issues, trends, and challenges • Exercises

4. I. Introduction • Outline • Motivation: Why real-time databases and data services? • A brief review: real-time systems • Misconceptions about real-time DBS • Comparison of different paradigms: • Real-time systems vs real-time database system • Conventional DBS vs real-time DBS

5. Some Facts about Real-Time Databases • Fact 1: As the complexity of real-time systems and application is going up, the amount of information to be handled by real-time systems increases, motivating the need for database and data service functionality (as opposed to ad hoc techniques and internal data structures) • Fact 2: Conventional databases do not support timing and temporal requirements, and their design objectives are not appropriate for real-time applications • Fact 3: Tasks and transactions have both similarities and distinct differences, i.e., traditional task centric view is not plausible to real-time databases.

6. Something to Remember ... Real-time  FAST Real-time nonosecs or secs Real-time means explicit or implicit time constraints A high-performance database which is simply fast without the capability of specifying and enforcing time constraints are not appropriate for real-time applications

7. A system whose basic specification and design correctness arguments must include its ability to meet its time constraints. Its correctness depends not only on the logical correctness, but also on the timeliness of its actions. A Brief Review: Real-Time Systems

8. Review: Real-Time Systems • Characteristics of real-time systems • timeliness and predictability • typically embedded in a large complex system • dependability (reliability) is crucial • explicit timing constraints (soft, firm, hard) • A large number of applications • aerospace and defense systems, nuclear systems, robotics, process control, agile manufacturing, stock exchange, network and traffic management, multimedia computing, and medical systems • Rapid growth in research and development • workshops, conferences, journals, commercial products • standards (POSIX, RT-Java, RT-COBRA, etc)

9. Time Constraints v(t) Hard and firm deadline v0 d t v(t) Soft deadline v0 d1 d2 t

10. Databases for Real-Time Systems • Critical in real-time systems (any computing needs correct data) • real-time computing needs to access data: real-world applications involve time constrained access to data that may have temporal property • traditional real-time systems manage data in application-dependent structures • as systems evolve, more complex applications require efficient access to more data • Function of real-time databases • gathering data from the environment, processing it in the context of information acquired in the past, for providing timely and temporally correct response

11. What is a Real-Time Database? A real-time database (RTDB) is a data store whose operations execute with predictable response, and with application-acceptable levels of logical and temporal consistency of data, in addition to timely execution of transactions with the ACID properties. C. D. Locke Chief Scientist, TimeSys Co.

12. What is the gain of using RTDBS? • More efficient way of handling large amounts of data • Specification and enforcement of time constraints • Improved overall timeliness and predictability • Application semantic-based consistency and concurrency control • Specialized overload management and recovery • Exploitation of real-time support from underlying real-time OS • Reduced development costs

13. Gain of Using RTDBS (More Specifically) • Presence of a schema - avoid redundant data and its description • Built-in support for efficient data management - indexing, etc • Transaction support - e.g. ACID properties • Data integrity maintenance But … • Not all data in RTDB is durable: need to handle different types of data differently (will be discussed further later) • Correctness can be traded for timeliness - Which is more important? Depends on applications, but timeliness is more important in many cases • Atomicity can be relaxed: monotonic queries and transactions • Isolation of transactions may not always be needed • Temporally-correct serializable schedules  serializable schedules

14. Objectives of Real-Time Databases • Correctness requirements: • consistency constraints • time constraints on data and transactions • Objectives • timeliness and predictability: dealing with time constraints and violations • Performance goals: • minimize the penalty resulting from actions either delayed or not executed in time • maximize the value accruing to the system from actions completed in time • support multiple guarantee levels of quality for mixed workloads

15. Why Not Using Conventional Databases? • Inadequacies of conventional databases: • poor responsiveness and lack of predictability • no facility to support for applications to specify and enforce time constraints • designed to provide good average response time, while possibly yielding unacceptable worst case execution time • resource management and concurrency control in conventional database systems do not support the timeliness and predictability

16. Differences from Traditional Databases • Traditional database systems • persistent data and consistency constraints • efficient access to data • transaction support: ACID properties • correct execution of transactions in the context of concurrent execution and failure • designed to provide good average performance • Databases for real-time systems • temporal data, modeling a changing environment • response time requirements from external world • applications need temporally coherent view • actively pursue timeliness and predictability

17. Misconceptions on Real-Time Databases....

18. “Advances in hardware till take care of RTDBS requirements.” fast (higher throughput) does not guarantee timing constraints increase in size and complexity of databases and hardware will make it more difficult to meet timing constraints or to show such constraints will be met hardware alone cannot ensure that transactions will be scheduled properly to meet timing constraints or data is temporally valid transaction that uses obsolete data more quickly is still incorrect “Real-time computing is equivalent to fast computing.” minimizing average response time vs satisfying individual timing constraints predictability, not speed, is the foremost goal Misconceptions about RTDBS (1)

19. “Advances in standard DBS technology will take care of RTDB requirements.” while novel techniques for query processing, buffering, and commit protocols would help, they cannot guarantee timeliness and temporal validity time-cognizant protocols for concurrency control, commit processing and transaction processing are mandatory “There is no need for RTDBS because we can solve all the problems with current database systems” adding features such as validity intervals and transaction deadlines to current database systems is in fact moving towards to developing a real-time database system such approach (adding features in ad hoc manner) will be less efficient than developing one from the ground up with such capabilities Misconceptions about RTDBS (2)

20. “Using a conventional DBS and placing the DB in main memory is sufficient.” although main-memory resident database eliminate disk delays, conventional databases have many sources of unpredictability, such as delays due to blocking on locks and transaction scheduling increases in performance cannot completely make up for the lack of time-cognizant protocols in conventional database systems “A temporal database is a RTDB.” while both of temporal DB and RTDB support time-specific data operations, they support different aspects of time in RTDB, timely execution is of primary concern, while in temporal DB, fairness, resource utilization, and ACID properties of transactions are more important Misconceptions about RTDBS (3)

21. “Problems in RTDBS will be solved in other areas.” some techniques developed in other areas (e.g., RTS and DBS) cannot be applied directly, due to the differences between tasks and transactions, and differences in correctness requirements there are unique problems in RTDBS (e.g., maintaining temporal consistency of data) “RTDBS guarantee is meaningless unless H/W and S/W never fails” true, in part, due to the complexity involved in predictable and timely execution it does not justify the designer not to reduce the odds of failure in meeting critical timing constraints Reference:Stankovic, Son, and Hansson, ‘Misconceptions About Real-Time Databases’, IEEE Computer, June 1999. Misconceptions about RTDBS (4)

22. Conventional Databases: Logical consistency ACID properties of transactions: Atomicity Isolation Consistency Durability Data integrity constraints Real-Time Database Systems: Logical consistency ACID properties (may be relaxed) Data integrity constraints Enforce time constraints Deadlines of transaction External consistency absolute validity interval (AVI) Temporal consistency relative validity interval (RVI) Conventional vs. Real-Time Databases: Correctness Criteria

23. Real-time Systems vs. RTDBS Real-time systems • Task centric • Deadlines attached to tasks Real-time databases • Data centric • Data has temporal validity, i.e., deadlines also attached to data • Transactions must be executed by deadline to keep the data valid, in addition to produce results in a timely manner

25. II. Characteristics of Data and Transactions • Outline • The origin of time constraints • Types of time constraints • Real-time data and temporal consistency • Real-time transactions

26. The Origin of Time Constraints • Meeting time constraints is of paramount importance in real-time database systems. Unfortunately, many of these time constraints are artifacts. • If a real-time database system attempts to satisfy them all, it may lead to an over-constrained or over-designed system. Issues to be discussed: 1. What are the origins of (the semantics of) time constraints of the data, events, and actions? 2. Can we do better by knowing the origins of time constraints? 3. What is the connection between time-constrained events, data, and real-time transactions?

27. Example #1: Objects on Conveyor Belts on a Factory Floor Recognizing and directing objects moving along a set of conveyer belts on a factory floor. • Objects’ features captured by a camera to determine its characteristics. • Depending on the observed features, the object is directed to the appropriate workcell. • System updates its database with information about the object.

28. Example #1 (cont’d) • Features of an object must be collected while the object is still in front of the camera. • “Current” object and features apply just to the object in front of the camera • Lose validity once a different object enters the system. • Object’s features matched against models in database. • Based on match, object directed to selected workcell. • Alternative: discard object and later bring it back again in front of the camera.

29. Example #2: Air Traffic Control System makes decisions concerning • incoming aircrafts’ flight path • the order in which they should land • separation between landings Parameters: position, speed, remaining fuel, altitude, type of aircrafts and current wind velocity. Aircraft allowed to land => subsequent actions of this aircraft become critical: cannot violate time constraints Alternative: Ask aircraft to assume a holding pattern.

30. Factors that Determine Time Constraints Focus: externally-imposed temporal properties • The characteristics of the physical systems being monitored and controlled: • speed of the aircraft, speed of conveyer belt, temperature and pressure • The stability characteristics as governed by its control laws: • servo control loops of robot hands, fly-by-wire, avionics, fuel injection rate • Quality of service requirements: • sampling rates for audio and video, accuracy requirement for results • Human (re)action times, human sensory perception: • time between warning and reaction to warning Events, data and actions inherit time constraints from these factors • They determine the semantics (importance, strictness) of time constraints.

31. All Time Constraints are Artifacts? May be not all of them, but even many externally-imposed constraints are artifacts: • Length of a runway or speed of an aircraft - determined by cost and technology considerations; • Quality of service requirements - decided by regulatory authorities; • Response times guaranteed by service providers - determined by cost and competitiveness factors

32. Designer Artifacts Subsequent decisions of the database system designer introduce additional constraints: • The type of computing platform used (e.g. centralized vs. distributed) • The type of software design methodology used (e.g., data-centric vs. action-centric) • The (pre-existing) subsystems used in composing the system • The nature of the actions (e.g., monolithic action vs. graph-structured or triggered action) Time constraints reflect the specific design strategy and the subsystems chosen as much as the externally imposed timing requirements

33. Decisions on Time Constraints • Difficulty of optimal time constraints • Determining all related time constraints in an optimal fashion for non-trivial systems is intractable => divide and conquer (and live with acceptable decisions) • Multi-layer decision process • The decisions made at one level affect those at the other level(s) • While no decision at any level is likely to be unchangeable, cost and time considerations will often prevent overhaul of prior decisions

34. Decisions on Time Constraints (2) • Decisions to be made • Whether an action is periodic, sporadic, or aperiodic • The right values for the periods, deadlines, and offsets within periods • Importance or criticality values • Flexibility (dynamic adaptability) of time constraints

35. Time Constraints of Events Three basic types of time constraints 1. Maximum: delay between two events Example: Once an object enters the view of the camera, object recognition must be completed within t1 seconds 2. Minimum: delay between two events Example: No two flight landings must occur within t2 seconds 3. Durational: length of an event Example: The aircraft must experience no turbulence for at least t3 seconds before the “seat-belt sign” can be switched off once again Constraints can specify between stimulus and response events (max, min, and duration between them can be stated)

36. Time Constraints of Events (2) • The maximum and minimum type of time constraints of recurring (stimulus) events: rate-based constraints • Time constraints determine the constraints on transactions: • Rate-based constraints -> periodicity requirements for the corresponding actions • Time constraints relating a stimulus and its response -> deadline constraints • Specifications of minimal separation between response to a stimulus and the next stimulus -> property of the sporadic activity that deals with that stimulus

37. Data in Real-Time Database Systems • Data items reflect the state of the environment • Data from sensors - e.g., temperature and pressure • Derived data - e.g., rate of reaction • Input to actuators - e.g., amount of chemicals, coolant • Archival data - e.g., history of (interactions with) environment • Static data as in conventional database systems

38. Time Constraints on Data • Where do they come from? • state of the world as perceived by the controlling system must be consistent with the actual state • Requirements • timely monitoring of the environment • timely processing of sensed information • timely derivation of needed data • Temporal consistency of data • absolute consistency: freshness of data between actual state and its representation • relative consistency: correlation among data accessed by a transaction

39. Static Data and Real-Time Data • Static data • data in a typical database • values not becoming obsolete as time passes • Real-time (Temporal) data • arrive from continuously changing environment • represent the state at the time of sensing • has observed time and validity interval • users of temporal data need to see temporally coherent views of the data (state of the world) • When must the data be temporally consistent? • ideally, at all times • in practice, only when they are used by transactions

40. An Example • Data object is specified by (value, absolute validity interval, time-stamp) • Interested in {temperature and pressure} with relative validity interval of 5 • Let current time = 100 temperature = (347, 10, 95) and pressure = (50, 20, 98) -- temporally consistent temperature = (347, 10, 98) and pressure = (50, 20, 91) -- temporally inconsistent

41. What Makes the Difference? We have a set of predicates to be satisfied by data • Why not use standard integrity maintenance techniques? • Not executing a transaction will maintain logical consistency, but temporal consistency will be violated • Satisfy logical consistency by CC techniques, such as 2PL • Satisfy temporal consistency by time-cognizant transaction processing AVI and RVI may change with system dynamics, e.g. mode changes

42. Time Constraints Associated with Actions Time constraints dictate the behavior of the environment • constrain the rates and times at which inputs arrive at the system • Example: seek permission to land only when aircraft is 10 mins from airport Time constraints prescribe performance of the system • dictate the responsiveness of the system to these inputs • Example: respond to a “landing request” within 30 seconds Time constraints are imposed to maintain data temporal consistency • Example: actions that update an aircraft’s dynamic parameters in 1 second

43. Distinct Types of Transactions • Write-only transactions (sensor updates): obtain state of the environment and write into the database • store sensor data in database (e.g., temperature) • monitoring of environment • ensure absolute temporal consistency • Update transactions (application updates) • derive new data and store in database • based on sensor and other derived data • Read-only transactions • read data, compute, and report (or send to actuators)

44. Time Constraints on Transactions • Time constraints on transactions • some come from the need to maintain temporal consistency of data • some come from the requirements on reaction time, dictating the responsiveness of the system • some come from the designer’s choice, specifying the rates and times at which inputs arrive at the system • transaction’s value depends on completion time

45. Types of Time Constraints Based on type of time constraints: • Periodic - Every 10 secs Sample wind velocity - Every 20 secs Update robot position • Aperiodic - If temperature > 1000 within 10 secs add coolant to reactor Based on Value: • Hard: must execute before deadline • Firm: abort if not completed by deadline • Soft: diminished value if completed after deadline

46. Dealing with Time Constraint Violations Large negative penalty => a safety-critical or hard time constraint • typically arise from external considerations • important to minimize the number of such constraints No value after the deadline and no penalty accrues => a firm deadline • typically, alternatives exist Result useful even after deadline => a soft deadline • system must reassign successors’ parameters - so that the overall end-to-end time constraints are satisfied Firm and soft time constraints offer the system flexibility - not present with hard or safety-critical time constraints

47. Examples of Time Constraints Specified using ECA (Event-Condition-Action) Rules The time constraints can be specified using ECA rules ON (10 seconds after “initiating landing preparations”) IF (steps not completed) DO (within 5 seconds “abort landing”) ON (deadline of “object recognition”) IF (action not completed) DO (“increase importance, adjust deadlines”) ON (“n-th time violation within 10 secs”) IF (crisis-mode) DO (“drop all non-essential transactions”)

48. Time Constraints: Discussion • Understand the issues underlying the origin and semantics of time constraints • not all deadlines are “given.” • need ways to deriving time constraints (and semantics) in the least stringent manner • flexibility afforded by derived deadlines must be exploited • deadline violation must also be handled adaptively • Control strategies can be specified by ECA rules

50. III. Real-Time Transaction Processing Outline Priority assignment Scheduling paradigms Priority inversion problem Concurrency control protocols Predictability issues Overload management and recovery