Support for context-driven applications in Ambient Intelligence environments

1. Support for context-driven applications in Ambient Intelligence environments PhD Defense Davy Preuveneers Promotor: Prof. Dr. ir. Y. Berbers July 1, 2009

2. Introduction Middleware Contributions Evaluation Conclusions Context-driven applications • Applications • That are aware of my current situation and context • That autonomously adapt their behavior accordingly • Using context • Who is calling me? • What time is it? • Where am I? • What am I doing? • Adapt behavior • Ring or vibrate • Redirect to other phone • Leave message

3. Introduction Middleware Contributions Evaluation Conclusions Ambient Intelligence • Personalized digital environments • Sensitive and responsive to presence of people

4. Introduction Middleware Contributions Evaluation Conclusions Ambient Intelligence Goals of Ambient Intelligence (AmI): • Devices work in concert • Support people with everyday activities • In a natural way • Using information and intelligence • Hidden in the network connecting these devices Ambient Intelligence Computing and Communication Intelligent Interfaces Context Awareness

5. Overview • Introduction • Middleware for context-driven applications • Contributions • Context modeling and reasoning • Middleware for context management • Context-driven application adaptation • Context-aware application collaboration • Evaluation • Conclusions and future work

6. Examples Context Types Human Concern Introduction Middleware Contributions Evaluation Conclusions Context-driven applications Auto Cell Phone Off In Meetings Identity Time Location Proximity Activity History … Convenience Tag Photos Finding Info Calendar Reminders Memory Health Alert Safety Auto Lights On / Off Efficiency

7. Introduction Middleware Contributions Evaluation Conclusions Problem statement • Developing context-driven applications is hard and complex • Availability of context sources

8. Introduction Middleware Contributions Evaluation Conclusions Problem statement • Developing context-driven applications is hard and complex • Availability of context sources • Deduction of high-level from low-level context Look up cell tower ID or IP address

9. Introduction Middleware Contributions Evaluation Conclusions Problem statement • Developing context-driven applications is hard and complex • Availability of context sources • Deduction of high-level from low-level context • Adaptation of applications to changing context Look up cell tower ID or IP address

10. Introduction Middleware Contributions Evaluation Conclusions Motivating scenario: E-health • Diabetes self-care: • Eat the right foods • Get and log daily physical activity • Take medications as prescribed • Test blood glucose regularly

11. Introduction Middleware Contributions Evaluation Conclusions Motivating scenario: E-health • Diabetes self-care with mobile phones: • People forget or don’t keep a detailed log • Recalling similar previous situations becomes difficult • Create a context-driven recommender application • Benefits of the application • Time and location monitoring • User input on food consumption and insulin dosage • Find correlations between time/location and activities • Augment blood glucose level logs with contextual data • Use context to find similar situations in the past

12. Overview • Introduction • Middleware for context-driven applications • Contributions • Context modeling and reasoning • Middleware for context management • Context-driven application adaptation • Context-aware application collaboration • Evaluation • Conclusions and future work

13. Applications Introduction Middleware Contributions Evaluation Conclusions Middleware A middleware approach Benefits: • Hide context sensor heterogeneity • Manage context efficiently • Simplify adaptation of mobile applications • Context-aware discovery and collaboration Operating System

14. Introduction Middleware Contributions Evaluation Conclusions A middleware approach Context • Current time, location, activity, preferences • People, services and devices around me • … DEFINITION “Any information that can be used to characterize the situation of an entitiy, where an entity can be a person, place or physical or computational object.”, Dey at al. (2001)

15. Introduction Middleware Contributions Evaluation Conclusions A middleware approach Middleware requirements: • Implement a semantically expressive and non-ambiguous context representation

16. Introduction Middleware Contributions Evaluation Conclusions A middleware approach Middleware requirements: • Implement a semantically expressive and non-ambiguous context representation • Deployment on resource constrained hardware in mobile ad hoc networks • PXA271 XScale 13-416 MHz • 256kB SRAM, 32 MB SDRAM • 32MB Flash

17. Introduction Middleware Contributions Evaluation Conclusions A middleware approach Middleware requirements: • Support application adaptation at runtime • PXA271 XScale 13-416 MHz • 256kB SRAM, 32 MB SDRAM • 32MB Flash

18. Introduction Middleware Contributions Evaluation Conclusions A middleware approach Middleware requirements: • Support application adaptation at runtime • Context-aware interoperability in dynamic and mobile device and service networks ISTAG Scenarios of Ambient Intelligence in 2010; Final Report, Feb. 2001.

19. Overview • Introduction • Middleware for context-driven applications • Contributions • Context modeling and reasoning • Middleware for context management • Context-driven application adaptation • Context-aware application collaboration • Evaluation • Conclusions and future work

20. Context modeling and reasoning Context management Context-driven adaptation Context-driven collaboration • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions Contributions

21. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 1. Context modeling and reasoning CoDAMoS context ontology in OWL Major concepts • User • Environment • Platform • Services Relevance and Ambiguity • Accuracy • Precision • Freshness • Spatial coverage • Sampling rate • …

22. Semantic Location is a Building Passage Room Floor • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 1. Context modeling and reasoning is a Auditorium MeetingRoom Kitchen instance 00.255 interpreted Graph Shape has has is a Node Edge Circle Rectangle has shape instance has instance Qualitative Quantitative Rect_1 Node_1

23. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 1. Context modeling and reasoning Ontologies in diabetes use case: • `Upper’ CoDAMoS context ontology • Common general purpose concepts • E.g. time, location, user preferences (base insulin plan) • Domain specific ontologies • Various user activities and tasks • Classified according to exercise level (none, light, moderate and high) • E.g. Sleeping, hobbies, profession, sports, … • Food and nutritional information • Convert amount of food into numeric values • E.g. carbohydrates, proteins, calories, fat, …

24. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 1. Context modeling and reasoning Efficient semantic matching algorithms • Semantic matching with subsumption or inheritance relationships • “Is one concept a subclass of another concept?” • Slow matching for large ontologies 3 classes 160 classes Shape is a Circle Rectangle

25. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 1. Context modeling and reasoning Efficient semantic matching algorithms • Compile ontologies ahead-of-time • Incremental encoding, partial loading • Orders of magnitude better (cpu/memory) than standard ontology reasoners hasProjector Room γ(D): 7 γ(A): 2 A D Meeting Room Lab Auditorium F B E γ(F): 2×13 γ(E): 2×7×11 γ(B): 2×3 Video Conference Computer Lab G C γ(G): 2×13×17 γ(C): 2×3×5

26. Ontology Binary Matrix HashSet (avg) Krall Prime (max) Prime (avg) • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions SUMO 630 244 30 83 42 1. Context modeling and reasoning Wine & Food 133 188 33 53 23 Pizza 99 194 37 40 23 Gene Ontology 20945 301 151 361 82 Java 1.3 5438 140 68 112 31 OpenCyc 25565 1166 350 681 272 Compared with other encodings in terms of bits per class

27. Context modeling and reasoning Context management Context-driven adaptation Context-driven collaboration • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions Contributions

28. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Main concerns (vs. other systems): • Manage context w/o wasting resources • Keep overhead for applications low END USER APPLICATIONS APPLICATION ADAPTATION USER PROFILE STORAGE ACQUISITION TRANSFORM & INFERENCE CONTEXT MANAGING COMPONENTS BILLING POLICY ENFORCEMENT DATABASE OTHER ENABLING SERVICES SENSORS

29. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Basic design principles: • Input: facade pattern • Transformation: pipe and filter • From: CID = 283F and LAC = 3070 • To: Lat = 50.8717 and Lon = 4.6875 • To: Current weather Pipe Pipe Pipe GSM Network Filter [infer] Filter [infer] T Geo-coordinates CID / LAC Weather

30. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Basic design principles: • Input: facade pattern • Transformation: pipe and filter • Storage and pub/sub: blackboard • Reason engines: chain of responsibility Reasoning problem App. Abstract Handler Spatial Reasoner Temporal Reasoner Blackboard Semantic Reasoner

31. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Managing context in diabetes use case: • Food consumption • Location and activity awareness Food consumption • Convert user input on meal into numeric values with nutrition database component • E.g.: 100 gr French fries • 38 gram carbohydrates • 40 gram proteins • 456 kilocalories • 16 gram fat Nutrition Table Carbs, Fat Calories, … Food Quantity

32. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Location awareness using cell towers • Graph of time-stamped cell tower transitions

33. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Location awareness using cell towers • Cell tower handover transition probability • Markov chain of first order

34. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Location awareness using cell towers • Handovers depend on current Sj and predecessor state Si • Total duration of connectivity to tower is used for probability of being in that state • Compute probability of connecting to a sequence of three cell towers: O = S2, S4, S7

35. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 2. Middleware for context management Activity awareness based on location • Hard to recognize without additional sensors • Only interested in the exercise level of activity • None: e.g. sleeping • Light: e.g. socializing, hobbies, watching TV • Moderate: e.g. profession • High: e.g. profession, sport • Use Hidden Markov Model • 4 exercise levels as hidden states • Discretization of blood glucose as 15 observable states • Correlate activities with locations

36. Context modeling and reasoning Context management Context-driven adaptation Context-driven collaboration • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions Contributions

37. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions Motivating scenario: E-health • Component-based e-diary application • (Non-) functional properties of each component are syntactically and semantically specified

38. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 3. Context-driven application adaptation Adapt GUI component to resources • Screen size, memory, CPU, … • Graphics support (e.g. virtual machine) Simple text report on small screen Graphical report on bigger display

39. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 3. Context-driven application adaptation 1. Collect context information • Hardware, resources, application requirements, dependencies 2. Find deployment • Component selection • Constraint solving and optimization 3. Adaptation at runtime • Add, remove, replace or relocate components • Change component configuration • Behavior defined by policies • Detect and resolve adaptation conflicts

40. Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions 3. Context-driven application adaptation Application adaptation policies: • Declarative event-condition-action rules that govern application behavior • If(battery.load < 30 %) Then disableComponent(‘BluetoothDiscovery’) • If(battery.load < 15 %) Then invoke(‘UserInterface.warning()’) • Application specific behavior: • Normal blood glucose: 80 – 150 mg/dL • < 50 mg/dL: hypoglycemic diabetic coma !!! • > 500 mg/dL: hyperglycemic diabetic coma !!!

41. Context modeling and reasoning Context management Context-driven adaptation Context-driven collaboration • Introduction • Middleware • Contributions • Modeling • Management • Adaptation • Collaboration • Evaluation • Conclusions Contributions

42. Overview • Introduction • Middleware for context-driven applications • Contributions • Context modeling and reasoning • Middleware for context management • Context-driven application adaptation • Context-aware application collaboration • Evaluation • Conclusions and future work

43. Context-aware adaptation Diabetes health-care of Instant Messaging client recommender application IWT SBO CoDAMoS Middleware for context-aware applications Introduction Middleware Contributions Evaluation Conclusions Evaluation – Case studies

44. ITEA LOMS Middleware support as a service Introduction Middleware Contributions Evaluation Conclusions Evaluation – Case studies

45. Modeling and simulation support Introduction Middleware Contributions Evaluation Conclusions Evaluation - Tools

46. > 20 international publications • Towards an extensible context ontology for Ambient Intelligence • Encoding semantic awareness in resource-constrained devices • Automated context-driven composition of pervasive services to alleviate non-functional concerns • Adaptive context management using a component based approach Introduction Middleware Contributions Evaluation Conclusions Evaluation - Publications

47. Overview • Introduction • Middleware for context-driven applications • Contributions • Context modeling and reasoning • Middleware for context management • Context-driven application adaptation • Context-aware application collaboration • Evaluation • Conclusions and future work

48. Model context with ontologies • Despite performance issue • Standardization (W3C Working Draft 2009) • Component-based design is key • Syntactically and semantically described black boxes • Simplify local application adaptation at runtime • Optimize workflow in context middleware • Our approach is not the holy grail • Fitness functions, evolutionary and genetic algorithms, distance metrics, learning, mixed-initiative, … • Trends towards software as a service, mash-ups, … • Context-oriented programming (COP), languages, … Challenges Contributions Publications Evaluation Conclusions Lessons learned

49. Evaluation and validation in various projects: • IWT SBO CoDAMoS • IBBT CROSLOCiS • ITEA LOMS Challenges Contributions Publications Evaluation Conclusions Conclusive remarks

50. Better algorithms and mechanisms for • Enhanced matching for hybrid context models • Predicting future contexts to drive proactive behavior • Comparing and measuring quality • Quality of Context (QoC) / Quality of Information (QoI) • Quality of Service (QoS) • Quality of Experience (QoE) • Automated testing framework • Simulate various situations • Unit testing for adaptive applications • Scalable (horizontal) collaboration • Interactions in large scale deployments • Delocalized sensing, decision making, and adaptation Challenges Contributions Publications Evaluation Conclusions Future work