The Endeavour Expedition: 21st Century Computing to the eXtreme

1. The Endeavour Expedition:21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley Berkeley, CA 94720-1776

2. The Endeavour Expedition:21st Century Computing to the eXtreme • New Ideas • Systems Architecture for Vastly Diverse Computing Devices (MEMS, cameras, displays) • Wide-area “Oceanic” Data Information Utility • Sensor-Centric Data Management for Capture and Reuse (MEMS + networked storage) • Negotiation Architecture for Cooperating Components (Composable system architecture) • Tacit Knowledge Infrastructure to support High-Speed Decision-Making • Information Management for Intelligent Classroom Environments • Scalable Safe Component-based Design and UI Design Tools R. H. Katz, Principal Investigator, University of California, Berkeley • Impact • Enhancing human understanding by making it dramatically more convenient for people to interact with information, devices, and other people • Supported by a “planetary-scale” Information Utility, stress tested by applications in decision making and learning, achieved thru new methodologies for design, construction, and administration of systems of unprecedented scale and complexity Schedule Usability Studies & Early Tool Design Implementation of UI &Sys Design Tools Tools Release & Final Evaluations Design Methodologies Initial Application Implementation & Evaluation Refined Implementation & Final Evaluation Information Applications Initial Architectural Design & Testbeds Initial Evaluation & 2nd Gen Redesign Final Deployment & Evaluation Information Utility Initial Experiments & Revised Design Doc Initial Architectural Design Document Final Experiments & Architecture Docs Jun 99 Start Jun 00 Jun 01 May 02 End

3. Agenda • Project Motivation and Overview, Katz • System Architecture for eXtreme Devices, Culler • “Oceanic” Data Storage Utility, Kubiatowicz • Sensor-Centric Data Management, Hellerstein • Usability and User Interface Design, Landay • Remaining Options and Wrap-up, KatzQ&A by DARPA PMs expected throughout

4. Agenda • Project Motivation and Overview, Katz • System Architecture for Extreme Devices, Culler • Oceanic Data Storage Utility, Kubiatowicz • Sensor-Centric Data Management, Hellerstein • Usability and User Interface Design, Landay • Remaining Options and Wrap-up, Katz

5. Why “Endeavour”? • Endeavour: to strive or reach; a serious determined effort (Webster’s 7th New Collegiate Dictionary); British spelling • Captain Cook’s ship from his first voyage of exploration of the great unknown of his day: the southern Pacific Ocean (1768-1771) • Brought more land and wealth to the British Empire than any military campaign • Cook’s lasting contribution: comprehensive knowledge of the people, customs, and ideas that lay across the sea • “He left nothing to his successors other than to marvel at the completeness of his work.”

6. Expedition Goals • Enhancing understanding • Dramatically more convenient for people to interact with information, devices, and other people • Supported by a “planetary-scale” Information Utility • Stress tested by challenging applications in decision making and learning • New methodologies for design, construction, and administration of systems of unprecedented scale and complexity • Figure of merit: how effectively we amplify and leverage human intellect • A pervasive Information Utility, based on “fluid systems” to enable new approaches for problem solving & learning

7. Expedition Assumptions • Human time and attention, not processing or storage, are the limiting factors • Givens: • Vast diversity of computing devices (PDAs, cameras, displays, sensors, actuators, mobile robots, vehicles); No such thing as an “average” device • Unlimited storage: everything that can be captured, digitized, and stored, will be • Every computing device is connected in proportion to its capacity • Devices are predominately compatible rather than incompatible (plug-and-play enabled by on-the-fly translation/adaptation)

8. Expedition Challenges • Personal Information Mgmt is the Killer App • Not corporate processing but management, analysis, aggregation, dissemination, filtering for the individual • People Create Knowledge, not Data • Not management/retrieval of explicitly entered information, but automated extraction and organization of daily activities • Information Technology as a Utility • Continuous service delivery, on a planetary-scale, on top of a highly dynamic information base • Beyond the Desktop • Community computing: infer relationships among information, delegate control, establish authority

9. Driving Factors • Technology Push • Accelerating developments at the eXtremes: • Cluster-based compute/storage servers • MEMS sensor/actuators, CCD cameras, LCD displays, … • User Pull • More effective community leverage: the next power tool • Desire: • Enhanced interaction, ease of use • Easier configuration, “plug and play” • Less fragile tools, “always there” utility functionality

10. PC + Internet Shared servers/ Dedicated computing Remote access Internet Mail Server Web Server Workstation Shared servers/ Dedicated computing Remote access LAN FS PS Time Sharing Shared resources Remote access Remote Job Entry One at a time use Remote access to machine Batch processing One at a time use User comes to machine Computing Evolution:Distribution with Sharing Increasing Freedom from Colocation Increasing Sharing & Distribution Increasing Personalization Increasing Ratio of Computers:Users

11. Information Appliances: Many computers per person, MEMs, CCDs, LCDs, connectivity Information Appliances: Scaled down desktops, e.g., CarPC, PdaPC, etc. Revolution Evolution Evolved Desktops Servers: Integrated with comms infrastructure; Lots of computing in small footprint Servers: Scaled-up Desktops, Millennium Mem BANG! Display Smart Spaces Disk Camera Mem Display Display Display mProc Camera Smart Sensors Disk Keyboard Information Utility mProc Server, Mem, Disk Computing Revolution WAN PC Evolution Computing Revolution: Devices in the eXtreme

12. Information Devices Beyond desktop computers to MEMS-sensors/actuators with capture/display to yield enhanced activity spaces InformationUtility InformationApplications High Speed/Collaborative Decision Making and Learning Augmented “Smart” Spaces: Rooms and Vehicles Design Methodology User-centric Design withHW/SW Co-design; Formal methods for safe and trustworthy decomposable and reusable components “Fluid”, Network-Centric System Software Partitioning and management of state between soft and persistent state Data processing placement and movement Component discovery and negotiation Flexible capture, self-organization, and re-use of information Expedition Approach

13. High Speed Decision Making Learning Classroom E-Book Vehicles Applications Collaboration Spaces Info Appliances Human Activity Capture Generalized UI Support Event Modeling Transcoding, Filtering, Aggregating Statistical Processing/Inference Proxy Agents Negotiated APIs Self-Organizing Data Information Utility Interface Contracts Wide-area Search & Index Nomadic Data & Processing Wide-Area Data & Processing Automated Duplication Movement & Positioning Distributed Cache Management Stream- and Path-Oriented Processing & Data Mgmt Non-Blocking RMI Soft-/Hard-State Partitioning Laptop PDA Wallmount Display Camera Information Devices Smartboard MEMS Sensor/Actuator/Locator Handset

14. “Today, scientists and adventurers are lured by exploratory challenges to all regions of the globe and beyond. The explorer attempts routes of greater difficulty, the researcher perfects field techniques in remote locales. All are breaking new ground in isolated areas of the world usually under harsh conditions over extended periods of time.” http://www.expeditionresearch.org/english/ MEMS and hardware devices Scalable computing architectures Networked-oriented operating systems Distributed file systems Data management systems Security/privacy User interfaces Collaboration applications Intelligent learning systems Program verification Methodologies for HW/SW design/evaluation Needed Expedition Expertise

15. Alex Aiken, PL Eric Brewer, OS John Canny, AI David Culler, OS/Arch Joseph Hellerstein, DB Michael Jordan, Learning Anthony Joseph, OS Randy Katz, Nets John Kubiatowicz, Arch James Landay, UI Jitendra Malik, Vision George Necula, PL Christos Papadimitriou, Theory David Patterson, Arch Kris Pister, Mems Larry Rowe, MM Alberto Sangiovanni-Vincentelli, CAD Doug Tygar, Security Robert Wilensky, DL/AI Interdisciplinary, Technology-Centered Expedition Team

16. D e s I g n M e t h o d o l o g y Applications Rapid Decision Making, Learning, Smart Spaces: Collaboration Rooms, Classrooms, Vehicles Information Utility Fluid Software, Cooperating Components, Diverse Device Support, Sensor-Centric Data Mgmt, Always Available, Tacit Information Exploitation (event modeling) Base Program Information Devices Option 1: Sys Arch for Diverse Devices MEMS Sensors/Actuators, Smart Dust, Radio Tags, Cameras, Displays, Communicators, PDAs Option 2: Oceanic Data Utility Option 3: Capture and Re-Use Option 4: Negotiation Arch for Cooperation Option 5: Tacit Knowledge Infrastructure Option 6: Classroom Testbed Option 7: Scalable Safe Component-Based Design Organization: The Expedition Cube

17. Base Program: Leader Katz • Broad but necessarily shallow investigation into all technologies/applications of interest • Primary focus on Information Utility • No new HW design: commercially available information devices • Only small-scale testbed in Soda Hall • Fundamental enabling technologies for Fluid Software • Partitioning and management of state between soft and persistent state • Data and processing placement and movement • Component discovery and negotiation • Flexible capture, self-organization, info re-use • Limited Applications • Methodology: Formal Methods & User-Centered Design

18. Smart Space Testbed Eval. & Initial Design Tools 1st Gen Fluid R/T Environ. 2nd Gen Fluid R/T Environ. 1st Gen Persistent Fluid Store 2nd Gen Persistent Fluid Store Perf Eval 1st Gen Sensor-Centric Info Mgmt 2nd Gen Sensor-Centric IM Perf Eval 1st Gen Comp Neg. Protocols 2nd Gen Negotiation Cooperative Learning App Refine & Use Rapid Decision Making App Refine & Use Final Doc + Experiments Design Document + Early Evaluation Refined Doc + Experiments Base Program Schedule Year 1 Year 2 Year 3 Design Methodology Refined Tools & Flow Information Utility Information Applications

19. In-Depth Technical Presentations • Option 1: “Systems Architecture for Vastly Diverse Computing Devices”, David Culler, Subexpedition Leader • Option 2: “Implementation/Deployment of the Oceanic Data Information Utility”, John Kubiatowicz, Subexpedition Leader • Option 3: “Sensor-Centric Data Management for Capture and Reuse”, Joseph Hellerstein, Subexpedition Leader • Parts of Options 5, 6, 7: UI Design Cross Cut (UI design tools with applications to Tacit Information Extraction and Intelligent Classrooms), James Landay, Subexpedition Leader

20. Roll-Up of Remaining Options • Option 4: “A Negotiation Architecture for Cooperating Components”, Robert Wilensky, Subexpedition Leader • Option 7: “Scalable Safe Component-based Design”, Alberto Sangiovanni-Vincentelli, Subexpedition Leader • Option 8: “Scale-Up Field Trials”, Randy Katz, Subexpedition Leader • (Essential elements of Option 5: “Tacit Information Infrastruction and High Speed Decision Making and Option 6: “Information Management for Intelligent Classroom Environment” covered by James Landay

21. Option 4: Negotiation Architecture for Cooperating Components • Cooperating Components • Self-administration through auto-discovery and configuration among confederated components • Less brittle/more adaptive systems • Essential for all pieces of the Endeavour Utility Infrastructure • Negotiation Architecture • Components announce their needs and services • Service discovery and rendezvous mechanisms to initiate confederations • Negotiated/contractural APIs: contract designing agents • Compliance monitoring and renegotiation • Graceful degradation in response to environmental changes

22. The Problem: Configuration Difficulties • Individual computing components require considerable manual configuration • OS, software installation • Local data (solved by Oceanic storage!) • Configuration to access services • Today: small number of machines per individual--(manual) configuration limits • State (software/data) is inconsistent across machines • Manual updating is time-consuming • Degrades poorly in the presence of failure/change • Future: orders of magnitude more machines per individual--manual configuration completely infeasible

23. Solution: Negotiation Architecture for Auto-Configuration • Allow components to dynamically configure themselves by having components • Specify the potential services they provide, the terms and conditions, and to whom • Disseminate the availability of these services • Specify the services they require, and their terms and conditions • Discover other objects that provide required services • Allow objects to enter into multi-phase negotiations of contracts, committing to provide services under terms and conditions • Provide compliance monitoring services of contracts • Provide means for dealing with non-performing confederates

24. Plan for Success • Develop: • Language for specifying services, and their terms and conditions • Protocol for negotiating contracts between objects • Infrastructural services, including discovery, service availability dissemination, and compliance monitoring services • Means to adapt to a non-performing service • Emphasis on system architecture/easy of use: • E.g., standard, parameterized “boilerplate” contracts between components, with standard “compliance officers” • Some related issues: • Can we assure interesting adaptive properties? • Recent development: HP’s e”speak

25. Option 7: SafeComponent Design Leader Sangiovanni • Information Appliances as an application of hardware/software codesign • Specification based on “Co-design Finite State Machines” • Exploited in software for eXtreme devices • Formal methods to verify safety from faults • Safe partitioning of components into communicating subcomponents placed into the wide-area • Component-Based System Design • Composition of third party components to build systems • Can such components be trusted? • Correctness (Necula) • Security (Tygar)

26. Option 7: SafeComponent Design Leader Sangiovanni • Formal Specifications and Methods • Decomposition of components into safe partitionings of communicating subcomponents placed in the wide-area • HW/SW “Co-design Finite State Machines” • Exploits success in embedded software arena • Use in software for eXtreme devices • Compositions of third party components • JAVA or C/C++ modules • Use in Oceanic Store, Sensor-centric Data Mgmt • Formal methods to verify • Correctness/safety from faults • Trust and assurance

27. An Essential Problem forComponent-based Fluid Software • Cannot be trusted to behave as advertised • If unknown origin: must be assumed to be malicious • If known origin: can be erroneous or even malicious • Concerned with: • Extrinsic properties (non-semantic properties) • e.g., author, time of creation, 3rd party-endorsements, ... • Semantic properties (behaviors) • e.g., memory safety, lack of information flow, etc. • Needed: • Safety enforcement technologies • Design and development methodologies

28. Solution: Proof Carrying Code • Safety without sacrificing performance • Works for low-level languages, machine code, optimized code • Small trusted code base • Checking is easier than proving • No need to use (and trust) a compiler • Flexible and general (in principle) • Need a specification that captures the property of interest • Plus proof of that property for the untrusted code • “If you can prove it, PCC can check it!” • Install one checker for a multitude of policies • Use tools that certify their output • Delegate but do not trust • Effective way to debug the tools themselves!

29. Proof Heuristic-based certifier Untrusted code Proof Checker Yes/No Code Plan for Success • “Ratify” a broad set of safe programming practices for component-based systems • Failures point to programming errorsOR tool suggests convenient stylistic adjustmentsOR tool inserts run-time checks • Build toolkit for producing provably-safe native methods with off-the-shelf Java compilers • Build toolkit for certifying type safety of C programs Slow Quick

30. Security and Assurance • Two issues for apps based on mobile code • Protecting the remote host from the mobile code • Protecting the mobile code from the remote host! • Automatic generation of “best” security protocol • Ad hoc and temporal access control • Access control/security negotiation • Cryptographic hardware tokens as type of Information Device • How to evaluate, build, break tamper-resistant boundaries • Differential power analysis

31. Infrastructure Enables • Microactions/economics for resource control • Pervasive need for authentication • Enables resource management based on privileges • Rights management tagging • Who can operate on what under what conditions? • Design for survivability • Exploit resource control to mitigate denial of service attacks • All of this with privacy • Users control when and to whom information is released • Trade better system support for privacy

32. Plan for Success • One year • Synthesis of code for optimal security protocols • Toolkit for cryptographic key management for mobile code • Design of ad hoc and temporal access control • “Little TEMPEST” protection for hardware tokens • Three year • Integration with applications across Endeavour • Privacy analysis for high assurance mechanisms • Automatic or semi-automatic resource allocation using micro-auctions. • High survivability mechanisms

33. Option 8: Scaled-up Field TrialsLeader Katz • Testbed Rationale • Study impact on larger/more diverse user community • Higher usage levels to stress underlying architecture • Make commitment to true utility functionality • Increasing Scale of Testbeds • Building-Scale • Order 100s individuals • Campus-Scale • Order 1000s individuals • City-Scale • Order 100000 individuals

34. Network Infrastructure Experimental Testbeds Soda Hall IBM WorkPad Smart Dust Velo Nino LCD Displays MC-16 Motorola Pagewriter 2000 CF788 Pager WLAN / Bluetooth Smart Classrooms Audio/Video Capture Rooms Pervasive Computing Lab CoLab H.323 GW GSM BTS Wearable Displays TCI @Home Adaptive Broadband LMDS Millennium Cluster CalRen/Internet2/NGI Millennium Cluster

35. Summary: Putting It All Together Devices Utility Applications 1. eXtreme Devices 2. Data Utility 3. Capture/Reuse 4. Negotiation 5. Tacit Knowledge 6. Classroom 7. Design Methods 8. Scale-up Component Discovery & Negotiation Fluid Software Info Extract/Re-use Self-Organization Decision Making Group Learning

36. Conclusions • 21st Century Computing • Making people’s exploitation of information more effective • Encompassing eXtreme diversity, distribution, and scale • Computing you can depend on • Key Support Technologies • “Fluid software” computational paradigms • System and UI support for eXtreme devices • Pervasive, planetary-scale system utility functionality • Active, adaptive, safe and trusted components • New “power tool” applications that leverage community activity

37. Conclusions • Commercial spin, but direct relevance for many DoD future information technology requirements • Survivable, secure communications systems • System support for pervasive sensor networks • Fluid infrastructure support for: • CONUS + forward basing concepts • Rapid force deployment • Coalition leverage of shared/untrusted infrastructure • Information apps serve are examples for • Training • Mission planning • Battlespace decision making

38. Conclusions • Broad multidisciplinary team spanning the needed applications, evaluation, and system technology skills • Builds on many existing DARPA investments • BARWAN, Digital Libraries, iStore, Marco, MASH, MEMS, Ninja, Proof Carry Code,Tertiary Disk, …), • Integrates and extends these into a comprehensive information system architecture for 21st century computing • History of building large-scale prototypes, influencing industrial development

39. Back-Up