Managing R&D in a World of Radical Change

Managing R&D in a World of Radical ChangePresentation at SDSCFeb., 2002 Dr. Kenneth W. Neves Boeing Senior Technical Fellow & Director, Strategic Planning Engineering and Information Technology, Phantom Works Boeing, Seattle

Personal Background in Computing Leading Research Agendas Technical View of Some Disruptive Trends Scenario Based Planning Agenda

BA Mathematics, San Jose State 1968, Great Distinction; Hamilton Award, Outstanding Student in the School of Arts & Sciences, President Scholar MA, PhD Mathematics, Arizona Statue University, 1973, Full 5 Year NSF Fellowship Dissertation: Theoretical and Numerical Solutions of Functional Differential Equations with State Dependent Lags Research interests include algorithm science, software, computer architecture, performance & scalability, grid computing Hobbies: Private pilot, music, aerobic exercise Educational/Personal Background

The Early Years Mathematical Software: created production software libraries at B&W and Boeing (BCSLIB oldest (30 years) continually sustained subroutine library). Licensed: CDC, DT Labs, Cray, FPS (array processor company) Application experience: CFD, Structures, Electric Power Flow and Transient Stability, Electron Beam Aberrations (IBM), LOCA (Babcock & Wilcox) Experience • Research Activity • Publications in computer architecture, algorithm science (FDEs, sparse lin. Alg.), software lib’s • Performance scalability • NATO commissioned monograph CFD/Vector Computing w/ W. Gentszch • Established Boeing HPC program-won 2nd Gordon Bell award • Research Contracts • NSF PM of NW Internet • PM DARPA, NASA, NSF contracts • PM six EPRI contracts in power systems • PM Fujitsu contract to consult on VP series • Established R&D collabs: IBM, Hitachi, NEC, Dassault Systemes (SGI, Intel in work) • Professional Activities • NATO HPC Conference Series (5 papers in this Springer Verlag Series) • Founding SIC Applications Chair and Board of Dir. - Cray User Group (8 yrs) • At request of the DoC, led trade mission (sole speaker) to Japan, 500 attendees & TV simulcast • Tech Co-Chair, SIAM Conference and Editor, Electric Power Systems: Mathematical Challenges • Tutorials Chair, SC 92;editorial journal boards; co-founder of SIAM SC Inter. Gp w/Dongarra

Established NWACC and led a consortium of Boeing and 10 NW Universities to establish the NW sector of the internet (86-94). Led development & architecture under NSF funding led to a commercial ISP spin-off (now part of VERIO) established a research foundation of $10M in assets (founding director) Established Boeing’s relationship with NSF (trained university profs during transition to SC Centers) Key player in Boeing’s supercomputer time sales business in the 80’s, led researchers in nuclear, petrochemical, electric power (EPRI), and NSF sectors. Staff management, line management, and executive positions (current) 1994 became Director of CS Research which led Boeing’s efforts in: R&D Management Highlights • Knowledge discovery • Intelligent agents, intrusion detection, • Distributed computing, grid computing*, • Hybrid satellite and mobile networks, • HPC & Performance and scalability* , • Intelligent document technology, • Natural language processing & ontologies • Adaptive pattern recognition (NN) • Visualization, AR, VR, haptics*, *personally established

Established NWACC and led a consortium of Boeing and 10 NW Universities to establish the NW sector of the internet (86-94). Led development & architecture under NSF funding led to a commercial ISP spin-off (now part of VERIO) established a research foundation of $10M in assets (founding director) Established Boeing’s relationship with NSF (trained university profs during transition to SC Centers) Key player in Boeing’s supercomputer time sales business in the 80’s, led researchers in nuclear, petrochemical, electric power (EPRI), and NSF sectors. Staff management, line management, and executive positions (current) 1994 became Director of CS Research which led Boeing’s efforts in: COMPUTERWORLD, July 17, 2000 R&D Management Highlights • Knowledge discovery • Intelligent agents, intrusion detection, • Distributed computing, grid computing*, • Hybrid satellite and mobile networks, • HPC & Performance and scalability* , • Intelligent document technology, • Natural language processing & ontologies • Adaptive pattern recognition (NN) • Visualization, AR, VR, haptics*, *personally established

Personal Background in Computing Leading Research Agendas (philosophical) Technical View of Some Disruptive Trends Scenario Based Planning Agenda

Research Director-Suiting Up

Create Processes to ANTICIPATE the technologies that will In short - create future intersections Science, Business or Program Needs IT Technology Opportunities now future Primary Goal of an R&D Portfolio in IT • Keep the institution/organization at the forefront of high performance computing to enable “new” science • Promulgate the institution/organization as a world class leader in all aspects of modern computing

What is key to stake holders What are the future “competitive1” enablers of the research center Information technology has a dual role As a discipline of science, IT research is bona fide, critical, and fueling the economic revolution of an eSociety As an enabler and partnerfor other disciplines, (e.g. biosciences, physics, . . .) One important role of R&D leadership is to insure the collective efforts are not blind-sided by “disruptive technologies”. Identify the “Battlefield” 1Competitive: (not a dirty industrial word) - improve one’s relative position in a free and open society. In academia, provide the leading research resource among peers in a particular discipline or technology, and garner long term support for sustained research.

“The Innovators Dilemma: When New Technologies Cause Great Firms to Fail,” Clayton Christensen Technologies which have no economic base or market, often too costly to adopt, yet devastating when ignored Technologies which often invalidate both an organization’s core competency Technologies that when implemented, surprisingly revolutionize the very process, business or even economy they were intended to serve Examples: electricity, the Web, the microprocessor, the automobile, . . . My summary: Winners guess what’s next, non-guessers lose The “Innovators Dilemma:”Disruptive Technologies

Example: Computation VS. Mainframes MICROs

Is it over? $$ Accelerating Moore’s Law thru aggressive parallelism at the high end: e.g. the DOE ASCI approach, build huge machines with biggest servers and do new science sooner

The Issue is not Platform, but Control “Commodity” Moore’s Law based (cheap) clusters: Dept. Controlled $$ VS.

Gulp! A Disruptive Trend: Intelligent Systems What will happen when commodity computers can think? VS.

Gulp! A Disruptive Trend: Intelligent Systems • LET’S IMAGINE 2006 • Virtual O/S (intra-grid or internet), virtual single image • Heuristic (agent based) mapping of appropriate resources • Natural language interaction with resources, data, and applications • Mobile agents that live on multiple CPUs and clone their backups • Dynamic “group-task” throughput (to support parallel apps is common) • Big tasks go to big machines(big2big) • Parallel jobs go to parallel environs., with preserved sync. (par2par) • Executive Nat. Lang. control software for total definition [session] (e2j) • Secure data, no firewalls (white world), seamless, DEN, credential • passing strong authentication, self-healing & fencing faults • Self organizing networks, load balancing, cost accounting • Personal computing environment based on identity, not the logon platform • (your “favorites” follow you, “self-purposed” to your device selection) What will happen when commodity computers can think? VS.

Gulp! A Disruptive Trend: Intelligent Systems • LET’S IMAGINE 2006 • Virtual O/S (intra-grid or internet), virtual single image • Heuristic (agent based) mapping of appropriate resources • Natural language interaction with resources, data, and applications • Mobile agents that live on multiple CPUs and clone their backups • Dynamic “group-task” throughput (to support parallel apps is common) • Big tasks go to big machines(big2big) • Parallel jobs go to parallel environs., with preserved sync. (par2par) • Executive Nat. Lang. control software for total definition [session] (e2j) • Secure data, no firewalls (white world), seamless, DEN, credential • passing strong authentication, self-healing & fencing faults • Self organizing networks, load balancing, cost accounting • Personal computing environment based on identity, not the logon platform • (your “favorites” follow you, “self-purposed” to your device selection) What will happen when commodity computers can think? Right or wrong - - Ask what research will SDSC be doing in 2006? Managing research requires being the custodian of the collective vision of the community VS.

Next generation systems grow in capability 100 Teraflops to Petaflops Huge data generation, much to be stored for perusal Network bandwidth, using Lamda technologies, 1000 gBytes/fiber link in hand today, . . . tomorrow? Resources will be distributed and dynamic Sensor systems, and other pervasive devices will dominate networks (by measure of IP addresses) Humans (unfortunately remain the same) Using all our senses we can process about a gByte/sec (largely vision) We are good at judgement, lousy at flops Our biggest assets are intuition, instinct, and anomaly detection But humans are likely to be the bottleneck in most processes A major realization for scientists in the future (computers need brains, and brains need computers!) We (humans) will need help in doing the human tasks in future processes Part of future petaOps will be needed for implementing human cognition in our processes, some have coined the phrase COGNITIVE PROSTHESIS FLASH: Reality Check for Humans

Next generation systems grow in capability 100 Teraflops to Petaflops Huge data generation, much to be stored for perusal Network bandwidth, using Lamda technologies, 1000 gBytes/fiber link in hand today, . . . tomorrow? Resources will be distributed and dynamic Sensor systems, and other pervasive devices will dominate networks (by measure of IP addresses) Humans (unfortunately remain the same) Using all our senses we can process about a gByte/sec (largely vision) We are good at judgement, lousy at flops Our biggest assets are intuition, instinct, and anomaly detection But humans are likely to be the bottleneck in most processes A major realization for scientists in the future (computers need brains, and brains need computers!) We (humans) will need help in doing the human tasks in future processes Part of future petaOps will be needed for implementing human cognition in our processes, some have coined the phrase COGNITIVE PROSTHESIS Joe’s Cognitive Prosthesis Yak, Yak Hal 1-cluster awk FLASH: Reality Check for Humans

The nature of computing is changing in fundamental ways, & computing advances are fundamental to the very fabric of scientific endeavors. The Point

Intellectual resources are always limited, and since disruptive changes occur in the infrastructure as well as the scientific processes supported, where does one place the emphasis? What happens when things get out of balance If computing R&D ignores the “science customer-needs”, the results are tools of little utility. If the science research reigns supreme, the computing tools and paradigms will decay to the point that the science will not be competitive (remember before SC Centers, researchers went to Europe) In short: a partnership needs to exist in IT Research and the science objectives it supports. Industry has clear business goals and the IT R&D exists to support them Academic research is broad and IT R&D can often have an identity crisis (particularly infrastructure and tool research) The Big Dilemma: Balance Between Comp R&D/Infrastructure and Science

$ $ $ IT R&D Portfolio Management What IT R&D is critical to advancing the science research of the community served What are the science areas critical to the institution/organization and its partners/customers NSF UCSD

MIT - http://www.technologyreview.com/magazine/jan01/tr10_toc.asp Brain Machine Interface – literal brain prosthesis Flexible Transistors – super pervasive computing, smart news papers Data Mining-broad sense of knowledge discovery Digital Rights Management – IP gone mad, ITAR**2 Biometrics – Authentication, iris scan, retina, finger print, DNA NLP – HAL 9000 XP, intelligent search etc. Microphotonics – nano electronics etc. Untangling Code - aspect programming, crosscut tracing, intelligent code, coherency of data Robot Design Microfluidics – micro technology in fluids RED Herring (VC mag.) Data center and telcom integration, plus teraflops on tap Security, safety, protection (protect from the intruders within) Small worlds-micro to nano Mobile virtual network operators Biotech/drugs Digital media networks in business (more than Voice over IP) Sample: Top Trends (lists abound)

One can be embarrassed by “disruptive technologies” purchase expensive high-end equipment which is eclipsed by the low-end in performance, even before high-end is fully functional! (oh no!) spend millions on software for the wrong computing paradigm miss a paradigm shift, causing needless delay in the scientific endeavor—rework application design build an e-business on a web that crashes due to denial of service attacks (I.e. ignore security until your violated) When competing researchers are “conversing” with their application interface, yours are fixing the card reader Ignoring Disruptive Technology Key is to match tech-trends with mission & vision

Over the next decade, many key principles of our work will need to change Deterministic algorithms, and engineered data movement are the cornerstone of running applications on large parallel systems This is very true today and has been the case historically But this has to change, or computing will be limited to human bandwidth! Intelligent systems are not an event, a piece of software, or a smart middleware system. Intelligent systems Use non-deterministic techniques at all layers: network, O/S, aggregated resource allocation, application processes 50% of this change will occur in the applications layer and require key plays from Steve Ashby’s Organization 50% will be platform/systems related and require Mike McCoy’s team However, half will not come from evolution, but from disruptive tech. I.T. Trends Cause Cultural Change in both Computing Methods and Applications Disruptive Platforms/Sys. Applications

Core competencies (I.e. key long term R&D strengths) should be identified and validated in multiyear plan/vision What’s core? This must be a collective vision, strategy, yet tactically managed. Key science areas to impact Key research areas The vision may be “bigger” than the core capabilities Create partners to fill out the missing capabilities Include the partners in the vision (e.g. NPACI, etc) Make roles visible to everyone: customers, researchers, partners, funders, the media (press) Using Vision for Partnering: Force Multiplication (military term)

Application frameworks & Grid Computing Data integration & exchange Information assurance Pervasive intelligent devices Intelligent Systems & interfaces ? Disruptive Impacts: 5 sample areas

Goals Improved processes and quality of the final design Easy collaboration among disciplines Gain insight,not simply produce results Help for the human in the loop with statistical and cognitive aids Offer cognitive aids (within the application and within the system) Take advantage of distributed resources, data, and expertise Flexible and extensible usage Create secure authentication model among programs and data sets Characteristics Systematic use of existing analysis codes Provide cognitive tools and techniques to help the human bottleneck Provides tools for integrating multiple disciplines Provides tools for data manipulation and viewing Algorithm choices (meta algorithms) Reuse of middleware, libraries, common data Application Frameworks - A definition

An Example: Design Explorer-MDO Validate 0.2 Surrogate Model 0.0 x2 -0.2 Statistical analysis of global modeling evaluation pts. Y X -0.4 X X X 0.4 0 X X 0.2 X 0.4 0 X x2 0.2 -0.2 0 X x1 -0.4 -0.2 0.0 0.2 0.4 -0.4 -0.2 x1 Build surrogate multidimensional model -0.4 DOE Intelligent sensitivity

Types of data require differing approaches: physical data, computed data, design data - - all need Persistent storage Distributed access and coherence under revision Intuitive interfaces (NLP, glossary based and ontologically structured) Innovative data gathering (temporal data bases) Knowledge discovery (text, causal rules, clustering, flow vortices) Scientific data Temporal data storage Cognitive search engines Anomaly detection Spatial queries CAD tools for test/manufacturing E.g. electronic mockup and haptic interfaces for building NTF before it was built! Data Architecture, Standards, & Policies Mediate, warehouse, or translate among disparate data repositories? SAN’s, grid based collaborative data sharing (XML?) Data persistence over time and through disaster Save data, or recreate it? Data Integration Issues

Secure Data • Data Encryption • Transmission Encryption • Certificate Based Data Access & Authorization • Digital Signature • Data Ownership • User Authentication • Biometric Authentication • Smart Cards • X.509 Digital Certificate • Public Key Infrastructure • Secure Computing Infrastructure • Intrusion Detection Modelling - Global Analysis!! • Firewall, VPN, Intelligent Gateway • Certificate Based Access Control Security Infrastructure, Requires User/App Participation

Secure Data • Data Encryption • Transmission Encryption • Certificate Based Data Access & Authorization • Digital Signature • Data Ownership • User Authentication • Biometric Authentication • Smart Cards • X.509 Digital Certificate • Public Key Infrastructure start Application 2 Surrogate credential Accepted for data access Application 1 Credential accepted forwarded • Secure Computing Infrastructure • Intrusion Detection Modelling - Global Analysis!! • Firewall, VPN, Intelligent Gateway • Certificate Based Access Control Security Infrastructure, Requires User/App Participation

Secure Data • Data Encryption • Transmission Encryption • Certificate Based Data Access & Authorization • Digital Signature • Data Ownership • User Authentication • Biometric Authentication • Smart Cards • X.509 Digital Certificate • Public Key Infrastructure start Application 2 Surrogate credential Accepted for data access Application 1 Credential accepted forwarded • Secure Computing Infrastructure • Intrusion Detection Modelling - Global Analysis!! • Firewall, VPN, Intelligent Gateway • Certificate Based Access Control Security Infrastructure, Requires User/App Participation SomeKey New Technologies Predictive behavior modeling to feed intrusion dectection Strong (application transferable) authentication Dynamic role based directory enabled security Trusted agent handling

Secure Data • Data Encryption • Transmission Encryption • Certificate Based Data Access & Authorization • Digital Signature • Data Ownership • User Authentication • Biometric Authentication • Smart Cards • X.509 Digital Certificate • Public Key Infrastructure start Chief Big Shot Application 2 Surrogate credential Accepted for data access Application 1 Credential accepted forwarded • Secure Computing Infrastructure • Intrusion Detection Modelling - Global Analysis!! • Firewall, VPN, Intelligent Gateway • Certificate Based Access Control (or do we?) Security Infrastructure, Requires User/App Participation SomeKey New Technologies Predictive behavior modeling to feed intrusion dectection Strong (application transferable) authentication Dynamic role based directory enabled security Trusted agent handling

Growth of miniaturization and device size Network presence for devices [ CA(IT)2] Sense radiation, moisture, vibration, smell, location Find trends, anomalies and react Personal networks (body health) DARPA Pheromone project Deploy thousands of devices Sense environment Point (physical gradient field) Implications for Defense? ASCI customers? Computing infrastructure to support thinking environments CIO Questions: Is there an end user play at LLNL? Issue: lead or wait, and support? Should there be a matrixed support contingent for these applications, or is each program to invest in this expertise? Gulp! Pervasive Computing

Recently, II’s have been cute Next, II’s will provide great ease of use based on solid intelligent agent infrastructure/architecture and communication grammers In the future II’s will provide indispensable tools for job completion and morph into “autonomic systems” Credential handling and validation Data organization and exploration Administration of all kinds within a growing complex net based computing world Automatic failure detection and mitigation Repurposed environments Credential based view and access Exploration within authenticated access Rapid recovery of historical actions and shared “cleared” knowledge NL and ontological knowledge in interfaces related superset ignition lift Turbojet thrust subset engines Aero Ontology-nouns Pneumatic equipment starting Propulsion engines systems Engine Jet starters engines Flame Ramjet Hydrogen propagation flameout engines fuels Flame stability combustion afterburning Combustion stability Burning spray rate Jet spray Intelligent Interfaces (II’s)

QoS - thinking routers? Dynamic schedulers? Mobile self-configuring sensor networks? Service level capacity? Telephony issues - Voice over IP Mobile agent arbitrators, cloned and reliable Policy and standards positions MANET (Mobile Ad hoc NETs, unclassified or in the field) HIP (Host Identity Payload, mobile IP) SCTP (Stream Control Transmission Protocol, large data set perusal) DEN (Directory Enabled Networks for data access/protection) XML data/knowledge exchange standards and policies Predictive and dynamic performance and scalability models Non-linear capacity planning Non-linear load modeling Data & Information Standards and common approaches Standards and tools for human computer interaction: (NLP, Ontologies) System health and intrusion detection modelling Disruptive Infrastructure Activities

Identify Organization/Business Drivers (applications, impact areas, etc) Identify supporting technologies that are key and likely to be key over the long term Build Application Scenarios (drivers) and Core Technology Stories that are symbiotic and mutually interactive derive new application possibilities from technology road maps keep technology research relevant against the application drivers At Boeing we have build 7 major new business drivers and some 15 key supporting IT R&D programs Method

4 4 3 Example Driver Airport Security Integrated Solutions Approach Maintenance Warning. Missing Tool/FOD not picked up Ramp A Warning. You should be at gate C Gate A 1 • Airport Arrival • Unique encrypted ID Bands • applied at arrival. • Authorizes access to lobby only • Detects tampering • Decrypt time > travel time • Portals challenge exceptions 3 • Zone or Perimeter • System TBD • Candidates to review • Laser curtain • Pressure Mats • Linear transceivers • Antennas • RTLS Gate B Ramp B • Check in • Unique Bag Tags to match owners • no mix-ups or leave behinds • Authorizes access to specific gate @ specific time 2 • Security • Passengers only to gate • Carry-ons can be sealed Warning. You don’t have correct luggage Ramp C Gate C • Boarding • Doubles as boarding pass • Passengers match checked and carry-on luggage • Detect opened carry-ons Warning. You forgot 1 piece of luggage on board • Deplaning & baggage claim • checks for left luggage • baggage claim security Warning. You only have permission to work ramp A Warning. Your Ramp pass used twice, check for piggybacks. Ramp Access Joe Fletcher, 9-14-01

System of Systems Link all airports: Use rule based systems to identify dangerous trends triage identities for terrorist/felons using national travel card. Is the passenger who they claim? 4 4 3 Example Driver Airport Security Integrated Solutions Approach Maintenance Warning. Missing Tool/FOD not picked up Ramp A Warning. You should be at gate C Gate A 1 • Airport Arrival • Unique encrypted ID Bands • applied at arrival. • Authorizes access to lobby only • Detects tampering • Decrypt time > travel time • Portals challenge exceptions 3 • Zone or Perimeter • System TBD • Candidates to review • Laser curtain • Pressure Mats • Linear transceivers • Antennas • RTLS Gate B Ramp B • Check in • Unique Bag Tags to match owners • no mix-ups or leave behinds • Authorizes access to specific gate @ specific time 2 • Security • Passengers only to gate • Carry-ons can be sealed Warning. You don’t have correct luggage Ramp C Gate C • Boarding • Doubles as boarding pass • Passengers match checked and carry-on luggage • Detect opened carry-ons Warning. You forgot 1 piece of luggage on board • Deplaning & baggage claim • checks for left luggage • baggage claim security Warning. You only have permission to work ramp A Warning. Your Ramp pass used twice, check for piggybacks. Ramp Access Joe Fletcher, 9-14-01

External Aircraft Communications Emergency Low Interference Communications Cruise at LRC Aircraft Telemetry Sampling (e.g. Iridium) Climb Descent to 10,000ft. Accelerate to Climb Speed Climb to 10,000ft. Descent to 1,500ft. Climb to 1,500 10 min. Hold Land Ground Troubleshooting Support

Inter-Department Event Data Coordination and Analysis • In-Time Data Publication for Authorized Subscribers • Intelligent Data Retrieval and Management • Airport Security Agencies • Office of Homeland Defense • Federal Bureau of Investigation • Intelligence Agencies

Integrated Safety Systems (People/Transportation) Scope Relevance to Boeing Homeland Safety – Tracking of felons and terrorist, screening of transport people/goods, immigration, transportation systems (e.g. fuel supply vulnerability) Airport Ground Ops – baggage/owner tracking, employee monitoring, access control, airport systems GTU – peoplebusiness opportunity Intrusion Detection of Battle Management systems – Intelligent monitoring of information and comm access behavior, security and authentication of users Airspace Operations – Monitoring flight paths, communication to pilots and crew, onboard safety systems that integrate with ground, black box monitors, real time re-routing The events of 9/11 have created a long term need for safe, reliable, and secure information systems to support the transportation industry. Many of the SoS type requirements are also fundamental to logistics and C3I. People, goods, and events need to be tracked, stored, mined and manipulated to create knowledge and information regarding safety and movement of all people/goods within the context of a free an open society. The underlying task is to integrated a broad set of technologies such as biotech for strong authentication (RBAC), pervasive smart devices to monitor goods and their owners, and intelligent active systems to discover anomalous behavior of all kinds. The technologies are at hand, and the integration task in various forms provide business opportunities. M&CT Core Technologies Performance and Scalability Modeling Intrusion Detection and System Health Pervasive computing devices & middleware Operations Research Novel Information Storage/Access/Retreival Architecture Grid-based Sys. ( & small area nets) Intelligent Agent/Systems Mobile & Hybrid Network Technology Information Management Knowledge Discovery

IA Roadmap Enablers Transparent Information sharing over great distances: wireless and satellite Grid Computing Intrusion Detection Isolation Protocol (IDIP) Neural Networks Autonomous secure agent framework 2015 Distributed Denial of Service (DDOS) Tolerant Networks Application Intrusion Detection System (IDS) Virtual Private Content Network Peer-to-peer Authentication, threshold cryptography Automated and secure agent-based information processing Pervasive & mobile Publish/subscribe Integrated biometrics authentication Secure Downgrader Smartcard USB tokens Secure & self-healing automated system-of- systems Policy Engine Role Based Access Control (RBAC) Data Separation Design, build everywhere Constructive Key Mgmt (CKM) Public Key Infrastructure Coalition warfare & information superiority AviationSafety HomelandSecurity 2002 Secure Network Server (SNS) Global enterprise business integration Intelligence gathering and sharing Impact Firewall LANS

IT Research can and should be grounded in applications Applications (Science) should acknowledge their dependence on IT A concerted effort should be made to describe the long term broad impact of IT Research at a program level To insure quality research, disruptive technologies should be anticipated by all Future computing systems will be cognition rich and autonomic in nature, highly secure, and self-healing The role of the IT Research leadership is to be the custodian of the collective integrated vision of the constituent researchers Conclusions

Managing R&D in a World of Radical Change

Managing R&D in a World of Radical Change

Presentation Transcript