1 / 66

Introduction to Distributed Systems

Introduction to Distributed Systems. Prof. Walter Kriha, fall term 2013/14. Hochschule der Medien. Overall Goals. Learn the basic concepts of Distributed Systems like concurrency and remoteness Understand different programming models for Distributed Systems

thyra
Download Presentation

Introduction to Distributed Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to Distributed Systems Prof. Walter Kriha, fall term 2013/14 Hochschule der Medien

  2. Overall Goals • Learn the basic concepts of Distributed Systems like concurrency and remoteness • Understand different programming models for Distributed Systems • Acquire a theoretical foundation of computability in distributed systems • Learn to design distributed systems with a focus on performance, availability, scalability and security • Understand the constraints imposed by hardware

  3. Timeline • Introduction to DS • Socket based DS and delivery guarantees • Remote procedure calls • Remote objects • Theoretical foundations of DS (FLP, time, causality and consensus) • Distributed Services 1 and 2 • Distributed Security 1 and 2 • Distributed Components • System Management in DS • Design of Distributed Systems: Methodology and Examples • Web Services: SOA and REST • Peer-to-Peer Systems • Ultra-large-scale Systems

  4. Goal for today • Give an overview of distributed systems. Later lectures will dig into the gory details like security, transactions, remote calling mechanisms etc.

  5. Introduction • What is a Distributed System (DS) • Why distribute? • Types of DS • Characteristics of DS • Middleware for DS • Resources • Exercises

  6. Definition of a Distributed System: Independent agents repeatedly interacting in a way that a coherent behavior („system“) emerges Agents are intelligent, with incomplete local information, differ in capabilities, suffer from random events (local or network). See: A.B.Downey, Think Complexity

  7. Why learn about Distributed Systems? Because most IT systems ARE ALREADY DISTRIBUTED SYSTEMS (and not only the IT Systems) The demands of a consumer Web site/API or multitenant enterprise application simply exceed the computing capacity of any one machine. [MC] An enterprise moves an existing application, such as a three-tier system, onto a cloud service provider in order to save on hardware/data-center costs. [MC] See: M.Cavage, There's Just No Getting around It: You're Building a Distributed System, ACM Queue, April. 2013

  8. Why Distribute? • Robustness: avoid single points of failures (e.g. Use hot stand-by data centers) with replication • Performance: Split processing • Throughput: use many connections • Security: create different security domains • Price per request: use cheaper horizontal scaling or free resources

  9. Types of Distributed Systems • Energy grid, telcom net • Villages, towns and big cities • It-Infrastructure of large companies • High-performance clusters • The WWW • The human body, organizations, states • A flock of birds

  10. The energy grid now: hub and spoke office home home Power- plant factory home Electricity flows in one direction only, with a lot of it lost during transport. Control resides with the power plant. www.wired.com/wired/archive/9.07/juice.html

  11. The future grid : micropower Fuel cell Fuel cell office home home car Fuel cell Power- plant Fuel cell factory home Fuel cell car car Power flows many directions, controlled by independent sensors in the grid. A tenfold increase of transactions. Modern GRID computing allows users to tap into a wealth of distributed computing resources. http://www.thegridreport.com/

  12. The power law of settlements There are many villages, quite a few towns but only a small number of big cities

  13. IT-Infrastructure of large corporations Un-trusted clients Customer data zone Processing zone

  14. Warehouse-Scale Machines Datacenters are huge distributed systems, requiring special middleware. Diagram from: Barroso et.al., The Datacenter as a Computer

  15. World Wide Web Internet DNS Intranet Dialup clients live at the „edges“ of the internet (no fixed IP address, slow upload). How many graphs are layered on top of the physical network structure? (hyperlinks, search-engines, DNS)

  16. The New Web P2p overlay Social network overlay Mobile PAN network mashups Internet Location based service Sensors Cams Real world Aggregation of external information and collaboration based on social networks will bring new forms of content production and consumption and consumer areas will influence companies („consumerization“, Gartner Group). More interconnection of different net-types brings more emergent phenomenons.

  17. Distributed Application Structure

  18. Views of Distributed Systems Enterprise View: the role within an enterprise Information View: the flow of information in the DS (information architecture) Computational View: the processing of information in the DS (logical architecture) Engineering View: the system infrastructure (nodes, connections, system management, replicas etc.) (physical or distribution architecture) Technology View: the specific technology used to build the DS (From Open Distributed Processing plus my architecture categories)

  19. Characteristics of Distributed Systems • Influence of distribution topology and remoteness • emergent behavior and computational irreducibility • No analytic solutions • Simulation is proportional to prediction time • heterogeneous components • a strong need for security • Concurrency, Parallelism and replication Don‘t worry, we‘ll dig into all this another day!

  20. IT-Problems with Distributed Systems • Badly supported in programming language concepts • heterogeneous components • a strong need for security • Scalability and Availability • Network problems • Global naming/adressing • Consensus • transactions across nodes • no global state • no centralized control • many points of failure • Asynchronous communication

  21. Programming Languages and Distributed Systems DS: • system defines security • Objects are versioned • Global identity • Components PL and DS are orthogonal. PL: • Design security (private, protected etc.) • no versioning, no components as types • memory address used for identity • platform dependent basic type size (int)

  22. The „small world“ effect: Distribution Topology It takes only a small number of intermediate persons to connect any person on this world to any other one. (A knows B, B knows C, .... F knows G.) From: The Milgram experiments on social networks. (Andy Oram, Peer-To-Peer, Harnessing the power of disruptive technologies). OpenBC or LinkedIn create a social network from distributed participants. Look at the facebook profiler by Thommy Fankhauser at http://profiler.mi.hdm-stuttgart.de/

  23. Systems showing the small world effect High local clustering How efficient can this DS transport messages? Queries? How robust is it against random attacks on nodes, targeted attacks on the important connecting nodes?

  24. The power law of DS – a law of nature? Cities, Companies, Power, social networks etc seem to exhibit the power law. Each size is a tenth of the next bigger size but has ten times more instances. „In defense of cities“, Clay Shirky, www.openp2p.com

  25. Metcalfe’s law - Network Effects • The usefulness of a network grows by the square of the number of users (think about a fax machine – how useful is one?) • The adoption rate of a network increases in proportion to the utility provided by the network. (That‘s why companies give away software e.g.)

  26. Emergent Behavior – a flock of birds There is no central controller, no „Super-bird“. No bird has a representation of the figure in its head. Instead, every bird follows very simple rules. The resulting figure shows EMERGENT behavior. Many distributed systems show it as well – for good or for bad. (Kevin Kelly, Out of Control – The biology of the new machines. Peter Wegner, Interaction vs Algorithm.) Picture: http://www.pdfnet.dk/ (PD)

  27. Emergence „An emergent property is a characteristic of a system that results from the interaction of its components, not from their properties. [] Emergent properties are surprising: it is hard to predict the behavior of the system, even if we know all the rules“. A.B. Downey, Think Complexity (www.thinkcomplexity.com)

  28. Heterogeneous Components Hardware unreliable Frequent downtimes Little endian byte order Java Data Types No callbacks Slow, no access control Fault tolerant hardware System management Big endian byte order C++ data types Fast, access controlled

  29. Security in Distributed Systems Authentication Authorization Integrity, Confidentiality But: sometimes anonymity is needed!! (peer-to-peer systems) Authentication Authorization

  30. Firewalls Certificates, Public Key Infrastructure, Digital Signature Encryption (methods and devices) Software Architecture Intrusion Detection Sniffing PGP, SSL etc. Denial of Service attacks Security Topics • Authentication (who are you?) • Authorization (what can you do?) • Confidentiality (can someone spy on us?) • Integrity (Did somebody change your message?) • Non-repudiation (It was you who ordered X) • Privacy/Anonymity

  31. Theoretical Foundations of DS • No global time (logical clocks, vector clocks) • FLP theorem of asynchronous systems • The problem of failure detection and timeout • Concurrency and deadlocks • CAP theorem: consistency, availabilty and partitioning: choose two only! • End-to-end argument • Consensus, leader selection, etc.

  32. Important Programming Terms for DS • Identity • Value vs. Reference • Exception • Interface vs. Implementation • Interface Definition Language (IDL) • Quality of Service (QOS) • Stubs/Proxies

  33. Distributed System Design • Common Problems (performance, fail-over, maintenance, policies, security integration) • Information Architecture (define and qualify the information fragments and flows) • Distribution Architecture (create a map of all participating systems and their quality of service)

  34. Middleware for Distributed Systems

  35. What is Middleware? software that helps two separate systems communicateseamlessly. (www.knownow.com/middleware/lexicon.html) In a strict sense middleware is transport software that is used to move information from one program to one or more other programs, shielding the developer from dependencies on communication protocols, operating systems and hardware platform (“plumbing”) (www.talarian.com)

  36. Positioning Middleware General structure of a distributed system as middleware. 1-22 From: van Steen/Tanenbaum

  37. The Transparency Dogma • Middleware is supposed to hide remote-ness and concurrency by hiding distribution behind local programming language constructs Critique: Jim Waldo, SUN Full transparency is impossible and the price is too high

  38. Distribution Transparencies • Access: mask differences in data representation and invocation mechanisms between heterogeneous systems • Failure: mask failures to enable fault tolerance (e.g. Intelligent load-balancing) • Location: use logical, not physical names to access services • Migration: hide the true location of a service or object from clients. If the location changes, the client won‘t notice it.

  39. Distribution Transparencies cont‘d • Replication: hide a group of equal objects behind an interface (performance, availability) • Persistence: hide the storage mechanisms and internal policies from a client. Make a remote object look like it is persistently activated. • Transaction: hide the complex coordination necessary to achieve consistency. Source: ISO/IEC 10746-1 Open Distributed Processing, www.iso.org

  40. Where do we find Middleware? LDAP or DCE Distributed Cache Directory Quotes JMS WebService JNDI Application Server Web- Tier Application Server EJB Tier Web Server JDBC RMI XML-RPC CORBA News E-bank Part of a Portal running on a Web Cluster.

  41. Classification • Socket Based Services • Remote Procedure Calls (RPCs) • Object Request Brokers (CORBA, RMI) • Message Oriented Middleware (MOMs) and Event-Driven-Systems • Web-Services (XML-RPC, SOAP,UDDI) and SOA • Component Systems (Enterprise Java Beans, J2EE) • Peer-To-Peer (Napster, Gnutella, Freenet, seti@home) • Agent based (Jini, Aglets) • Tuple-Spaces, distributed blackboards

  42. The “ilities” • Reliability • Availability • Security • Scalability • Quality • Performance • Maintainability Before using a specific middleware, always make sure that the “ilities” aka non-functional requirements are met. Middleware almost always differs implementation quality between vendors.

  43. Real-World Problems • Skills/Understanding: Best practice patterns? • Single-Point-Of-Failures: replication, load-balancing etc. • Tooling: generators, deployment tools • “Brittle-ness” if interfaces change (Compiler “illusion”)

  44. RPC type Middleware • E.g. Sun-RPC, OSF DCE, SOAP • Main idea: distribute functions, use concurrent processing • On top of it: Distributed Directory, File system, Security (cells, principals) • XML-RPC over http (www.userland.com) • Apache Thrift, jnb.ociweb.com/jnb/jnbJun2009.html Layer foundations: UUIDs, value vs. reference, marshaling, versioning etc.

  45. Java only (e.g.Introspection used) Lightweight method call semantics Java Implementations Wire Protocoll now mostly: RMI over IIOP Distributed Objects CORBA RMI • Object Request Broker • Multi-language support (platform independence) • Interface Definition language • Wire Protocoll: IIOP, GIOP Both try to preserve object semantics. Interface/Implementation separation

  46. Solutions: Enterprise Java Beans CORBA Components COM+ .NET Spring, Ruby on Rails etc. Distributed Components • Objects are too granular: performance and maintenance problems • Programmers need more help: separation of concerns and context

  47. System Management defines Data Sources and Containers Example: Enterprise Java Beans EJB Framework (Separation of concerns): Deployment (Separation of context): Automatic Transaction Management System Management defines Pool sizes Concurrency Control System Management defines Role/User Binding Automatic, method level Security

  48. EJB Container Client Entity Bean invoke Load/ persist delegate At the point of interception the container provides the following services to the bean: Resource management, life-cycle, state-management, transactions, security, persistence

  49. Distributed Messages (MOM) Asynchronous, loosely-coupled (fault tolerant), persistent messages with either publish/subscribe (topics) or queuing semantics. Scales well. Delivery guarantees differ. Sub Get Sub Pub Pub Put (M1,M2) Sub Topic M2 M1 queue publish Sub send Sub Get MOM MOM

  50. Distributed Code I (Agents, Aglets) The Problem: who wants a new runtime system? Agent Agent Perform work, come back with results pack unpack Serialized Agent Agent Runtime Agent Runtime Channel OS OS

More Related