Web Services

1. Web Services Distribution paradigm between hype and revolution Walter Kriha

2. Goals Web Services were the most overhyped topic in 2001. Then came SOA, then Cloud Computing. One goal of this session is to discuss the pattern of “disruptive technology” and see if and how it applies to Web Services. Another goal is to discover strength and weaknesses of xml based distribution technology – and as we do so also learn how such technology can be evaluated properly. We will meet a lot of false arguments about distributed systems. And finally we will use our distributed systems design knowledge from the last session and try to apply it to applications built with Web Services: what style of application works best? What fits into the Web Services paradigm? Does old distributed system knowledge still apply? Part of our job is to evaluate new technology, e.g. for project use. As with every new technology its value and application are unclear – and even old computing hands can be wrong. So don´t hesitate to voice your opinion!

3. Overview • The concept of disruptive technology • What are Web Services? • SOAP and the performance discussion • A way to describe messages: Web Services Description Language • How to find Web Services: Universal Discovery and Description Interface • Web Services and the “ilities”: Reliability, Security, Transactions etc. • Web Service applications • Trends and possibilities • Discusion: Are Web Services disruptive? Is it a Peer-to-peer technology?

4. Successfull companies and Innovators Dilemma • Listen to customers • Invest aggressively in technologies that give those customers what they say they want • Seek higher margins (e.g go after the most wealthy customers with the hardest problems) • Target larger markets (from Christensen, page 263ff.) This behavior of successfull companies is well known. On their way to success these companies restructure themselves to better achieve the goals from above: they invest only in the high margin area, they develop an internal cost structure that matches the high-margin market they are after. Clayton Christensen shows that it is exactly this behavior that makes successfull companies miss disruptive technology

5. Disk Drive Product Features and Competition capacity reliability speed price time axis of product evolution Most product display a similiar evolution: from technical possibility over improvements (convenience) and feature overload (think about hi-fi products!) to battling over price only. This is the reason why most companies try to go „upmarket“ where higher prices are paid for more complex products.

6. Commoditization of IT? First, regardless of whatever definition you use to describe cloud computing (a fairly hopeless task in my view), the term merely identifies an underlying shift of I.T. from a product to a service based economy. It's a consequence of : - Certain I.T. activities becoming suitable for service provision through volume operations (i.e. those activities are well defined and ubiquitous The existence of mature enough technology to support this (Popek et al wrote the book on virtualisation back in 1974) -A change in business attitude towards I.T. (Carr and others, namely Strassmann, pointed out that much of I.T. is simply seen as a cost of doing business.) - The concept of utility computing provision (i.e. the provision of suitable I.T. resources much like other utility providers, as forecast by McCarthy back in the 1960's.) Take away any of these elements and cloud computing wouldn't represent the upheaval and the disruption that it does today. From: Simon Wendley‘s blog http://blog.gardeviance.org/

7. Disruptive Technology (1) • Is simpler and cheaper and lower performing • Promises lower margins, not higher profits • Leading firms most profitable customers generally can´t use and don´t want them • are first commercialized in emerging or insignificant markets (from Christensen, page 267) It is hard to find a market or application for something that is a looser in all known applications or markets. Our Peer-to-peer group noticed how hard it is to find usefull applications for a disruptive technology. Imagine you are a developer who wants to convince a well established manager of a successfull company to invest into a disruptive technology and you have to acknowledge the things from above! Imagine you are an employee of a database company and you´ve invented the www. Do you think your manager would be excited about some „network database“ where you can´t find things deterministically? And the reliability issues etc....

8. Disruptive Technology (2) • needs new markets to shine • needs new people and organization: it won´t grow in old structures • Starts weak but has potential • It does most things worse than established technology but it can do SOME things those can´t or do badly. This characterization is surprisingly close to how Thomas Kuhn explained progress in science („The structure of scientific revolutions“). He clearly showed that old theories literally die out with their teachers and that there is a large piece of believe both in old and new theories.

9. What are Web Services? „A Web service is a software component that represents a business function (or a business service) and can be accessed by another application (a client, a server or another Web service) over public networks using generally available ubiquitous protocols and transports (i.e. SOAP over http)“. (http://www3.gartner.com/Init by M.Pezzini, April 2001) • The core definition usually comprises: • simple requests • public networks/Internet • http transport • XML message format • business function

10. WWW: from GUI driven to B2B <FORM action="http://stockservice.com/getquote" method="post"> <P><LABEL for=valor">valor: </LABEL> <INPUT type="text" id=„valor"><BR> <INPUT type="submit" value="Send"> </FORM> stock server stockservice: valor=IBM html document with IBM=44.56 <xml-rpc><service>stockservice</service><request>getquote><parameter><name>valor</name><value>IBM</value></parameter></request></xml-rpc> myYahoo <xml-rpc><service>stockservice</service><response>getquote><parameter><name>IBM</name><value>44.56</value></parameter></response></xml-rpc> The concept of a web service is extremely simple: use XML to create requests and responses and send them using http. This allows machines to communicate with each others, e.g. to perform supply chain management or other business to business processing. XML-RPC by David Winer (userland.com) was one of the earliest standard proposals. Companies have used this technology internally for quite a while.

11. Web Services Components single sign on services: Hailstorm/liberty alliance global registry Digital Signatures Transactions Metering Universal Description, Discovery, Integration UDDI (XML) Web Service Description Language WSDL (XML) Request format: SOAP (XML) Transport layer: http(s), smtp, httpr XML is the standard format used in Web Services. On top of standard transport mechanisms are requests formatted using the SOAP XML schema. Clients learn about service providers by browsing the UDDI registry. Services are described in a special description language, again a XML schema.

12. The Web Services Architecture UDDI registry look for service in UDDI registry publish services in registry retrieve provider location and WSDL service description requester provider bind and send request via SOAP/http or other transport to provider create request from WSDL description This type of architecture is called „service-oriented“. It uses a broker for service advertisement and lookup. Requester and provider bind dynamically with respect to transport (http, smtp etc.) (Raghavan N. Srinivas, Web services hits the Java scene part 1, http://www.javaworld.com)

13. Request Transport in Web Services Currently http or https are mostly used for Web Services. But asynchronous messaging is also supported e.g. through SMTP. Both transports have deficits with respect to reliability. IBM has introduced a proposal for HTTPR, a reliable asynchronous message protocol. more information on HTTPR: ibm developerworks: Web Services: A primer for HTTPR. It has some excellent diagrams on the relation between transactional safety and resource allocation in asynchronous protocols. Shows why asynchronous protocols potentially scale better.

14. Request Format of Web Services: SOAP hello-request hello-response <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" xmlns:xsd="http://www.w3.org/1999/XMLSchema"> <s:Body> <m:sayHello xmlns:m='urn:Example1'> <name xsi:type='xsd:string'>James</name> </m:sayHello> </s:Body> </s:Envelope> <s:Envelope xmlns:s="http://www.w3.org/2001/06/soap-envelope" xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"xmlns:xsd="http://www.w3.org/1999/XMLSchema"> <s:Body> <n:sayHelloResponse xmlns:n="urn:Example1"> <return xsi:type="xsd:string"> Hello James </return> </n:sayHelloResponse> </s:Body> </s:Envelope> SOAP is essentially an RPC protocol with XML. It provides elements for type marshalling and RPC semantics. A header element contains meta-information but is optional. See Snell et.al. Programming Web Services... for details. Find a complete SOAP implementation at apache.org

15. SOAP: performance aspects (1) Object to XML conversion XML Parsing and construction of objects transport of XML stream over http XML stream XML stream The use of a non-binary transport format has created heated discussions about performance issues. Bad coding techniques (too many objects created), XML misunderstandings (e.g. to use validating parser without need) and wrong parsers (e.g DOM instead of SAX) have created a lot of confusion. Lately it looks like XML encoded messages can come close to the performance of binary formats like IIOP.

16. SOAP: performance aspects (2) effect of size on transport marshaling time de- marshaling time Internet transport time Object to XML conversion XML Parsing and construction of objects transport of XML stream over http XML stream XML stream The only way to find an answer on possible performance problems is to measure the effect of individual processing steps or transport times on the overall request time. It became clear that the internet transport time with lacking QOS has far greater effects on overall request time than the size and interpretation effort of a textual format. In other words: it is NOT the XML that is problematic, it is the public network (Internet) that takes a toll on request/response protocols. (watch Amdahls law in action)

17. loose coupling with XML • all Web-services API´s are expressed in XML and therefore completely programming language independent • XML messages are self-describing and can be validated on the receiver side if a schema is used. • XML messages can be read by humans and machines. • No pre-compiles stubs and skeletons are required These properties of XML, together with the use of http as transport have given web services the touch of simplicity. But beware: Changes in XML schemas WILL surprise receivers, the question of semantics is still open (see UDDI later)!

18. Web Services and Firewalls CORBA port SOAP request Web Service Application Server SOAP request http port SOAP request RMI port Firewall Web server Object server The firewall „friendliness“ of Web Services has been touted all along. But firewalls were introduced for a reason: to block protocolls that cannot be tracked and filtered properly – perhaps because the necessary infrastructure was never developed – perhaps because the protocols were not intended for the Internet like CORBA and RMI. But Web Services make no sense without such an infrastructure.

19. WSDL Web Services Description Language (WSDL) is the metadata language of Web Services. It defines how service providers andrequesters understand Web Services. When exposing back-ends as Web Services, WSDL defines and exposes components and lists all the data types, operations, and parameters used by that service. WSDL provides all the information that a client application needs to build a valid SOAP invocation that in turn is mapped by theWeb Services platform onto back-end enterprise logic. (after P.J.Murray, Web Services and CORBA, CapeClear)

20. WSDL: The IDL of Web Services <?xml version="1.0"?> <definitions name="StockQuote" <schena targetNamespace=http://example.com/stockquote.wsdl [...] <types><schema targetNamespace="http://example.com/stockquote.xsd" [...] <element name="TradePriceRequest"> <complexType> <all> <element name="tickerSymbol" type="string"/> </all> </complexType> </element></schema> </types> <message name="GetLastTradePriceInput"> <part name="body" element="xsd1:TradePriceRequest"/></message> <portType name="StockQuotePortType"> <operation name="GetLastTradePrice"> <input message="tns:GetLastTradePriceInput"/> <output message="tns:GetLastTradePriceOutput"/> </operation> </portType> <binding name="StockQuoteSoapBinding" type="tns:StockQuotePortType"> [..] <operation name="GetLastTradePrice"> [..] </binding> <service name="StockQuoteService"> <documentation>My first service</documentation> <port name="StockQuotePort" binding="tns:StockQuoteBinding"> <soap:address location="http://example.com/stockquote"/> </port> </service> </definitions>

21. WSDL Elements • Types– a container for data type definitions using some type system (such as XSD). • Message– an abstract, typed definition of the data being communicated. • Operation– an abstract description of an action supported by the service. • Port Type–an abstract set of operations supported by one or more endpoints. • Binding– a concrete protocol and data format specification for a particular port type. • Port– a single endpoint defined as a combination of a binding and a network address. • Service– a collection of related endpoints. A WSDL document defines services as collections of network endpoints, or ports. In WSDL, the abstract definition of endpoints and messages is separated from their concrete network deployment or data formatbindings. This allows the reuse of abstract definitions: messages, which are abstract descriptions of the databeing exchanged, and port types which are abstract collections of operations. The concrete protocol and data format specifications for a particular port type constitutes a reusable binding.

22. Dynamic Binding in WSDL Binding A Binding B Service Port A Port B Operation Operation Operation Message Message Message WSDL allows the specification of a service with operations on a logical level. The physical binding to host, transport etc. happens in a binding declaration. Clients negotiate the binding to be used with a provider.

23. Service Discovery (1): UDDI UDDI registry with find and publish API Yellow pages: business categorization, type and industry White pages: information about companies (loc., contact etc.) Green pages: meta information about services and their qualities most distributed services use some kind of central registry for service lookup. The Universal Description, Discovery and Integration registry plays this role in web services. Especially the green pages property has led some people to proclaim automatic service matching by service requesters browsing the meta-information contained there. For the difficulties behind ontologies and automated discovery see: Steve Vinoski, Web Services and Dynamic Discovery on webservices.org

24. Service Discovery (2): UDDI content <businessEntity>name, contact, location etc. <tModel>meta info on service specification of a service <businessService> <bindingTemplate> All content in UDDI is expressed in XML. Besides the obvious elements for companies and services a number of meta-information elements like tModel exist. A core feature of UDDI is the expectation that requester and provider do a dynamic bind where they agree on service and transport characteristics. A local registry can be downloaded from www.alphaworks.ibm.com

25. UDDI Organizational Alternatives UDDI registry UDDI registry provider/ index UDDI index provider/ index federation is possible between registries provider provider provider/ index provider provider provider provider/ index provider provider Several indexes are created provider UDDI index providers register services in central registry create request from WSDL description Different levels of economic control and threats are behind each model. Technically each of them is possible.

26. Common Business Processes: ebXML implement purchase process according to specification ebXML registry retrieve specification company A • Standard Purchase Business Process Specification: • Operations • Parameters • Flow register own purchase service use company A‘s purchase service company B find service from company A Without standard schemas for services every company will implement their business processes differently. Clients will have to deal with many different interfaces for the same type of service. ebXML is a global electronic business standard and defines a framework for defining, finding and using standard business process services. see www.oasis-open.org

27. Transactional Web Applications Flights Lock seat #5 The travel agency needs to reserve a flight, book a car and a hotel room for a traveller. Travel Agency Rental cars lock car #24 hotel reservation lock room #47 The 2-phase commit protocol does distributed transactions. But it does not really fit to the web world because it requires resource managers to lock resources. On the un-reliable medium internet this should be avoided. SOAP does not yet specify a transactional service. OASIS is working on „Business Transaction Processing“ to support Web Services Transactions. IBM is proposing a model using „tentative“ reservation to overcome the locking problem.

28. Web Services Application Types Information Aggregation <xml-rpc><service>stockservice</service><response>getquote><parameter><name>IBM</name><value>44.56</value></parameter></response></xml-rpc> stock server myYahoo hotel reservation <xml-rpc><service>roomreservation</service><response>reserveRoom><parameter><name>Kriha</name><value>4711</value></parameter></response></xml-rpc> Business Transaction The web has always been an information space as well as a transaction space. Web services allow both kinds of exchanges: more document centric information aggregation and more transaction oriented service requests.

29. Web Services Example Applications • The ubiquitous „Stock Quote Example“ • A zip code lookup service from US Postal • Weather reports lookup etc. • Single-sign-on services: passport (hailstorm), liberty alliance Up to now most web services applications are technical showcases and cover only very simple types of requests. This may be because things like security and transactions have not been solved yet. Right now SSL is used to secure message transport. But work is on the way to include digital signatures in SOAP messages.

30. Trends and Future developments web services gateway („firewall“?) Internet Intranet Interceptors A hosting service provides metering for providers (accounting, authorization etc.) Hosting Service requester web service provider web service The implementation of common services through web services will be an interesting case. Will performance and reliability suffer? What are the business models behind such a technology? Security and privacy issues?

31. Reliable Messaging Application layer Application layer persistent messages persistent messages SOAP msg. with message ID, sequence number and QOS tag requester receiver message ordering request and ack. MUST send ack!!! Reliable B2B messages need guaranteed delivery (ack enforced), duplicate removal (message ID) and message ordering (sequence numbers). SOAP and http do NOT provide those qualities. Further QOS extensions could be: time to hold messages, number of retries etc. Proxies are considered transparent.

32. Secure Messages SOAP envelope Application layer Application layer signed encrypted Digital Signatures: XMLDsig Digital Encryption: XMLEnc WS-Security goes from channel based security to message (object) based security. Individual messages can be signed and encrypted. WSDL can advertise the QOS expected/provided by a receiver. End-to-end security is possible across intermediates. See my internet security lecture for details on WS-Security, security policies and expressions (SAML, WS-Policy), WS-SecureConversation and WS-Trust. The idea of the „Virtual Organization“ – overlay structures over existing real organizations is one of the driving factors here.

33. Coordination and Transactions • A generic coordination service providing coordination tpyes, context and protocol (e.g. atomic transactions, business activities • A transaction service which covers traditional TA‘s and conversational Business Activities. Loosely coupled services which last a long time cannot use regular locking or do a complete rollback if something fails. Business Activities comprise several low-leval atomic TA‘s but make progress even in case of individual task failures.

34. Transaction Models Transaction Atomic Activity • nested tasks • long running • loosely coupled business activity • compensating tasks and activities • errors: order cancellation etc. • not nested • short • tightly coupled business task • rollback in case of error • errors: system crash Complete rollback is too expensive for long running business activities. Intermediate results must be visible early – compensating acting try to undo tasks in case of business errors.

35. A Coordinator forward Context Transaction Starter Context Participant Context Context register indirectly Context register with Coordinator create Context Activation Registration Activation Registration Coordinator Coordinator (subordinate) Activation works like a factory method to create a new Coordination Context. This context is forwarded to participants which register through it either directly with the first coordinator or with their own coordinator which registers itself as a sub-coordinator. Protocol and type of coordination are contained in context.

36. Phase Zero Protocol Coordinator Participant Resource Manager register for phase 0 holds cached data phase 0 write cached date phase 0 OK prepare (begin two phase commit To allow for caching by intermediates ws-transaction defines an additional phase called phase zero BEFORE the usual two phase commit protocol starts. Reason is that most resource managers cannot handle change requests after 2pc has started. This „feature“ is not undisputed – see Roger Sessions objectwatch newsletter on this topic.

37. Stateful Web Services wsdl + resource property description endpoint reference including resource property identificator get/set operations, notification registration, lifecycle (create/destroy) requestor endpoint with get/set methods Stateful architectures like computational grids need the concept of a resource. WS-Resource adds this via meta-data descriptions contained in the WSDL and WS-Adressing schemas. An identifier is used to communicate state information between requestors and endpoints. On top of WS-Resource advanced notification requests can be built. See: The WS-Resource Framework (Czaikowski, Ferguson et.al.)

38. Virtual Organizations VO‘s need to be created ad-hoc using a common technology for communication and security. The best know grid project globus (www.gobus.org) now switched to a web services approach for interoperability reasons. (diagram taken from „Security for Grids“ (Von Welch et. al.)

39. GUI re-use: Web Services for Remote Portals WSRP Consumer WSRP Producer WSRP Service Portal WSRP Service WSRP Service Aggregated HTML, WML, VoiceXML, ... over HTTP Mark-Up Fragments Transferred via SOAP • Portals can aggregate presentation from many WSRP services • WSRP services can be aware of portal context • User profile from portal • Desired locale and markup-type • User‘s device type from: OASIS WSRP Technical Committee, April 2002 (see resources)

40. Interesting Projects • Use Web Services to provide a common service layer for logging, auditing, user profiles in a multi-language intranet. • Use Web Services Component Model to create components for integration in portals. See oasis-open for proposals from Epicentric, IBM („Web Services Experience Language“) • Computational Grids are now using Web Services to express communications, security and resources (see www.globus.org ) • Create automated workflows for B2B. BPEL4WS and other standards try to automate business processes using Web Services. This goes along with better semantics through Web Ontology Language, RDF etc. Yes, both projects might be used in intranets first. But companies face the same integration problems internally as the experience in the B2B area.

41. Best Practice for Promoting Scalable Web Services 1. Stay away from using XML messaging to do fine-grained RPC. Forexample, stay away from a service which returns the square root of anumber. Stay away from a service that returns a stock quote (this isthe classic-cited example of a Web service). 2. Conversely, use course-grained RPC. XML web services usually have tobe defined at a coarser granularity than ordinary software objects. That is, use Web services that "do a lot of work, and return a lot ofinformation". 3. When the transport may be slow and/or unreliable, or the processingis complex and/or long-running, consider an asynchronous messagingmodel. 4. Always take the overall system performance into account. Don'toptimize until you know where the bottlenecks are, i.e., don't assumethat XML's "bloat" or HTTP's limitations are a problem untilthey are demonstrated in your application. 5. Take the frequency of the messaging into account. A high rate ofrequests might suggest that you load (replicate) some of the data andprocessing back to the client. The client occassionally connects tosynch-up with the server, and get the lastest data. 6. Aggregation using replication. There will be Web services whichserve to aggregate data from other Web services. One approach is toperform the aggregation on demand - the services which supply the dataare invoked in real time, the data is aggregated, and returned to arequesting client. Alternatively, all the data from the supplierservices may be retrieved during off-hours in one large, course-grainedtransaction. Thus, the aggregation is performed in real-time (ratherthan trying to retrieve the supplier data in real-time). The later isrecommended whereever possible. this is the result of an interesting discussion at xml-dev. Do you agree?

42. Other Web Services Architectural Models Representational State Transfer Architecture (REST), SOA and Policies are models beyond mere messaging. Diagram taken from „Web Services Architecture (w3c)

43. RESTful Web – against the RPC Model • The WEB is based on representation of resources using URIs, Web Services create private, non-standard ways of information access • The envelope paradigm does not add any value over the generic http get/put/post • RPC mechanisms are not suitable for the WEB. Some extensions to get/put/post might be necessary though (going in the direction of tuple-space systems) This is a hot topic currently: ask yourself whenever you think about building a web services: could it be done with just an http get or post? REST btw. stands for Representational State Transer Architecture, a term coined by Roy Fielding, the father of http. see resources on REST. But in later versions Web Services have been extended through a document centric model as well.

44. RESTful Web: CRUD like Message Semantics Representation Requestor Resource GET -> Read (idempotent, does not change server state) POST –> Create resource on the server PUT -> Update Resorce on the server DELETE -> Delete Resource on server Is this separation of updates and reads something new? Not by far. Bertrand Meyer of OO fame calls this a core principle of sound software design and made it a requirement for his Eiffel programming language. He calls it “”command-query separation principle”: “Commands do not return a result; queries may not change the state – in other words they satisfy referential transparency” B. Meyer, Software Architecture: Object Oriented Versus Functional [Meyer]

45. RESTful Web Features • Four strands that make a servive RESTful: • explicit use of http protocol in a CRUD like manner • stateless design between client and server • meaningful URIs which represent objects and their relationships in the form of directory entries (mostly parent/child or general/specific entity relations) • use of XML or JSON as a transfer format and use of content negotiation with mime types All state change is reflected by a change in representation. Resources are manipulated through a very simple and uniform interface (CRUD like) and through the exchange of representations. This is how the WWW works. A subset of Web Services are REST-compliant. From A.Rodriguez, see resources.

46. Policies – Making Services More Intelligent • Intermediates acting as trust-centers, proxies etc. What does a service allow? • Advanced collaboration between businesses. What does a business require? • Virtual organizations on top of physical organizations. What are the rules for participation? Complex business relations, frequently ad-hoc and fast established, need ways to express the constraints for each partner, transaction etc. This covers security policies just like reliability and availability expectations which are generally subsumed under the „QOS“ term.

47. For advanced services architecture and security technology for ad-hoc end-to-end services see the lectures on SOA and Web Services Security

48. Resources (1) • Clayton M. Christensen, The Innovators Dilemma. Shows how disruptive technology can turn successfull companies into loosers. Explaines why companies find it so hard to embrace new technology. This book has been so successfull that nowadays every discussion of new and badly understood technology uses the term „disruptive“. • www.webservices.org The Web Services Portal. It covers most Vendor products and has good overview articles. • xml-dev: mailing list for XML developers. High traffic site. Had a good discussion on XML-RPC performance lately • Security for Web services, Raghavan Srinivas, http://www.sun.com/developers/evangcentral/totallytech/Warsaw/SecurityWarsaw.pdf

49. Resources (2) • Programming Web Services with SOAP, J.Snell et.al., O´Reilly 2002. • www.oasis-open.org , Portal for ebXML and other XML schema definitions. Work on business transactions over web-services. • Global XML Web Services Architecture, Microsoft paper October 2001, www.gotdotnet.com (.net portal for web services) • Michael Stal, Web Services im Überblick, Objectspectrum 7/8 2001 • www.uddi.org, portal for UDDI.

50. Resources (3) • The IBM UDDI registry: http://www.ibm.com/services/uddi • Microsoft's UDDI registry: http://uddi.microsoft.com • Andre Tost, UDDI4J lets Java do the walking. Good introduction to the concepts behind UDDI • Steve Vinoski, Web Services and Dynamic Discovery, Article on webservices.org about the real difficulties with ontologies and automatic understanding. Yes, Steve is one of the fathers of CORBA and IONA´s chief architect. • P.J.Murray, Web Services and CORBA, CapeClear. Good explanation of the mapping problems when exposing CORBA services via Web Services.