1 / 41

AMQP, Erlang, and RabbitMQ

AMQP, Erlang, and RabbitMQ. OOI Cyberinfrastructure Design Meeting San Diego, 17 th -19 th October 2007. Matthias Radestock matthias@lshift.net. distributed reliable & resilient manageable secure. So you want to build a computing infrastructure for ocean observation ….

woody
Download Presentation

AMQP, Erlang, and RabbitMQ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. AMQP, Erlang, and RabbitMQ OOI Cyberinfrastructure Design Meeting San Diego, 17th-19th October 2007 Matthias Radestock matthias@lshift.net

  2. distributed reliable & resilient manageable secure So you want to build a computing infrastructure for ocean observation … …that transfers, routes, transforms, processes and stores a variety of data streams, and works,so it needs to be evolves,so it needs to be • scaleable • portable • pluggable • interoperable … like an enterprise messaging system!

  3. AMQP Everyone uses email without thinking, so then why is universal commercial business messaging so hard? “business dialtone” → need an OpenStandard Protocol for Message Oriented Middleware AMQP aims to become THE standard for enterprise messaging • Made to satisfy real needs: • created by users and technologists working together • in development for 3+ years, went public on June 20th 2006

  4. AMQP’s place in the “stack” Product Layer Use Standard User Applications bank CDL-BPM-CEP treasury ESB / SOA / EDA settlement WS-* “Document Model” reports FpML WCF “Rich Object Model” positions Transaction participant trades FIX, FAST MQ Store & forward orders JMS AMQP Guaranteed Delivery send/ack AMQP Tibco Content based routing (1-Many, 1-1) ticks Utility Services e.g. exchange access links Cisco Network (TCP/IP, UDP, SCTP) data

  5. Comparison with some other protocols • SMTP – unreliable, slow • HTTP – synchronous, unreliable, no routing • XMPP – no delivery fidelity or queue management • FTP – point to point, transient, does not work well with NAT/SSL • MQ – exactly once • TCP – at least once, reliable but short lived, no app level state mgmt • UDP – fast but has no delivery guarantees AMQP - can do all of the above as ‘use cases’ … and switch between them

  6. How does it work? Queue File Transfer Queue Exchange Messaging Queue Bindings Transactions • any language (C, C++, Java, C#, Python, Javascript, Erlang, Lisp, Ruby, Tcl, PHP, …) • any model (native, JMS, Mule, WCF, …) • any payload (binary, XML, SOAP, JSON, …) • any transport (TCP, SCTP, HTTP, …) • any platform (desktop, router, mobile, EC2, …) • reliable • interoperable • manageable • performant • scaleable

  7. A typical AMQP client program # Addresses where a particular broker is reachable. endpoints = [amqp.Endpoint('hostname', 5672), amqp.Endpoint('alternate', 5673)] # Construct a session. session = amqp.Session("1234", 600, endpoints) # Set up some resources on the broker. session.ExchangeDeclare("xname",amqp.ExchangeTypes.DIRECT) session.QueueDeclare("qname") session.QueueBind("xname","qname","routingkey") # Publish a message. session.MessageTransfer("xname","routingkey","body") # Set up a consumer. session.MessageConsume("qname",MyConsumer(session)) class MyConsumer: def __init__(self,session): self.session = session def handle_MessageTransfer(self,destination,exchange, routingkey,body): print "Received message: " + body return amqp.Constants.ACCEPT

  8. Under the covers – AMQP Layering Application queues, exchanges, messages, transactions, … API calls Model command segmentation / assembly, ordering and acknowledgement commands Execution frameset sequencingframeset (dis)assemblyframe flow controlreliability (exactly once) and failover framesets Session frames Framing (de)multiplexing by channel/stream heartbeating framing, integrity check bytes / packets Transport

  9. Communication between peers messages Application Application Model commands Model controls Execution Execution control frames Session Session control frames Framing Framing Transport

  10. Under the covers: session reattachment

  11. Apache AMQP - Inclusive Governance Products Protocol JPMorgan D/Borse Credit Suisse Goldman Sachs TWIST Novell 29West Envoy Community Feedback Cisco Network iMatix OpenAMQ Red Hat Enterprise Messaging IonaCeltixAM RabbitMQ AMQP Working Group controls the standard Diverse products implement the standard

  12. distributed reliable & resilient manageable secure So you want to build a computing infrastructure for ocean observation … …that transfers, routes, transforms, processes and stores a variety of data streams, and works,so it needs to be evolves,so it needs to be • scaleable • portable • pluggable • interoperable … like a telecommunications system!

  13. Erlang History 2007: A NewBook! 1996: Open Telecom Platform No language was well suited for telecom systems development 1998: Open Source Erlang 1994: First Product 1993: Distributed Erlang 1987: Early Erlang Prototype projects 1993: The First Book 1984: Ericsson Computer Science Lab formed 1991: First fast implementation

  14. Erlang Highlights High abstraction level Concise readable programs No mutable state • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable

  15. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable Light-weight processes Highly scalable Message Passing

  16. Erlang Highlights: Concurrency Processes communicate by asynchronous message passing receive {start} -> ……… {stop} -> ……… {data,X,Y} -> ……… end Pid ! {data,12,13}

  17. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable Response times in the low milliseconds Per-process garbage collection

  18. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable Simple and consistent error recovery Supervision hierarchies "Program for the correct case"

  19. Erlang Highlights: Robustness Cooperating processes may be linked together

  20. Erlang Highlights: Robustness When a process terminates, an exit signal is sent to all linked processes … and the termination is propagated

  21. Erlang Highlights: Robustness Exit signals can be trapped and received as messages receive {‘EXIT’,Pid,...} -> ... end

  22. Erlang Highlights: Robustness Robust systems can be built by layering “Supervisors” “Workers”

  23. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable Explicit or transparent distribution Network-aware runtime system

  24. Erlang Highlights: Distribution B ! Msg C ! Msg Erlang Run-Time System Erlang Run-Time System network

  25. Erlang Highlights: Distribution Simple Remote Procedure Call {rex, Node} ! {self(), {apply, M, F, A}}, receive {rex, Node, What} -> What end loop() -> receive {From, {apply, M, F, A}} -> Answer = apply(M, F, A), From ! {rex, node(), Answer} loop() end.

  26. Erlang Highlights Easily change code in a running system Enables non-stop operation Simplifies testing • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable

  27. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable "Ports" to the outside world behave as Erlang processes

  28. Erlang Highlights: External Interfaces External process Port Port ! {self(), {command, [1,2,3]}},receive {Port, {data, Info}} -> ...end

  29. Erlang Highlights • Functional • Concurrent • Soft real-time • Robust • Distributed • Hot code loading • External interfaces • Portable Erlang runs on a Virtual Machine ported to UNIX, Windows, VxWorks, OS X, … Supports heterogeneous networks.

  30. T O P Open Telecom Platform Applications, Libraries & Tools System Design Principles

  31. Credits • Alexis Richardson, CohesiveFT –for providing much of the AMQP material • Francesco Cesarini, Erlang Consulting – for providing much of the Erlang material • Tony Garnock-Jones, LShift –for the sample code • Matthew Arrott – for inviting me Thank you!

  32. Backup slides

  33. AMQP features • In some ways like email but: • What goes in must come out • Very fast - think big - global scale communication • In some ways like TCP and HTTP, but delivers true MESSAGING • Routing and addressing - “to:phil@cohesiveft.com”, “buy.ibm.100” • Guaranteed Delivery • Delegation - the concept of a middleman → security, reliability, translation, …

  34. AMQP version history and roadmap

  35. Under the covers: session creation

  36. Under the covers: configuration

  37. Under the covers: message transfer

  38. Mule binding <mule-configuration id="RabbitMQ_Demo" version="1.0"> <model name="rabbitmqDemo"> <mule-descriptor name="Mulebot“ implementation="com.rabbitmq.examples.muledemo.Mulebot"> <inbound-router> <endpoint address="amqp://localhost/?vhost=/&amp;realm=/data&amp; exchange=chat&amp;exchange-type=fanout&amp;routing-key="/> </inbound-router> <outbound-router matchAll="true"> <router className="org.mule.routing.outbound.OutboundPassThroughRouter"> <endpoint address="amqp://localhost/?vhost=/&amp;realm=/data&amp; exchange=chat&amp;exchange-type=fanout&amp;routing-key="/> </router> </outbound-router> </mule-descriptor> </model> </mule-configuration>

  39. WCF binding service = new ServiceHost(typeof(Calculator), new Uri("soap.amq://dev.rabbitmq.com:5672/")); service.AddServiceEndpoint(typeof(ICalculator), new RabbitMQDualBinding(), "Calculator"); service.Open(); fac = new ChannelFactory<ICalculator>( new RabbitMQDualBinding() { ClientBaseAddress = new Uri("soap.amq://dev.rabbitmq.com:5672/")}, "soap.amq://dev.rabbitmq.com:5672/Calculator"); fac.Open(); calc = fac.CreateChannel(); Console.WriteLine("{0} + {1} = {2}", x, y, calc.Add(3, 4)); ((IChannel)calc).Close(); service.Close();

  40. Erlang Highlights: High-level Constructs Parsing an IP Datagram using the Bit Syntax -define(IP_VERSION, 4). -define(IP_MIN_HDR_LEN, 5). ……DgramSize = size(Dgram), <<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16, ID:16, Flgs:3, FragOff:13, TTL:8, Proto:8, HdrChkSum:16, SrcIP:32, DestIP:32, Body/binary>> = Dgram, if (HLen >= 5) and (4*HLen =< DgramSize) -> OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN), <<Opts:OptsLen/binary, Data/binary>> = Body, ….. end.

More Related