Coding sprint am api updates details
1 / 40

Coding Sprint AM API Updates Details - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Coding Sprint AM API Updates Details. Sarah Edwards November 4, 2011 AM API Status Review. Wednesday AM API Session discussed AM API changes AM API version 2 seems (mostly) settled Include Change Set A: RSpecs are in XML Include Change Set B Part 1: New options argument

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Coding Sprint AM API Updates Details

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Coding SprintAM API Updates Details

Sarah Edwards

November 4, 2011

AM API Status Review

  • Wednesday AM API Session discussed AM API changes

  • AM API version 2 seems (mostly) settled

    • Include Change Set A: RSpecs are in XML

    • Include Change Set B Part 1: New options argument

    • Include Change Set B Part 2: Richer return structures

      • If we can address the questions that remain today

    • Other topics require further discussion

  • Do we agree – AM API v2 is those changes?

  • When can SFA, PG, FOAM, and Orca implement these changes by?

Change Set B Issues

  • Error codes – Integers or Strings?

  • How are error codes documented?

  • Supporting multiple AM API versions

  • How are new options or returns documented?

  • Define GENI error codes

1. Error codes – type?

  • Proposal says error codes are integers with the space divided by AM type

  • Counter proposal: use namespaces to divide the space, making codes strings

    • No ‘lost’ code space when an AM can’t use 1000 codes

    • Larger code range possible

    • Code indicates explicitly the namespace


  • Discuss

2. Documenting Error Codes

  • Standard error codes must be documented

  • Proposal: a wiki page off of GAPI_AM_API on the wiki

  • 2nd Proposal: An XML document attached to the GAPI_AM_API wiki page

3. Supporting Multiple API Versions

  • API v2 is incompatible with v1

  • Some AMs may want to be nice and support both. More generally, AMs may want to support multiple API versions.

  • Since method semantics may change without signature changes, AM must know what version the client is talking

  • PG uses different URLs for different versions, eg

  • vs

  • If we do that, how do AMs advertise available API versions and URLs?

3. (continued)

  • Proposal: This change need not hold up API v2 – handle independently

  • Proposal: Use separate URLs, advertised in the GetVersion return of all API versions:

    geni_api_versions: {

    1: <URL>,

    2: <URL>,



4. Documenting New Options, etc

  • Aggregates should document any new optional ‘options’ or new return values they support

    • XMLRPC Introspection (methodHelp)?

    • GetVersion return?

    • Both?

    • Do we need to standardize?

    • Do we need to register option/return names or have namespaces?

  • Proposal: format remains ad-hoc. AMs using GetVersion are encouraged to use a format like the following

4. (continued): Sample Ad

  • A possible option advertisement in GetVersion:

  • methodOptions:

  • {

    • GetVersion: {

      • verbose: {

        • type=boolean,

        • description=“True means supply gory detail on AM internals and versions."

      • },

      • myOtherNewOption:..... },

    • ListResources....

  • }

5. GENI Error codes

  • Some (most) error codes will be standard across all AMs. We should pick those soon.

  • PG has some error codes it uses

  • Aaron H proposed a different classification on the geni-dev mailing list

  • Code format depends on the resolution of issue #1





















"Bad Arguments",


"Operation Forbidden",

"Bad Version",

"Server Error",

"Too Big",

"Operation Refused",

"Operation Times Out",

"Database Error",

"RPC Error",


"Search Failed",

"Operation Unsupported",



"In Progress",

"Already Exists",


5. (continued): PG error codes

5. (continued): Alternative Errors

  • Not authenticated

    • (do any calls not require authentication, such that we could ever get to the application layer anyhow?)

  • Not authorized

    • (with the following sub-types that could get their own codes for clarity)

    • Credential Expired

    • Credential not trusted

    • Missing credential

    • Insufficient Privileges

  • No such sliver

  • Malformed RSpec

  • Invalid Datetime

  • Resources unavailable, can't allocate

  • Unsupported operation

5. Alternative errors continued

  • Server error (PG error 5)

  • Server busy (PG error 14)

  • Other malformed request (is this PG error 1?)

  • Operation timed out (PG error 8)

  • Resource busy (or locked, which is different then server busy). (PG error 14 again)

  • Search Failed (as when a slice or sliver does not exist) (PG error 12)

    • or is this the same as 'no such sliver' above?

  • Already Exists (inverse of above). (PG error 17)

Change Set B Part 2 Approved?

  • Given this discussion….

  • Can we adopt Change Set B Part 2 – methods return a structure – for inclusion in AM API version 2?

Change Sets A&B Review

Change Set A: RSpecs

  • Aggregate Managers must advertise what RSpec formats they support

  • AM API must specify that the RSpec arguments and returns are in GENI standard format (GENI v3)

    • Was ProtoGENI v2

    • Same schemas, now on with a new namepace

    • SFA, emulab-devel, Flack, and Omni support in place

  • Jon Duerig of ProtoGENI proposed AM API revisions to support this and we discussed it on

  • Published to GENI wiki and


  • Revisions will be in AM API v2

Change Set B Part 1: New Options

  • Note: This set of 2 distinct changes is currently under discussion and has gotten no official or unofficial agreement.

  • Part 1: Additional options argument

  • Motivation:

    • Support innovation

    • Support aggregate/resource specific configuration options

    • Examples: Tree or chain mode in stitching; whether user keys in CreateSliver add or replace existing keys on the slice

  • Proposal

    • Add an XMLRPC struct (name-value pairs) called options

    • No required options, but argument is required (except in GetVersion)

Change Set B Part 1: Semantics

  • Aggregates are compliant with this API change by accepting this argument

  • Clients are required to supply this argument

  • In GetVersion, this argument remains optional.

    • V1 clients will get errors calling other methods

    • V2 AMs will include the geni_api argument as a top-level entry in the return struct, specifying 2 to indicate to clients that this AM speaks version 2 of the AM API. This allows experimenters to understand that they need to upgrade their client, or might instruct a clever client tool to automatically switch to version 2 syntax.

  • Aggregates should not require any new options to any method

    • Use defaults for any options

    • Clients must always be able to work with any aggregate using the AM API

Change Set B Part 1 continued

  • Aggregates are encouraged to document any new options

    • to bootstrap coordination with clients, and

    • provide documentation for human experimenters.

    • One way to provide partial documentation, is to implement XML-RPC introspection.

      • Through the use of method help, aggregates can provide human readable text describing options.

    • Alternatively or additionally, aggregates may provide a capability descriptor as part of their return from GetVersion.

      • We have not specified the format for advertising this capability descriptor and those extra options in GetVersion.

Change Set B Part 1: Sample Ad

  • A possible option advertisement in GetVersion:

  • methodOptions:

  • {

    • GetVersion: {

      • verbose: {

        • type=boolean,

        • description=“True means supply gory detail on AM internals and versions."

      • },

      • myOtherNewOption:..... },

    • ListResources....

  • }

Change Set B Part 1: Discussion To Date

  • Points from email discussion

    • Not ioctl; use the server to determine you have a valid method signature

    • Don’t add new methods for each new option – that isn’t an API

    • Could have just allowed arbitrary optional arguments - equivalent

    • AMs should advertise new options

      • XMLRPC introspection, but that gives us only the help string

      • GetVersion (not standardized)

        • Could specify syntax during coding sprint

      • Nothing specific required

  • No significant objections on the mailing list

  • AMs must not require any new options – provide defaults

  • General agreement on details

  • Does anyone have new concerns, comments or objections that they want to raise now?

Change Set B Part 2: Return Structures

  • Change Set B Part 2: Methods return rich structures

  • Motivation

    • AM API v1 method failures are inconsistent, confusing

      • XMLRPC faults wrapping messages from the server

      • Sometimes returns ‘False’ with no feedback

  • Proposal

    • Expand and formalize returns to give better feedback on both failure and success

  • AMs will

    • Return more information

    • Support richer client-server communication.

    • Give clients hints on how to use successful returns,

    • Innovate within the bounds of the AM API.

Change Set B Part 2: Proposal

  • The proposal changes the return values of all methods

  • Return an XMLRPC struct (aka property list)

    • On any application layer success, failure, and even on an error or for most exceptions.

    • Note that a malformed XMLRPC request should still raise an XMLRPC Fault, and other Faults dictated by the XMLRPC specification should still be raised.

    • This struct will contain the return value from the previous revision of the AM API as an entry. This struct will have 3 defined entries, and aggregates are free to include other entries to give more information to clients.

Change Set B Part 2: Sample Return

  • For example, SliverStatus could return on success:

  • {

    • code: 0

    • value: {

      • geni_urn: <sliver URN>

      • geni_status: ready

      • geni_resources: [ { geni_urn: <resource URN> geni_status: ready geni_error: ''}, { geni_urn: <resource URN> geni_status: ready geni_error: ''} ]

      • }

  • output: <none>

  • }

  • Change Set B Part 2 continued

    • On failure, DeleteSliver might return:

    • (That code and output are merely examples.)

    • An exception: At the top level, GetVersion adds a required entry: 'geni_api'=2.

      • This allows v1 clients to determine that they are indeed talking to a GENI AM, but since the version is 2, that is why other function calls will fail.

    • {

      • code: 1

      • value: False

      • output: 'No such slice here'

    • }

    Change Set B Part 2: Specification

    • The three required entries in the return structure are code, value, and output:

    • code: An integer, non-zero on error.

      • 0 = Success

      • <0 = XMLRPC required error codes

      • 1-1000 = GENI negotiated return codes (none so far)

      • 1001-2000 = ProtoGENI specific return codes

      • 2001-3000 = PlanetLab specific return codes

      • 3001-4000 = Orca specific return codes

      • 4001-5000 = OpenFlow specific return codes

      • others as needed

    Change Set B Part 2 Spec. continued

    • value: On success, this is required. Optional on failure or error. Object representing the successful return value. The value is not defined on error, though aggregates are free to use it.

      • For GetVersion, the value is an XMLRPC struct

      • For ListResources, the value is an RSpec

      • For CreateSliver, the value is an RSpec

      • For RenewSliver, the value is a boolean

      • For DeleteSliver, the value is a boolean

      • For SliverStatus, the value is an XMLRPC struct

      • For Shutdown, the value is a boolean

    • output: On failure or error, this is required. Optional on success. This is a String with a human readable message explaining the result.

      • Specifically, this might include an error string, a stacktrace, or other useful messages to help the Experimenter resolve or report the failure or error. It is not defined on success, though aggregates are free to use it.

    Change Set B Part 2 continued

    • AMs should use code values and output messages that help experimenters distinguish among bad input, temporary server errors, or server bugs.

    • AMs may use returns to suggest solutions to failed requests

      • A failed RenewSliver call might return a new date string in the value field that would be allowed.

      • A failed CreateSliver call might return a modified request RSpec in the value field.

    • Avoid raising an error (XMLRPC Fault) for application layer errors or any other cases where the XMLRPC specification does not require a Fault

    • For comparison,

      • Orca functions return property lists internally.

      • The ProtoGENI CMv2 API returns a struct with exactly these 3 values. ProtoGENI however uses a different range of return codes, and largely does not define the value slot on errors.

    Change Set B Part 2: Agreed?

    • No significant objections on the mailing list

    • Details seem fairly worked out

    • To do: GENI standard error codes

      • Start with ProtoGENI codes as starting point?

      • Discuss at the coding sprint?

    • Compatibility

      • AMs will likely switch to AM API v2, not support both versions

      • AMs can support both if they choose

        • Note that ListResources arguments have not changed, so the AM may not know which API version is desired without a further hint, like the server URL

      • Clients will need to support both v1 and v2 clients if they want to talk to all aggregates during the transition

      • Future API changes will have similar transitions

    • Does anyone have new concerns, comments or objects that they want to raise now?

    Other API Discussion Topics Follow

    Change Set C: UpdateSliver

    • Change Set C: Add UpdateSliver

    • Motivation

      • Coffee girl: ‘make my slice bigger’

      • In current API, you must delete your slice and recreate it – possibly losing state and even resources

    • Approaches to UpdateSliver in the community

      • Orca: can modify resources, depending on the resource. Working on add/remove resources

      • PG: UpdateSliver in CMv2 API. Takes request RSpec, returns a ticket

        • Experimenter separately redeems the ticket and restarts the sliver

        • Aggregate computes the difference from the existing state

      • PL: SFA has UpdateSlice, which is a synonym for CreateSliver in AM API.

    Change Set C: Proposal

    • Proposal

      struct UpdateSliver(string slice_urn,

      string credentials[],

      string rspec,

      struct users[],

      struct options)

      Success Return:



      value= <GENI v3 manifest RSpec string>

      output = <None>


    Note: rspec is in the GENI v3 request schema

    Change Set C: Semantics

    • Proposed Semantics:

      • RSpec argument is a complete request RSpec

        • Question: Can we support a request from an edited manifest easily?

        • Should we instead support 2 modes: complete and diff?

      • Atomic operations: all changes succeed or all fail, with an explanation of why it failed and how to modify the request to succeed if possible

      • AM automatically starts/restarts resources

        • Do not wait for tickets or adding Start and Restart calls to the API. But perhaps support that variation later

      • AM API requires no guarantees about whether unchanged resources in the reservation will be moved or restarted

        • A per aggregate/resource decision

        • For example, aggregates may load balance VMs

        • Aggregates should document any guarantees or behavior in this respect

    Change Set C: Discussion Points

    • Differences and options

      • Take full desired state or a delta? Who calculates the delta? Who synchronizes requests among multiple experimenters within a slice?

        • Ilia and Nick argued we should support BOTH deltas AND full RSpecs

        • PG argues that new resource requests to link to existing reserved resources are hard to express, and merging RSpecs is hard

      • Should the AM partially fulfill requests? (Give you 3 out of 5 new nodes, while deleting the ones you asked to drop)

        • But giving you nodes without a link to connect them is silly

      • Does the AM restart nodes immediately? Or does the experimenter pick which nodes to restart and when?

        • Leigh and Rob argued the AM should not restart nodes

      • RSpec: request? Or a manifest, to allow mods to what you got?

        • PG specifies a request but accepts manifests.

    We must discuss!

    Change Set C: Discuss

    • Any new concerns? New comments or suggestions?

    • Approaches to address Ilia, Leigh, and Rob’s comments?

    • Agree to include in AM API v2?

    Change Set D: Slivers and sliver groups

    • Problem: Current API methods take a Slice URN

      • Create, delete, renew everything in the slice at an Aggregate.

      • No way to add to a slice at an aggregate, delete part of a slice at an aggregate, etc.

    • Definitions

      • Sliver: Some of the resources at one aggregate in one slice

        • NOT ALL the resources at the aggregate in the slice

        • Typically the smallest set of resources that the AM will independently reserve or allocate

        • 1 Slice may have multiple slivers at a single aggregate

        • 1 Sliver may contain multiple components

    • ProtoGENI definitions for comparison:

      • AggregateSliver: all resources in the slice at an aggregate

        • This is what CreateSliver creates

      • ComponentSliver: things with a sliver_id, roughly 1 resource in each

    Change Set D: Function implications

    • This change would make methods operate on individual slivers or on other groups of slivers – specifically CreateSliver, DeleteSliver, RenewSliver, and new methods not yet in the API

    • UpdateSliver gives us the ability to add/remove slivers within an aggregate

    • Start, Stop, Restart and similar functions don’t exist in the AM API – yet

    • RenewSliver is not addressed (yet)

    Change Set D: Proposing Nothing

    • Proposal: Given we add UpdateSliver, do nothing else (for now)

      • CreateSliver gives a bunch of slivers, DeleteSliver deletes all slivers at the AM, and we only support Renewing all slivers at an AM

    • Anyone have serious concerns with not changing the AM API to address this now?

    • We can discuss this on the GENI dev mailing list and include changes in a future API revision

    Change Set E: Tickets

    • What are tickets?

      • A promise to deliver (like a reservation)

      • Proof of purchase (like a receipt that can be turned in for your goods)

      • A way to reference and trade resources without physically moving boxes

      • A feature which, in name, appears in Orca, SFA, PlanetLab, and ProtoGENI

    • Benefits and features of tickets

      • Support brokers to allocate, consolidate resources on behalf of others

      • Allow transferring, lending resources

      • Support scheduled reservations

      • Negotiated reservations, or coordinated reservations across aggregates

      • 2 phase commit

    Change Set E: Implementations

    • Implementations in the community

      • Concept defined in SFA as a signed RSpec that promises to allocate resources.

      • ProtoGENI CM interface constructs them similarly, using them for 2 phase commit

      • Orca uses leases and tickets extensively for brokering, lending resources and for negotiated reservations / 2 phase commit. But they are structured differently.

    • Parts of a proposal

      • Structure

      • Semantics

      • Additional ticket related methods or changes to existing methods

  • Login