First Indico Workshop

1. First Indico Workshop HTTP API 29-27 May 2013 CERN Adrian Mönnich

2. API Uhh… what’s an API and why do we need one? An application programming interface (API) is a protocol intended to be used as an interface by software components to communicate with each other. Wikipedia

3. Before the api export.py Not powerful at all: Only events inside categories XML, RSS, iCal or HTML  Accessing it requires an XML parser Protection? Not really  Must be restricted to trusted machines by IP Reusable code? None /export.py?fid=2l12&date=today&days=1000&of=xml

4. And the developers said……let there be a proper API……and they created one

5. Design ideas How to make the API awesome? • Handle authentication in a clean and secure way (API Keys, OAuth2) • Various security levels available (API keys, HMAC signatures, …) • Let developers write code for the actual task • Serialize Python data structures as JSON, XML, … • Provide utilities for common things (relative dates, pagination)

6. Using the API Before we go into details, look at how it works • Indico Administrators: • API Modes: • Control how strict the API key requirements are • At CERN: We always require an API key, but a signature is only needed to • access non-public information • Persistency: • Lowers security a little bit (one leaked URL is valid forever) • Allows people to use e.g. a RSS reader (cannot sign requests)

7. Using the API Before we go into details, look at how it works • Indico Users: • Creating an API key is easy – just one click! • Keys can be reset at any time, e.g. in case of a (suspected) leak • Last request including IP address and full path is displayed • Persistence can be toggled (if enabled globally)

8. Using the API Before we go into details, look at how it works • The same data – available as XML, JSON and sometimes additional more formats (iCal, RSS, HTML) / export/categ/4751.[json|xml|html]

9. Using the API Before we go into details, look at how it works • What it we need a signed request? /export/categ/123.json?limit=10 • 1. Add the API key to the params …&ak=api-key • 2. Add the current timestamp to the params …&timestamp=1234567890 • 3. Sort the query string paramsalphabetically …?ak=api-key&limit=10&timestamp=123… • 4. Merge path and the sorted query string • /export/categ/123.json?ak=api-key&limit=10&timestamp=1234567890 • 5. Create a HMAC-SHA1 signature of this string using the secret key as the key. • 6. Append the hex-encoded signature to the query string: • …&signature=xxx

10. EXTENDING THE API Creating a custom API endpoint • Let’s keep it simple: We want an API that returns a range of numbers or characters. • This does not access any indico-specific data but demonstrates some of the utilities available when using the API as a developer. / export/num/1-10.json /export/char/a-z.json

11. EXTENDING THE API

12. EXTENDING THE API Relevant classes • HTTPAPIHook: Hooks into the /export/ URL subtree, handles arguments • IteratedDataFetcher: Provides pagination, sorting, relative dates, etc. • IFossil: Describes how an object is converted to primitives (dict, list, int, str, bool) that can be serialized (JSON, XML, …) later

13. EXTENDING THE API

14. EXTENDING THE API The hook, Pt. 1: Metadata @HTTPAPIHook.register class RangeHook(HTTPAPIHook): TYPES = ('num', 'char') RE = r'(?P<start>[0-9]+|[a-z])-(?P<end>[0-9]+|[a-z])' DEFAULT_DETAIL = 'simple' MAX_RECORDS = { 'simple': 10, 'palindrome': 5 }

15. EXTENDING THE API The hook, Pt. 2: Arguments def_getParams(self): super(RangeHook, self)._getParams() self._start = self._pathParams['start'] self._end = self._pathParams['end'] • Perform additional validation if necessary • Use self._pathParams for arguments from the path regex • Use self._queryParams to access arguments in the query string

16. EXTENDING THE API The hook, Pt. 3: The export methods defexport_num(self, aw): try: start = int(self._start) end = int(self._end) except ValueError: raise HTTPAPIError('Invalid value', 400) return RangeFetcher(aw, self).numbers(start, end) defexport_char(self, aw): if len(self._start) != 1 or len(self._end) != 1: raise HTTPAPIError('Invalid character', 400) return RangeFetcher(aw, self).chars(self._start, self._end) • Perform actions/validations specific to the export type • Must return an iterator, usually provided by an IteratedDataFetcher

17. EXTENDING THE API

18. EXTENDING THE API The fetcher, Pt. 1: Metadata class RangeFetcher(IteratedDataFetcher): DETAIL_INTERFACES = { 'simple': IDummyFossil, 'palindrome': IDummyPalindromeFossil } • Remember DEFAULT_DETAILand MAX_RECORDS? Here we specify how objects are fossilized in those detail levels. • More about fossilization later, let’s get some data first!

19. EXTENDING THE API The fetcher, Pt. 2: Python iterator magic def numbers(self, start, end): iterable = xrange(int(start), int(end) + 1) iterable = itertools.imap(str, iterable) iterable = itertools.imap(DummyObject, iterable) return self._process(iterable) def chars(self, start, end): iterable = itertools.imap(chr, xrange(ord(start), ord(end) + 1)) # Besides that first line everything is the same like in numbers() • Remember: We call these methods from our export methods. • In this case we just iterate over numbers/characters in the given range. • DummyObject? You will see soon…

20. EXTENDING THE API The Dummy Object class DummyObject(object): def __init__(self, value): self.value = value defgetValue(self): return self.value • Usually we deal with custom objects in Indico. Because of historical reasons it is common to have getter methods (yes, we know it is ugly and unpythonic). • The fossilization system is built around this  thin wrapper providing a getter for our primitive string values.

21. EXTENDING THE API

22. EXTENDING THE API Fossils class IDummyFossil(IFossil): defgetValue(self): pass class IDummyPalindromeFossil(IDummyFossil): defgetPalindrome(self): pass getPalindrome.produce = lambda x: x.value + x.value[::-1] • IDummyFossil is really simple: It fossilizes the getValue() return value of the actual object to a field named value. • For the palindrome detail level we specify a custom callable to generate the value on the fly. The field name is determined automatically again.

23. EXTENDING THE API Does it work? / export/num/128-256.json?detail=palindrome /export/num/128-256.json /export/num/1-1.xml

24. EXTENDING THE API Yes! Did you notice the pagination? / export/num/128-256.json?detail=palindrome /export/num/128-256.json /export/num/1-1.xml

25. What else The API is even more powerful! • Caching (unless disabled by the user per-request) • POST (to modify things, e.g. booking a room) • Additional URL prefixes besides /export/ (e.g. /api/) • Many things (like the export_* method name) can be changed easily when subclassingHTTPAPIHook • Reading the existing code is the best way to get used to it!

26. EXTENDING THE API

27. EXTENDING THE API Another example, using POST @HTTPAPIHook.register class AnnouncementHook(HTTPAPIHook): PREFIX = 'api' TYPES = ('announcement',) RE = r'set' GUEST_ALLOWED = False VALID_FORMATS = ('json', 'xml') COMMIT = True HTTP_POST = True def _getParams(self): super(AnnouncementHook, self)._getParams() self._message = get_query_parameter(self._queryParams, ['message'], '') defapi_announcement(self, aw): am = getAnnoucementMgrInstance() am.setText(self._message) return {'message': self._message}

28. Questions? Adrian Mönnich https://github.com/ThiefMaster View the example code on GitHub:https://goo.gl/btdVc [Diff] https://goo.gl/wGJXG [Gist]