1 / 27

(In)security in URL shortening and redirection services

(In)security in URL shortening and redirection services. Anthony Bettini McAfee Labs 6 May 2011 – CARO 2011 Workshop. Table of Contents Analysis of a URL shortening and redirection provider. What is a URL shortening and redirection provider? (Shortening vs. Redirection)

rocco
Download Presentation

(In)security in URL shortening and redirection services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. (In)security in URL shortening and redirection services Anthony Bettini McAfee Labs 6 May 2011 – CARO 2011 Workshop

  2. Table of Contents Analysis of a URL shortening and redirection provider • What is a URL shortening and redirection provider? • (Shortening vs. Redirection) • Who are the popular providers? • Who uses a shortening provider and why? • Hypotheses • Case study: Deep-dive of a popular shortening provider • Hardening the Net • For malware researchers, why does all this (app-level security) matter? • Future research areas • Conclusions • Questions 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  3. What are we talking about? URL shortening and redirection what? • Technically, they are two separate services, although generally all providers combine them into one seamless service via APIs • Shortening concatenates a string • Redirection does an HTTP 3xx redirect • Shortening example: • We take this string: • http://www.gadgetguy.com.au/small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-sunrise-australia-42.html • We are returned this: • http://bit.ly/2Ngz9Q 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  4. What are we talking about? Redirection as compared to shortening • Redirection example: • We make an HTTP request for this, via an HTTP connection to bit.ly:80: • GET /2Ngz9Q HTTP/1.1 • (with other appropriate HTTP headers) • Our client gets an HTTP 3xx code and makes a connection to: • www.gadgetguy.com.au:80 • While making a seamless request for: • /small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-sunrise-australia-42.html • Throughout the remainder of the presentation, we’ll just say shortening, when we really mean shortening & redirection (we do this to save space!)  6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  5. Who are the players in the game? Crowded space with big names • Most widely used providers: • bit.ly (bitly – startup, widely used) • goo.gl (Google) • fb.me (Facebook) • t.co (Twitter) • There are many other less popular ones… 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  6. Who is shortening URLs and why? Only consumers, right? • Users of various Twitter clients are having their URLs auto-shortened to stay within the 140 character limit • Facebook and other pages are doing similar things, for similar reasons • Corporate users are increasingly using shortening providers to convert long internal URLs (such as direct links to docs buried on an internal Sharepoint) to to ease sending a link to a coworker • Evidence of SPAM and malware authors leveraging shortening providers to mask URLs from being immediately read as suspect by users (making programmatic URL checking/filtering more important) 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  7. Hypotheses These came before the testing, really • Complexity of the systems would increase over time as competition in this space increased • Developers building URL shortening and redirection applications had limited exposure to real world security attacks; same for QA • This limited understanding would result in mistakes, from a defense, safety, and security standpoint • Testing for a wide variety of imperfections, across all providers, would have surprising security findings • Summary: It would be easy to find INFOSEC problems 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  8. Case study: Analysis of a major shortening service Overview • Examined one of the major URL shortening and redirection providers for security implementation limitations (weaknesses) • Many provide APIs that require API keys • Reading the API documentation gives a good understanding of the system • API keys exist to: • Track behavior • Rate limit requests • Restrict functionality • We skipped the API key and wrote our own HTTP client • Problem solved 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  9. Case study: Analysis of a major shortening service Some of the tests we ran, at a high-level • Example tests we ran (basic unit testing, primarily for security): • Shortening of invalid URLs • Space efficiency when shortening strings with meaningless repetitions • Sequence predictability for keys in the dictionary • Shortening of self-referential URLs • Quick note on dictionary sizing • CAPTCHA requirements • Updating the source URL • Attempting to obtain competitive intelligence from providers on marketing campaigns 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  10. Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten invalid URL (invalid based on TLD) • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: www.microsoft.ny -> http://svc/a0ewwA • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  11. Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten what obviously isn’t a URL • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: junk -> INVALID_REQUEST • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  12. Case study: Analysis of a major shortening service The findings… • TEST NAME: The same URLs should have the same result • EXPECTATION: GOOD_URL should be properly shortened, SAME_URLS should receive the same result • RESULT (additional related findings on next slide): • Result for GOOD_URL[0]: www.bing.com -> http://svc/9kkMVF • Result for SAME_URLS[0][0]: wWw.bing.com -> http://svc/b43j5I • #fail • Result for SAME_URLS[0][1]: http://www.bing.com -> http://svc/9kkMVF • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  13. Case study: Analysis of a major shortening service The findings… • RESULT (continued): • Result for GOOD_URL[1]: www.google.com/advanced_search -> http://svc/aRxMmM • Result for SAME_URLS[1][0]: www.google.com////advanced_search -> INVALID_REQUEST • #fail • Result for SAME_URLS[1][1]: www.google.com///advanced_search -> http://svc/alc49z • #fail • Result for SAME_URLS[1][2]: www.google.com//advanced_search -> http://svc/dpTc57 • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  14. Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten the shortener’s URL (self-reference) • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: http://svc/9kkMVF -> INVALID_REQUEST • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  15. Case study: Analysis of a major shortening service The findings… • Quick note on dictionary/key space sizing, which we’ll discuss later • Provider A uses a default keyspacesize, excludingcustomURLs (we’ll get tothis later), of: 62^6 ([A-Z][a-z][0-9]^[fixednumber of characters]) [56 billion] • Provider B uses a default keyspacesize of: 36^6 [2 billion] 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  16. Case study: Analysis of a major shortening service The findings… • TEST NAME: Quick check on predictability of keys in the dictionary • EXPECTATION: Keys should at least appear pseudo-random at first glance • RESULT: • Resultfor www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo • Resultfor www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB • Resultfor www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx • Resultfor www.7b21b10c-9380-497c-b933-a51ad56b66e2.ny -> http://svc/bbzXBM • Resultfor www.e0c42320-16be-4b3d-a627-b5cc7a2eae05.ny -> http://svc/cG2EpA • Resultfor www.0df5718c-7738-45bc-a1ac-a8c67b0433f4.ny -> http://svc/aGp6P8 • Resultfor www.be2cb653-072b-44d6-a280-2f060a2e7e02.ny -> http://svc/dbIyo2 • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  17. Case study: Analysis of a major shortening service The findings… • TEST NAME: Determine if programmatic scraping is occurring • EXPECTATION: People are brute force scanning URL shortening providers for URLs • RESULT: • Resultfor www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo • Resultfor www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB • Resultfor www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx • Didanalytics checks on the results; 0 clicks • #success • However, if the scraping was forhttp://svc/123456+, the inverse lookupscrapingwouldn’tbedetectable in thismanner (otherthanby the provider) 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  18. Case study: Analysis of a major shortening service The findings… • TEST NAME: Account registration valid email and CAPTCHA • EXPECTATION: We estimated that registered users of the system have “more rights” and that CAPTCHA was a requirement upon registration (to make it harder to programmatically generate accounts) • RESULT: • Confirmedifyou register, you have additional privileges • #success • No registration information is validated; fake email addresspermitted • #fail • No CAPTCHA requiredtocreatean account • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  19. Case study: Analysis of a major shortening service The findings… • TEST NAME: Source URL replacement (i.e. the rickrolling test) • EXPECTATION: We estimated that registered users of the system may have some means to replace a source URL • RESULT: • Result for GOOD_URL: link.confusion.com/assumed_clean-> http://svc/fB3xbu • (Requested to “Edit” the link) • http://svc/fB3xbu now points to link.confusion.com/assumed_dirty • #fail • An attacker could get Pointer A to become popular, where Pointer A redirects to Site A; then the attacker edit Pointer A to redirect to Site B, while keeping the integrity of Pointer A, thus compromising user’s who viewed the page after the change 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  20. Case study: Analysis of a major shortening service The findings… • TEST NAME: Two providers friends that won’t play nicely together • EXPECTATION: We estimated that registered users of the system may have some means to replace a source URL • EXAMPLE TEST: • Shorten URL http://blah.invalid.url.com/blah on Provider A and get http://svc/dAf32X • Shorten URL http://svc/dAf32X on Provider B and get http://svc2/Adf392z • Edit http://svc/dAf32X on Provider A and redirect it to http://svc2/Adf392z • What happens? • Left as a reader exercise… 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  21. Case study: Analysis of a major shortening service The findings… • HACK IS WACK • “Upload your 2-min or less anti-cybercrime rap video” • Really clever and interesting marketing campaign  http://www.hackiswack.com/ • Went to bitly and did an inverse lookup (shortened an already shortened URL) • Was provided with: http://bit.ly/bOJg8r • Made an HTTP request for: http://bit.ly/bOJg8r+ • Was handed all the analytics on the campaign’s “success” 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  22. Future research areas Why dictionary size matters • Have established that creating a bunch of keys in the dictionary: • Can be programmatically done and doesn’t require valid URLs • There are a lot of way to hand the provider an invalid URL and get a key • Doesn’t require a valid email • Doesn’t require CAPTCHA • Seems a lot like the traditional buffer overflow problem • When the dictionary fills up, what happens? • Do old links get expired? If so, can they be tracked and then re-used? • Do links get overwritten? If so, can we first determine the most popular links? Can we predict which will get overwritten first? • How fast can a botnet fill up a keyed dictionary with between 2-56 billion keys? • Answer: TBD 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  23. Hardening the Net Making the interweb a safer place • As we learned yesterday in Igor’s keynote, we’re not going to find problems where we’re not looking • The same way we proactively scrape such providers as IRC, Usenet, and web sites for malware, we should be scraping the shortening providers • While web security scraping systems and web site reputation systems have had to deal with site redirection, the mass-use of shortening and redirection services presents new challenges to these systems 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  24. Why does application security matter? URL shortening security design’s applicability for malware researchers • New systems to scrape for malware • New user experience (at a consumer level), where URLs are intentionally obfuscated and redirected, leading to a higher importance level for web site warning systems (ex. SiteAdvisor) • Their increasing usage, presents new opportunities as well, to scan URLs prior to user’s even browsing to the page • New challenges as well, such as the dynamic nature of shortened URLs pointing to new locations 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  25. Conclusions As with any new and widely used technology, exercise caution • New technology brings new risks • Game theory can be used to present a nice backdrop to INFOSEC • As new systems become prevalent, which were built limited awareness of attacks, those systems will become attacked • The security of URL shortening and redirection providers will improve with time • But whether they improve their security, faster than their complexity increases, is TBD! • Programmatic scraping of shortening providers is likely in the best interest of anti-malware vendors, for the purpose of finding suspect behavior (the same way older systems, like IRC and Usenet are scraped) • In short, exercise caution 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

  26. Q&A Questions? • New technologies bring new challenges; if you need assistance, don’t hesitate to ask for help • For additional questions and follow up, I can be reached at: • Anthony_Bettini@McAfee.com 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

More Related