inter peer noc communication n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Inter-Peer NOC Communication PowerPoint Presentation
Download Presentation
Inter-Peer NOC Communication

Loading in 2 Seconds...

play fullscreen
1 / 26

Inter-Peer NOC Communication - PowerPoint PPT Presentation


  • 176 Views
  • Uploaded on

Inter-Peer NOC Communication. Mike Hughes mike@linx.net. Scene Setting: Straw Poll. Who here in this room does peering?. Scene Setting: Straw Poll. Who here in this room does peering? Have you ever had issues resolving problems with your peerings?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Inter-Peer NOC Communication' - lazaro


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
inter peer noc communication

Inter-Peer NOC Communication

Mike Hughes

mike@linx.net

scene setting straw poll
Scene Setting: Straw Poll
  • Who here in this room does peering?
scene setting straw poll1
Scene Setting: Straw Poll
  • Who here in this room does peering?
  • Have you ever had issues resolving problems with your peerings?
    • Difficulties contacting peers, finding the right contact, communication problems?
scene setting straw poll2
Scene Setting: Straw Poll
  • Who here in this room does peering?
  • Have you ever had issues resolving problems with your peerings?
  • Do you maintain a local db of contacts?
    • Why? Issues with freshness of data?
scene setting straw poll3
Scene Setting: Straw Poll
  • Who here in this room does peering?
  • Have you ever had issues resolving problems with your peerings?
  • Do you maintain a local db of contacts?
  • When a peer needs to talk to you, where does their call/email arrive?
    • Main NOC contact? Dedicated peering contact? “Customer Care”?
scene setting straw poll4
Scene Setting: Straw Poll
  • Who here in this room does peering?
  • Have you ever had issues resolving problems with your peerings?
  • Do you maintain a local db of contacts?
  • When a peer needs to talk to you, where does their call/email arrive?
  • Some names have been changed to protect the innocent… and guilty…
why do you go peering
Why do you go peering?
  • Long term money savings
  • Less Transit
  • Lower latency, better performance
  • Traffic Control
  • Diversity, Reliability
  • Presence

…and so on…

where s the problem
Where’s the problem?
  • Poor inter-peer communication seems to be common
    • Friendly IX operator called in to “mediate”
  • Communication hitting the wrong place
    • Customer NOCs
    • IX Operator
    • IP address maintainer (e.g. whois contact)
identifying the right contact
Identifying the right contact
  • Sources of information:
    • Whois queries to databases
    • IXP-maintained NOC and Peering contact db
    • Internal databases
    • Third-party voluntary databases
      • http://puck.nether.net/netops list
      • peeringdb.com
  • All above are vulnerable to information “rot”
how to drive ripedb ra etc
How to drive RIPEdb/RA, etc
  • Some really subtle differences in the implementations
    • RIPE expects “AS” before an AS number!
  • Which contacts are useful
  • Which objects to look up
    • Like the Peer ASN, not the Peer IP address!
  • Why can’t ASN be logged in adjacency changes on routers?
    • This seems to drive IP-based lookups
drive the data sources properly
Drive the Data Sources Properly!
  • Example: using WHOIS queries
  • “Oh, I have an outage on WAIX, I’ll look up the IP address”

$ whois -h whois.arin.net 198.32.212.11|less

OrgName: Exchange Point Blocks

RTechHandle: WM110-ARIN

RTechName: Manning, Bill

RTechPhone: +1-310-322-8102

RTechEmail: bmanning@karoshi.com

bad data enters the system
Bad Data Enters the System
  • “Okay, I’ll phone Bill Manning”
    • But all Bill did was give WAIX some v4 space
    • Bill doesn’t run WAIX, and isn’t an operational contact for WAIX
  • So, Bill either ignores your voicemail, or tells you to call someone else
  • Whatever – it’s added delay, increased frustration – it’s how not to do it
driving whois properly
Driving Whois Properly
  • Always lookup the PEER ASN
    • Not the IP address!
    • It’s a BGP problem, we use ASNs in BGP

$ whois -h whois.ra.net AS3856|less

aut-num: AS3856

as-name: UNSPECIFIED

descr: Packet Clearing House

www.pch.net

admin-c: Bill Woodcock

tech-c: Bill Woodcock

remarks: peering@pch.net, +1 866 BGP PEER

driving whois properly1

$ whois -h whois.ra.net AS3856|less

aut-num: AS3856

as-name: UNSPECIFIED

descr: Packet Clearing House

www.pch.net

admin-c: Bill Woodcock

tech-c: Bill Woodcock

remarks: peering@pch.net, +1 866 BGP PEER

Driving Whois Properly
  • Always lookup the PEER ASN
    • Not the IP address!
    • It’s a BGP problem, we use ASNs in BGP
so you ve found the contact
So you’ve found the contact
  • How do they respond to you?
    • Confusing recursive call trees?
    • Recalcitrant ticketing systems?
    • First-line NOC – “Is it switched on?”
    • “You’re not a customer, go away”
  • Once negotiated, peering is an engineering relationship
    • So backbone ops, not “customer care”
expectations of peer contacts
Expectations of Peer Contacts
  • Choose your points of contact carefully
  • Big problems with
    • What’s peering/BGP/WAIX?
    • Are you a customer?
    • What’s your circuit ID?
    • Go away, you aren’t a customer
  • All serious no-no’s – be nice to your peers!
pch inoc dba phones
PCH INOC-DBA Phones
  • PCH operate a “dial by ASN” NOC hotline system
    • They run the SIP registry/proxy
    • “Bring your own” SIP compliant phone
  • The idea is that it should get through to someone clueful
    • No call-trees, no music-on-hold
  • http://www.pch.net/inoc-dba/
suggested role contacts
Suggested Role Contacts
  • Peering@
    • For setting up new peerings, changing existing ones, no 24x7 expectation
    • Shouldn’t go to exclusively to sales@ ;-)
  • NOC@
    • Reaches your 24x7 NOC, which is either BGP friendly and has enable, or knows when, how and where to escalate
  • Support@
    • Is generally your “customer-care”/call center
getting the message across
Getting the message across
  • Okay, so you’ve made contact
    • Now, make your point
  • Provide the peer with useful information
    • Start with the subject line
    • Be informative, who, when, what
    • Messages like “Help” and “Peering down” aren’t helpful
how not to do it
How not to do it…

-----Original Message-----

From: Joe Schmoe <schmoe@noc.foo.com>

Sent: Wednesday, January 25, 2006 5:41 PM

Subject: Maintenance Notification

Dear Peers,

  • Where? How does it affect me?
    • All detail buried in wordy message body
  • When? No TZ stamp!
  • Help me handle my huge NOC inbox!
example useful subject headers
Example: Useful Subject Headers

AS7132’s preferred subject line format:

<IX location> -

<peer writing to/ASN> -

<peer writing from/ASN> -

<what is the issue> -

<date of initial correspondence> -

<time of initial message>

Example subject line:  

Equinix-Ashburn - RCN/6079 - SBC/7132 - new session turn-up - 29- Mar-06 - 9:45 am EST

Thanks to Ren Provo

look clueful
Look clueful

Subject: Traffic Drop

Dear Peer,

We suddenly noticed a 300Mb drop in traffic on our connection to the PIE-IX. Can you investigate, and help us find where the traffic has gone?

Regards,

  • What does this say about your peer?
    • Don’t you think they look silly?
  • Run tools to help you answer these questions yourself
    • Netflow, MAC accounting, etc.
how to escalate
How to escalate
  • Check your equipment first
  • Ask your peer - “What’s up?”
    • Often you can resolve a problem bi-laterally
  • Go to the IX only if you need to
    • Not all IX operators can provide a 24x7 contact
  • When to escalate a customer fault
    • Don’t stonewall customer reports
    • Don’t point them to the IX operator
    • Co-ordinate directly with your peers
how the ixp op can help
How the IXP Op can help
  • Provide an up-to-date list of IX participants and their NOC/Peering contact information
    • Usually password protected
  • Help break comms deadlock
    • Help fix “dead ends”
  • Otherwise, they can only help with “physical” problems
    • “link down”, packet loss, broken cables, packet corruption to all destinations connected to the IXP
in summary
In Summary
  • Keep your own information up to date
    • Whois db objects, third party dbs
  • Make sure your peering and NOC contacts are appropriate
    • No-one likes call-trees and holding
  • Find the right contacts at your peers
  • Be nice to your peers!
thanks
Thanks
  • mike@linx.net