Some thoughts on deploying measurement infrastructures

1. Some thoughts on deploying measurement infrastructures Dan Magorian Director of Engineering and Operations Mid-Atlantic Crossroads Albuquerque, NM 2/7/2006

2. MAX recently hosted a Perf workshop • Went well, I was happy about participation • Until I counted institutions represented instead of people. Only 4 out of 40, inc MAX folks. • Started thinking: why do some people and institutions “get” the need for deploying a measurement infrastructure, and others not? • Not really cost or effort: just a couple pcs and a day • Realized that people who came and got around to actually doing it, were probably people who had been burned by lack of it at least once. • Everyone else, on their list for things they should be doing, but never quite rises to the top. Albuquerque, NM 2/7/2006

3. We talked about this at Gigapop Geeks BOF • Folks had some interesting comments: • “We know places that get burned over and over again, yet don’t get around to deploying them”. Given how easy it is to do, this is depressing. • “The problem is that they’re weird kernals that are different from our usual service boxes. If they looked like our standard campus distro, we’d have done a lot more of them.” Effort making sure they’re secured, patched, etc. • “The retention rate of configured boxes after workshops that remain deployed is only around 25%” After people put the effort into configuring them, why? Albuquerque, NM 2/7/2006

4. So to try and help this situation • At MAX, we’re completing a pair of portable mini Spacewalker-style pcs that have onboard giges and 10G Myricom pci-express nics that take XFPs, with iperf tuned and ready. • We needed them ourselves for testing 10G lambdas that we bring up for customers. • But as a side benefit, they will be loaners for MAX participants experiencing performance problems, to borrow for few days for “divide-and-conquer” testing. • We’re also considering MAX-supplied pcs that go on each participant edge router. Like old AMP pcs, easier if people just have to plug them in. But don’t get “buy in” Albuquerque, NM 2/7/2006

5. Next topic: Best places to put them. • For Abilene, ESnet, national backbones: One or more at all router nodes. • Gigapops: At least one on routers facing national backbones, better one on each router node. • But think about non-congruent L1/L2/L3 infrastructures these days. Fewer with router consolidation? MAX used to have mostly a router at each node, but these days uses optical backhaul from 6 pops to “big fat routers” (Juniper T640s) in middle. Putting a L3 iperfer on a remote L2 switch may not accomplish much more than testing one of many lambdas. Points out lack of tools at L1/L2. Albuquerque, NM 2/7/2006

6. Best places to put boxes on campuses • The minimum is easy: one iperfer on each edge router facing national backbones. Other test boxes can be centralized • If everyone even did that much, would be really good • More starts to get complicated. How do you pick sample “strategic locations”? At compute clusters and critical resources are obvious choices. Behind firewalls/nat boxes? “Worst-case” subnets? One per campus switch? Might be 100s. How many is affordable or reasonable? Diminishing returns? Albuquerque, NM 2/7/2006

7. Once boxes deployed, getting maximum benefit • If just lie around doing testing on demand, may not establish “everything working” baseline needed for problem resolution. • MAX had iperfers since 2000, but only now starting to schedule regular testing with bwctl. • Coordinating gigapop-campus, inter-gigapop, and gigapop-Abilene testing takes more people effort. Benefits clear, but not many get around to doing it. • Looking forward to new perfSonar framework still under development and benefits that will bring with mode autodiscovery mechanisms. Albuquerque, NM 2/7/2006