Building an Internet Router (P33)

1. Building an Internet Router (P33) Dr Andrew W. Moore andrew.moore@cl.cam.ac.uk Handout 1: What’s a router? Class project and logistics Thank you supporters: Building an Internet Router (P33)Handout 1

2. Some logistics Web page: http://www.cl.cam.ac.uk/teaching/0910/P33/ TAs: David Miller (Hardware) david.miller@cl.cam.ac.uk Stephen Kell (Software) stephen.kell@cl.cam.ac.uk Hot-spare TAs: Phil Watts (Hardware) philip.watts @cl.cam.ac.uk Chris Smowton (Software)chris.smowton@cl.cam.ac.uk Grades: Mphil (ACS) – Pass / Fail - based on a mark out of 100 All others (DTC) – Mark out of 100 Building an Internet Router (P33)Handout 1

3. Some more logistics Class Mailing list: bir-list@cl.cam.ac.uk think of this as a self-help mailing list for all of you Staff Mailing list: bir-staff@cl.cam.ac.uk Submission mail: bir-tick@cl.cam.ac.uk Group aliases can appear if you require them.

4. D R3 R1 R4 D A B E R2 C R5 Destination Next Hop F D R3 E R3 F R5 What is Routing?

5. R3 R1 R4 D A 1 4 16 32 D Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset B E TTL Protocol Header Checksum 20 bytes Source Address R2 C R5 Destination Address Destination Next Hop F D R3 Options (if any) E R3 Data F R5 What is Routing?

6. R3 R1 R4 D A B E R2 C R5 F What is Routing?

7. POP3 POP2 POP1 D POP4 A B E POP5 POP6 C POP7 POP8 F Points of Presence (POPs)

8. Where High Performance Routers are Used (10 Gb/s) R2 (10 Gb/s) R1 R6 R5 R4 R7 R3 R9 R10 R8 R11 R12 R14 R13 R16 R15 (10 Gb/s) (10 Gb/s)

9. What a Router Looks Like Cisco CSR-1 Juniper T320 19” 19” Capacity: 92Tb/sPower: 4.7kW Capacity: 320Gb/sPower: 2.9kW 6ft 3ft 2ft 2.5ft

10. Software Hardware Basic Architectural Componentsof an IP Router Management & CLI Routing Protocols Control Plane Routing Table Datapath per-packet processing Forwarding Table Switching

11. Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup packet destination address in the forwarding table, to identify outgoing port(s). 3. Manipulate packet header: e.g., decrement TTL, update header checksum. 4. Send packet to the outgoing port(s). 5. Buffer packet in the queue. 6. Transmit packet onto outgoing link.

12. Data Hdr Data Hdr IP Address Next Hop Address Table Buffer Memory Generic Router Architecture Header Processing Lookup IP Address Update Header Queue Packet ~1M prefixes Off-chip DRAM ~1M packets Off-chip DRAM

13. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table Data Data Hdr Hdr Data Hdr Generic Router Architecture Buffer Manager Buffer Memory Buffer Manager Buffer Memory Buffer Manager Buffer Memory

14. Software Hardware The P33 Project:Build a 4-port x 1GE Router Management & CLI Software Team Develop s/w in Linux on PC in lab Routing Protocols Exception Processing Routing Table Hardware Team Develop h/w in Verilog on NetFPGA board in lab Forwarding Table Switching

15. Management & CLI Routing Protocols Exception Processing Routing Table Emulated h/w in VNS 1 2 3 4 5 6 Build basic router Command Line Interface Routing Protocol (PWOSPF) Integrate with H/W Interoperability Wow us! Management & CLI Management & CLI • Innovate and add! • Presentations • Judges Routing Protocols Routing Protocols Exception Processing Exception Processing Routing Table Routing Table Emulated h/w in VNS Emulated h/w in VNS Management & CLI Routing Protocols Exception Processing software Routing Table 21/Oct 28/Nov 5/Nov 11/Nov 18/Nov 10/Jan hardware Forwarding Table Switching Learning Environment Modular design Testing Forwarding Table Switching 4-port non-learningswitch 4-port learningswitch IPv4 routerforwarding path Integrate with S/W Interoperability Wow us! 21/Oct 28/Nov 5/Nov 11/Nov 18/Nov 10/Jan

16. What is the NetFPGA? PC with NetFPGA 1GE FPGA 1GE 1GE Memory 1GE NetFPGA Board NetworkingSoftware running on a standard PC CPU Memory PCI A hardware accelerator built with Field Programmable Gate Arraydriving Gigabit network links

17. Running the Router KitUser-space development, 4x1GE line-rate forwarding PW-OSPF user kernel Routing Table “Mirror” 1GE FPGA Fwding Table Packet Buffer 1GE 1GE 1GE 1GE IPv4 Router 1GE Memory 1GE 1GE Management (including CLI) CPU Memory Exception Processing PCI

18. Next steps • Explore the web page (http://www.cl.cam.ac.uk/teaching/0910/P33/) • Decide if you still want to take the class • Build a team of 3 people: 2 s/w and 1 h/w • Come to SW02 on Tuesday 10am… We will meet with you every week during class time.

19. Using the nf-test machines • Each group is allocated an nf-test machine {nf-test1,nf-test2,nf-test3,...}.nf.cl.cam.ac.uk • These machines are headless (no KVM) and located in SW02 (alcove) • Login via ssh; set your own password and go We suggest running “vncserver” and then using “vncviewer”; as a useful way to leave jobs/desktop running and reconnect if you need to move location. • SW02 is a busy classroom – please respect others If you need to gain physical access to an nf-test machine Do it quietly! You maybe refused entry in the afternoons – don’t take it personally.

20. nf-test machine rules • Abuse the machines – that’s the end of this module for you – zero tolerance policy. • only connect eth0 to the cisco (access) switch • these machines run a firewall for a reason • if it doesn’t work for you, lets fix the rules • Play nicely with each other; your interoperability marks depend on it.

21. nf-test machine information • Warning: Files stored on nf-test machines are not backed up - and may be lost at any time. • You must make sure that you regularly copy your files into your REAL (computer laboratory) home directory as a back-up • Machines fail – they usually fail 24 hours before the big deadline!

22. Tree Structure NF2 bin (scripts for running simulations and setting up the environment) bitfiles (contains the bitfiles for all projects that have been synthesized) lib (stable common modules and common parts needed for simulation/synthesis/design) projects (user projects, including reference designs)

23. Tree Structure (2) lib C (common software and code for reference designs) java (contains software for the graphical user interface) Makefiles (makefiles for simulation and synthesis) Perl5 (common libraries to interact with reference designs and aid in simulation) python (common libraries to aid in regression tests) (scripts for common functions) scripts verilog (modules and files that can be reused for design)

24. Tree Structure (3) projects doc (project specific documentation) (contains file to include verilog modules from lib, and creates project specific register defines files) include (regression tests used to test generated bitfiles) regress (contains non-library verilog code used for synthesis and simulation) src sw (all software parts of the project) (contains user .xco files to generate cores and Makefile to implement the design) synth verif (simulation tests)

25. State Diagram to Verilog (1) MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter Output Port Lookup Output Queues MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ

26. Inter-module Communication Module i Module i+1 data ctrl wr rdy

27. State Diagram to Verilog (2) • Projects: • Each design represented by a project Format: NF2/projects/<proj_name> e.g., NF2/projects/bir_starter • Consists of: • What is Missing? • refer to the web page “Implement four port hard-wired switch” is the start. • Verilog source • Simulation tests • Hardware tests • Libraries • Optional software

28. State Diagram to Verilog (3) • Projects (cont): • Pull in modules from NF2/lib/verilog • Generic modules that are re-used in multiple projects • Specify shared modules in project’s include/lib_modules.txt • Local src modules override shared modules • bir_starter: • Local in_arb_regs.v input_arbiter.v output_port_lookup.v • Everything else: shared modules

29. Testing: Simulation (1) • Simulation allows testing without requiring lengthy synthesis process • NetFPGA provides Perl simulation infrastructure to: • Send/receive packets • Physical ports and CPU • Read/write registers • Verify results • Simulations run in ModelSim/VCS

30. Testing: Simulation (2) • Simulations located in project/verif • Multiple simulations per project • Test different features • Example: • bir_starter/verif/test_fwd_sourcePort0 • Test forwarding of packets from port 0 to port 1

31. Testing: Simulation (3) • Example functions: • nf_PCI_read32(delay, batch, addr, expect) • nf_PCI_write32(delay, batch, addr, value) • nf_packet_in(port, length, delay, batch, pkt) • nf_expected_packet(port, length, pkt) • nf_dma_data_in(length, delay, port, pkt) • nf_expected_dma_data(port, length, pkt) • make_IP_pkt(length, da, sa, ttl, dst_ip, src_ip)

32. Running Simulations • Use command nf2_run_test.pl • Optional parameters • --major <major_name> • --minor <minor_name> • --gui (starts the default viewing environment) test_fwd_sourcePort0 • Problems? check the environment variables reference your project • NF2_DESIGN_DIR=/root/NF2/projects/<project> • PERL5LIB=/root/NF2/projects/<project>/lib/Perl5: /root/NF2/lib/Perl5: major minor

33. Running Simulations • When running modelsim interactively: • Click "no" when simulator prompts to finish • Changes to code can be recompiled without quitting ModelSim: • bash# cd /tmp/$(whoami)/verif/<projname>; make model_sim • VSIM 5> restart -f; run -a • Problems? do ensure $NF2_DESIGN_DIR is correct

34. Synthesis • To synthesize your project • Run make in the synth directory (/root/NF2/projects/bir_starter/synth)

35. Regression Tests • Test hardware module • Perl Infrastructure provided includes: • Read/Write registers • Read/Write tables • Send Packets • Check Counters

36. Example Regression Tests • For a Router… • Send Packets from CPU • Longest Prefix Matching • Longest Prefix Matching Misses • Packets dropped when queues overflow • Receiving Packets with IP TTL <= 1 • Receiving Packets with IP options or non IPv4 • Packet Forwarding • Dropping packets with bad IP Checksum

37. Perl Libraries • Specify the Interfaces • eth1, eth2, nf2c0 … nf2c3 • Start packet capture on Interfaces • Create Packets • MAC header • IP header • PDU • Read/Write Registers • Read/Write Reference Router tables • Longest Prefix Match • ARP • Destination IP Filter

38. Regression Test Examples • Reference Router • Packet Forwarding • regress/test_packet_forwarding • Longest Prefix Match • regress/test_lpm • Send and Receive • regress/test_send_rec

39. Creating a Regression Test • Useful functions: • nftest_regwrite(interface, addr, value) • nftest_regread(interface, addr) • nftest_send(interface, frame) • nftest_expect(interface, frame) • $pkt = NF2::IP_pkt->new(len => $length,            DA => $DA, SA => $SA,            ttl => $TTL, dst_ip => $dst_ip,            src_ip => $src_ip);

40. Creating a Regression Test (2) • Your task: • Template files /root/NF2/projects/bir_starter/regress/test_fwd_sourcePort0/run.pl • Implement your Perl verif tests

41. Running Regression Test • Run the command nf2_regress_test.pl --project bir_starter