MSR Command Cell Facility

1. MSR Command Cell Facility Fred Kuhns Applied Research Laboratory

2. Highlights • Overview • High level description - Application Layer • MSR Command Interface Overview • Cell format and field definitions • Example

3. Definitions • Session: Open connection between the CP and a specific SPC. Intended to represent open connections and command state • Transaction: Represent a complete command. A transaction terminates with either an EOF is received by the CP or and error occurs. • EOF: End of File is returned to CP with the last bit of command data is returned or in response to a Cancel message (or an error occurs)

4. Overview - Cmd Interface on CP • Synchronous Request/Response protocol • Timeout can be specified as well as the number of retries - Per session option • Essentially provides a reliable service • Issue: if no reply, cmd/reply msg lost in port, channel or CP. Retries may be a bad thing. • Address - MSR Port and Command • <MSR_Port, MSR_Command> • Message destination - Callback function within the Port’s kernel (implements command)

5. Command Interface on CP • Types of messages: • New Command,Get Nextset of reply data Command,CancelCommand • Error Reply, EOF Reply, Continued Reply • Message Identifiers - Only requires a sequence number initialized to 0 for each New Command: • One sending entity on CP, • One outstanding command for each port, • Ports send exactly one reply msg per command msg, • Command must fit within one cell, • Replies may span multiple cells.

6. Command Interface on Port • Callback function registered with MSR kernel and called under 3 cases: • New Command • Flags = Command; Sequence = 0; Length = valid bytes in buffer; Buffer = application data • Next Command • Flags = Command | Next; Sequence = previous+1; Length = valid bytes in buffer; Buffer = application data • Cancel Command • Flags = Command | Cancel; Sequence = previous+1; Length = 0; Buffer contains no valid data

7. Command Interface on Port • Callback function must: • Read from/Write to supplied buffer • Set length = Bytes written to buffer (in/out param) • Indicate if an error occurred (return -1) • Whether more data exists (return 0 => EOF, return > 0 => Not EOF, return < 0 => ERROR | EOF) • Framework: • generates reply message using same Command value and Sequence number. • sets flags indicating status (EOF, Error etc)

8. Failure Modes • Library support for lost messages: • if (timeout > 0, Replies > 0), then CP API library will re-send with RETRY flag set. • if (timeout > 0, Replies = 0 or all replies failed), then API library returns error to application • If (timeout = 0 - No Timeout), then send operation blocks indefinitely. • Lost Command message - • if (timeout > 0 and retries > 0), CP resends command; same sequence number but RETRY flag set. Command buffer and flags passed to callback fn.

9. Failure Modes • Lost Reply message, • if no retries, Any issues? • if retries then CP resends • New Command - Port knows this is a duplicate command (RETRY flag). Application responsible for handling retries. If an issue can use unique message ids. Extreme case use a history (last reply message). • Next Command - Port receives Command w/Sequence > 0, w/RETRY flag. Passed to application which chooses the correct course of action. The intent is to ensure there are no holes in the reply data received by the CP. • Cancel message - same as Next command.

10. Possible Enhancements • Support asynchronous messaging: • Multiple outstanding commands per port • Asynchronous I/O on CP • Speed up boot process and dynamic configuration • Facilitates implementing port monitoring (ping or heartbeat) for fault detection and recovery. • two methods for reporting results: • upcall - function registered by application is called when results arrive • poll - application periodically polls library for results. • Support Broadcast and/or Multicast

11. MSR Command Layer • Simple messaging facility optimized for MSR. • Command message (CP sends): • Sent by CP to a specific MSR port (unicast) • Must fit within one AAL0 cell. • Message header, includes: • protocol version • Command • Sequence number • flags • Application data follows header • Library implements Request/Reply protocol.

12. MSR Command Layer • Reply Message (Port sends): • Port must send reply message in response to a Command message. • Reply message Header: • version and sequence number: same as command msg. • Includes application data and flags indicating if command was successful and if more data exists (EOF). • Application registers command specific callback function at port. • Callback function must conform to specified interface.

13. MSR Command Overvew • Command Protocol description • Control Processor sends command messages to a specific port and expects to receive a reply message indicating either Success or Failure. This is termed a Command Cycle. • There is the notion of a Command Transaction which may include one or more command cycles. A command transaction is terminated when the target (port) responds with a reply msg containing an EOF

14. MSR Command Overvew • Command Protocol description, continued • CP processing of Reply msg, depends on EOF flag: • If EOF is set then no further reply data is available and the command transaction is closed. • If EOF is not set then there is remaining data and the command transaction is still open. • If remaining data (Not EOF), then CP must follow with a either a Next or Cancel command message. • Sequence number indicates the “chunk” of data to be returned. • Command indicates the message’s destination • sequence number = previous + 1

15. 24 16 8 0 x x x x x x x x x x x x x pin x pout xx cc ld cid gfc vpi vci pti cl ver length command/status sequence number flags MSR Command Message APIC Cell Format • Cell payload contains the MSR Command • Command header is 4 Bytes, leaving 44 Bytes for sub-commands and data.

16. x x x x x x x x x x x x x pin x pout xx cc ld cid gfc vpi vci pti cl ATM/APIC Header • pin (Ports-In) - Port cell arrived • Tx not used (set to 000b) • Rx: port cell arrived on (is the below correct?) • 001 Port 0, 010 Port 1, 100 Port 2, etc. • pout (Ports-Out) - Set of output ports. • Tx: Command library sets: • 001 Fiber/Link, 010 Ribbon/Switch, 011 Both • 101 Loopback MV0, 110 Loopback MV1 • Rx: Set by VCXT, see pin above.

17. x x x x x x x x x x x x x pin x pout xx cc ld cid gfc vpi vci pti cl ATM/APIC Cell Format • cc (Control Cell Indicator) - Not used, set to 0b • ld (Low Delay) - Not used, set to 0b. • Should we use low delay? • cid (Connection Identifier) - set to vci value. • gfc (Generic Flow Control) - set to 0000b.

18. x x x x x x x x x x x x x pin x pout xx cc ld cid gfc vpi vci pti cl ATM/APIC Cell Format • vpi (Virtual Path Identifier) - Set to 0x0. • vci (Virtual Circuit Identifier) - Equal to cid. • See presentation on MSR configurations for a complete list of VCI assignments. • pti (Payload Type) - Set to 000b (data cell) • cl (Cell Loss Priority) - Set to 0b (High Priority)

19. MSR Command Header • Version (2 bits) - Protocol version. Allows for at most 4 versions. Current version set to 0. • field width was a trade off with the length field. • Length (6 bits) - Number of valid data bytes. • 0 <= Length <= 44, so 6 bits sufficient. • This field is indirectly set by the application or command implementation. The CP library and kernel interfaces allow for applications to pass a buffer pointer and indicate the number of valid data bytes. ver length command/status sequence number flags

20. MSR Command Header ver length command/status sequence number flags • Command/Status (8 Bits) CP inserts command value, SPC/port inserts status information. • Valid Commands are listed in $SYS/msr/msr_ctl.h, also see $MSR/utils/command/*.{c,h} • Library API on CP accepts Command as argument. implementation in kernel - array of function pointers, uses Command as index • Reply msg Status indicating success or an error code (Upcall, ATM, Cmd Invalid, Cmd Not Implemented, or Other Cmd Error).

21. Current Commands • Current set of commands: • set_debug/get_debug - set debug flags/masks. • port_init - set local port number. • policy - Manage MSR policy object. Currently sets the Policy control flags (Gate 0 enable) • dq - Preshanth? • fipl -Ed? • rp_pcu - extensive set of sub-commands. • rp_inst - send message to plugin instance • rp_class - send message to plugin base class

22. MSR DEBUG Command • Dynamic setting of debug mask - affects what messages are sent to syslogd • set_debug or get_debug command • Valid debug categories/modules • apic, ipfwd, iprx, iptx, mem, dq, stats, ctl, conf, kern, natm, pcu, plugin, gate • Valid debug levels - 0 - 255 • predefined: verbose, warning, error, critical • Interface in MSR kernel • MSR_DEBUG((MSR_DEBUG_<category>|MSR_DEBUG_LEVEL_<level>, “format”, args));

23. MSR RP_PCU Command • rp_pcu Command sub-commands • addfltr/remfltr - add/remove pkt filter at gate x • flist - port prints current gate x filter list • bind/unbind - bind/unbind instance to fltr/gate combo • create - create plugin instance • free - remove plugin instance • clist - port prints current plugin class list • ilist - port prints current instance list • load - load plugin - not implemented • null - no-op. Can be used as a ping operation

24. MSR Command Header ver length command/status sequence number flags • Sequence Number (8 Bits) - Is of primary use by the applications. • When command message first sent, sequence = 0. • If the reply does not include an EOF flag, then CP increments sequence by one for each subsequent command message. • When EOF is received the Command Transaction is complete and the sequence number is reset to 0.

25. MSR Command Header ver length command/status sequence number flags • Flags (8 bits) - Bit field, valid flags are: • Invalid - flag = 0, should not occur • CMD - cell contains a valid command from CP • REPLY - cell contains reply from Port • ERROR - Reply only, error processing on Port • EOF - No reply data remains, end of cmd transaction • NEXT - get next set of reply data • CANCEL - cancel current cmd transaction • RETRY - set if cp resend a command after it was lost

26. CP Library API • Library API for application on CP, • int sendcmd(int sid, int cmd, char *data, int flags, int *dlen) • sid = session id, • cmd - Command to execute on port • data = buffer pointer, • flags = • RETRY (reply timeout), • CANCEL (cancel current command), • Next (get next set of reply data)

27. MSR Kernel API • MSR kernel interface $SYS/msr/msr_ctl.{h,c} • Callback function signature: • msr_ctl_<cmd> (void *buf, u_int8_t flags, u_int8_t seq, u_int8_t *dlen) • buf = command buffer w/application data, • flags = • CMD, • NEXT, • RETRY or • CANCEL, • seq = sequence number indicating reply data set, and • dlen is input/output parameter, data length in bytes.

28. Kernel State Diagram Next Retry Command Command Command Cancel EOF Proto Error Closed Idle Proto Error Retry

29. CP Library State Diagram Retry Next Result of a timeout Wait (for reply) Protocol Error Command EOF Open Session Closed Idle Protocol Error

30. cell hdr cmd data Example Sending Cmd to Port Lookup sub-command perform function call then report results wugs 192.168.200.X 192.168.204.X P4 P0 SPC/FPX SPC/FPX Next/Prev Hop Next/Prev Hop msr_ctl DQ DQ 192.168.205.X 192.168.201.X P5 P1 SPC/FPX SPC/FPX reply(); plugin instance created: Status, Instance ID Next/Prev Hop Next/Prev Hop DQ DQ sendcmd(); create plugin instance: port id = 0, PluginID = 200 192.168.202.X 192.168.206.X P6 P2 Report command completion status to application. SPC/FPX SPC/FPX Next/Prev Hop Next/Prev Hop DQ DQ 192.168.202.2 192.168.203.X 192.168.207.X P7 P3 SPC/FPX SPC/FPX Next/Prev Hop CP DQ DQ 192.168.203.2