Cplant Myrinet Switch Monitoring

Myrinet switches provide the high speed cluster message passing network for passing messages between compute nodes and for I/O. The Myrinet switches have a few counters that can be accessed from an ethernet connection to the switch. These counters can be accessed to monitor the health of the connections, cables, etc. The following information refers to the 16-port, the clos-64 switches, and the Myrinet2000 switches.

Myrinet Switch Counters

The Myrinet switches maintain counters for the following types of received packets between reboots or clears:

• good_high,good_low - A 64-bit counter of the number of good packets received in on this port.
• bad_high,bad_low - A 64-bit counter of the number of non-zero CRC8 packets received on this port.
• timeouts - The number of times that the switch didn't see a tail for too long coming in on a port. Timeouts are on the receive side, so they indicate that a packet went too long without a tail, or that the packet was blocked for too long.
• bad_routes - The number of packets that entered this port (receive), but had a routing byte that was off either end (out-ofrange) of the switch.
• dead_routes - The number of packets that entered this port (receive), but had a routing byte destined for a valid port that was down, disconnected, or off.
• bad_packets- Received packets with non-zero CRC bytes at the tail.
• missed_beats - A port is not connected. Connected ports receive heart beat signals from their opposites, and when these stop arriving, the counter goes up. If you have HA features enabled, the port will also shut itself down (portDownOnMissedBeat). A missed beat is no beat in 10 usecs, but this is polled for by the xbar microcontrollers around every 10 msecs and updated in the monitoring card probably about every 100 msecs.
• port_state - A mask of the following bits:
  • #define SW_PORT_ON 0x01
  • #define SW_PORT_BEAT 0x02
  • #define SW_PORT_TRANSMIT 0x04
  • #define SW_PORT_RECEIVE 0x08
• control is in 2 parts.
  • The 0xff bits are an internal state machine state of the following values:
    • case 0: /* port was in reset. initialize control register */
    • case 1: /* port is initialized */
    • case 2: /* port is off, but automatic shutdown is enabled */
    • case 3: /* port is on, automatic shutdown is on */
    • case 10: /*port was disabled by user. */
  • The 0xff00 bits are a bit mask of the following bits:
    • #define HW_CTRL_XMIT_OFF 0x80
    • #define HW_CTRL_RECV_OFF 0x40
    • #define HW_CTRL_MISS_BEAT_OFF 0x20
    • #define HW_CTRL_ILL_OFF 0x10

System Administration

The normal location for Myrinet adminstration Cplant scripts and programs is /usr/local/system/myrinet. The following files are found in this directory. Cron is normally setup to call the sw-report script to periodically dump the switch counters for offline analysis. This may be hourly or daily depending on the reliablility of the cluster's network.

• bob_test16 - Version of the Myrinet switch query program for the 16-port switches that dump the above counters or other switch values.
• bob_test64 - Version of the Myrinet switch query program for the 64-port clos switches that dump the above counters or other switch values.
• bob_test - A link to the appropriate bob_test program.
• create - C program that parses files of output from the bob_test programs and generates a more concise, compact file. These files have a *.tot extension by default.
• ip_file - Contains the IP, MAC, and name of the switches from the /etc/dhcpd.conf file. This file is used by many scripts to get the switches to access for that cluster.
• read_db - Reads the /etc/dhcpd.conf and pulls the "myri" device IP, MAC, and names. Normally, this is output to the file, ip_file.
• report - C program that compares *.tot files and produces a tabular output.
• set_macs - Script that sets the IP addresses in the switches using the rarp command. Uses the ip_file to know what MACs and IPs are called.
• sw-dump - Script that calls the appropriate bob_test program and dumps the switch responses to a *.dat file. This dat file is input to create. Uses the ip_file to know what switches to call. This script supports Myrinet 2000 (m3), Clos-64 (m2), or 16-port switches. See M3 switch notes below.
• sw-get-timeout - A script that returns all the timeout values of the switches, a settable parameter. Uses the ip_file to know what IPs and MACs are valid.
• sw-ping-all - A script that pings all the switches to see if they are connected and alive. The -r switch emails a notice if there is an error pinging a switch. Uses the ip_file to know what IPs and MACs are valid.
• sw-reboot-all - A script that issues a reboot command to all switches.
• sw-report - A script that generates a report of the switches on that cluster. This script is normally called by a periodic cron job. sw-report calls sw-dump, sw-set-timeouts, set-macs, read-db, and any other periodic functions.
• sw-set-timeouts - A script that sets the timeout values of the switches, a settable parameter. Uses the ip_file to know what IPs and MACs are valid.

Naming Conventions

The /etc/dhcpd.conf file is built from the Cluster tools database. Some naming conventions exist that are used in the above scripts. The "mf" devices are m3 fiber switches. The "my" devices are m3 switches too. Other devices are clos-64 or 16 ports switches.

System Setup

The Myrinet switches can be queried from their ethernet connections. This is normally connected to the SSS1 admin node but some systems have the switches connected to the mothers that in turn are connected to the SSS1 node. Connections to a SSS1 admin node are preferable for speed of query and uniformity of configurations.

Clos-64 or 16 port switch requirements

• The Linux kernel must support the RARP protocol. Kernel configuration defaults to "no" so a new kernel may have to be configured. This can be a a module or as part of the kernel but as a module has to be loaded. It's not very big.
  • RARP support can be confirmed by looking for a /proc/net/rarp. If the file exists, then RARP is supported. If not, a module may have to loaded or it is not supported.
  • RARP support might be configured as a module.
    • The command > lsmod can be used to list modules such as rarp.o that have been installed.
    • If there is a file in /lib/modules/[release number]/rarp.o then that release is configured as a module.
    • RARP support can be added with the command:
      > insmod /lib/modules/[release number]/rarp.o.
• IP addresses must be assigned for each switch. This implies that at the switches must be connected to an ethernet hub. Each switch has its hardware MAC address printed on the switch. The IP address must be broadcast from a Linux host using the rarp command. This command informs the Myrinet switch its IP address.
• Setting the IP address in the switch: /sbin/rarp -s [IP address] [MAC Address]
• The "ping" command can be used to verify that the Myrinet switch has assigned its IP address.
• The Myrinet switch will store that IP address until it is powered off or rebooted with the bob_test -reboot command. If it is powered off or rebooted, the rarp command will have to be repeated. The switch will not recognize a changed IP address with a second rarp command. The switch will have to be powered off or rebooted to change the previous address.
• Download the corresponding 16-port or 64-port version of the Myrinet Switch Query Program, bob_test, from Myrinet at http://www.myri.com:80/scs/switch/. This will require a username/password from Myrinet. This can be obtained at help@myri.com.
• There are several precompiled versions of the 64-port version. There is no Alpha version. The Makefile requires some simple modifications to make it work. It does build correctly on the Alpha platform using gcc.
  
  

  Myrinet2000, or M3, Switches

Myrinet 2000 Switches are considerably different from the Clos-64(m2) and 16-port switches. First, these switches get their IP addresses from the dhcpd demon, not from the RARP command. So, the switches' IP and MAC addresses must be in the dhcpd's configuration file, /etc/dhcpd.conf, which is generated from the Cluster Tools on Cplant.. Dhcpd must be restarted when the file is modified, it does not respond to a SIGHUP or notice that the file is changed. Once the m3 switch has its address it is accessed by HTML browsing, SNMP commands, or one of several small programs supplied by Myrinet. Note that these programs do not all run correctly so use with caution.

Software

Myrinet 2000 Notes

Myrinet 2000 Switches are considerably different from the Clos-64(m2) and 16-port switches. First, these switches get their IP addresses from the dhcpd demon, not from the RARP command. So, the switches' IP and MAC addresses must be in the dhcpd's configuration file. Once the m3 switch has its address it is accessed by HTML browsing:

> lynx IPADDR where IPADDR is the switches name or IP address. Then navigate through the pages to desired data.

> lynx IPADDR/bad or
> lynx IPADDR/good lists the pages without having to navigate.

> lynx -dump IPADDR/bad dumps the page so it can be redirected to a file.

In the lynx interface each switch's ports are numbered 0..255.

The M3 switches can have values such as the timeout value set by the lynx interface. This is pretty painful, so the values can be set or read using snmp commands such as the below.

> snmpset IPADDR public .1.3.6.1.4.1.1771.3.1.10.1.4.oid i microseconds

Where IPADDR is the IP address, oid is the switch (1..16), and microseconds is the number of microseconds.
"public" is not changed and "i" is integer type which is not changed.
The absolute value for the SNMP is used since the snmp tools provided don't compile the MIB values correctly.

Building Clos64 bob_test

The 64-port bob_test is located at ~/[release]/config/user_sixteen. In order to build the clos64 version on the Alpha, the following modifications are required.

• mkdir alpha-linux
• In the Makefile replace:
  arch := $(shell arch.sh)
  with
  arch=alpha-linux
  and
  HOST_CC = [whatever]
  to
  HOST_CC=gcc or [whatever]
• This "hardcodes" the alpha_linux version.
  The bob_test version is then built in a subdirectory at ~/[release]/config/user_sixteen/alpha_linux/bob_test using gcc.

Building Myrinet2000 - M3 Version

• Download and untar the m3dist version.
• Modify the Makefile as above. mkdir alpha-linux.
• Follow the README: cd snmp++/src and make snmp++.
• Go back to the m3dist directory. Don't run arch.sh but make host snmp++dir=snmp++. The GM tools will error off as expected. The test_port, etc. tools are in the alpha-linux directory.

bob_test Use

It is useful to routinely dump the contents of all the switches to a file for later analysis. A README file comes with the Myrinet diagnosis software that vaguely describes the bob_test program. Bob_test communicates with each switch over a socket at a fixed address, 4002. Each switch has sixteen or eight ports, depending on the hardware. Depending on each cluster these may be compute or service nodes or network interconnects. Also, not all ports may be populated depending on the cluster architecture.

Sample 16-port Switch sw-dump:

# sw-dump [su] [sw-addr]
# where [sw-addr] may be an IP address not in hosts file, for instance.
bob_test=/usr/local/system/myrinet/firmware/bob_test
host=`hostname | cut -d. -f2,3`
su=$1
echo "Switch m-2.SU-$su.$host -----------------------------------------"
$bob_test m-0 4002 -get-version-string
$bob_test m-0 4002 -get-uptime
$bob_test m-0 4002 -get-timeout
for p in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    echo "Switch m-2.SU-$su.$host Port $p"
    $bob_test $2 4002 -get-port-state $p
done

The 64-port Clos64 switches have 0..15 ports per 0..16 xbars, or crossbars. Again, these may not all be populated depending on the cluster.

Sample clos64 Switch sw-dump:

The following is a sample script that dumps the data from Clos64 style switches:

#!/bin/bash
path=/cplant/myrinet
# Nested loops do all switches, all xbars, all ports.
# Currently, the limits are hard-coded until other config files
# setup the topology
m=0
while [ $m -lt 29 ]
do
    xbar=0
    echo "Switch m-$m.SU-0.SM-0.zermatt ------------------------------------"
    $path/bob_test m-$m 4002 -get-version-string
    $path/bob_test m-$m 4002 -get-uptime
    $path/bob_test m-$m 4002 -get-timeout $xbar

    while [ $xbar -lt 16 ]
    do
        for p in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
        do
            echo "Switch m-$m.SU-0.SM-0.zermatt Xbar $xbar Port $p"
            $path/bob_test m-$m 4002 -get-port-state $xbar $p
        done
    xbar=`expr $xbar + 1`
    done
m=`expr $m + 1`
done

 

Analysis

The first step is to periodically run a script such as above to collect a snapshot of the switches' counters in a file. The script may be run as a cron job to periodically save the counter's values.

The create program can be used to parse the data and save it in a file as a serialized form of the bob_struct.

The report program then can be used to compare snapshots for differences in any of the counters reported by the switch. These differences may be then compared by day, hour, or whatever time-period is used.

It has been noted that powering off a compute node attached to a switch sometimes results in thousands of bad packets being counted. This is probably some kind of noise spike. Interestingly, this does not appear during power-on.

Source Files

bob.h - Defines the bob_strucg used in the applications.
bob.c - Implements functions that load or dump the bob_struct, parses files, or compare bob_structs.
Makefile - Builds the programs.
create.c - Parses output from the switches, builds bob_structs, and saves the serialized form in a file.
report.c - A report writer that compares the counts of a switch count type for any number of files.