Assigning IP Addresses to FFSC's

Rob Bradshaw
Version 1.4

Problem

The FFSC's in a multi-rack lego system are connected to each other by a private ethernet. VxWorks, the real time OS that runs on the FFSC, has support for networking, but mainly at the IP level. This implies that unique IP addresses need to be assigned to each FFSC before the FFSC's can talk to each other. While an SE or system administrator could in theory plug a console into each FFSC and assign an IP address manually, this is cumbersome and prone to errors. Furthermore, because this private ethernet isn't really an externally visible feature of the machine, it is likely to cause a fair amount of confusion and resistance. Therefore, the FFSC's in a lego system need to negotiate and assign addresses among themselves, without intervention from users.

Assumptions

Proposal

During initialization, each FFSC will start an "address daemon" task that sends and receives several types of messages:

IDENTITY
Sent by an individual FFSC to describe the various identifiers associated with it. These identifiers include:

QUERY
This message is identical to an IDENTITY message, except that it also asks the receiver to identify itself, typically by broadcasting an IDENTITY message.

ASSIGN
Assigns a new set of identifiers to an FFSC. This is generally done by the "master FFSC" (see below) when it has detected two or more FFSC's that are using the same identifier. In this case, it will arbitrarily reassign addresses to avoid conflicts.

REQUEST_RACKID
This message is used when an FFSC is requesting a specific rack ID, presumably to satisfy a user request. It is treated the same as an IDENTITY message except that the master FFSC will always resolve rack ID conflicts in favor of the sender.

All of these messages are sent as raw ethernet packets with a fixed, non-standard protocol type, in order to distinguish them from normal IP packets. VxWorks has a mechanism for filtering incoming raw ethernet packets. When an ethernet packet with the special address negotiation protocol is detected, it will be sent to the address daemon rather than the normal networking code via a named pipe. (Note that VxWorks does indeed allow writing to named pipes from interrupt services routines.)


The general operation of the address daemon is as follows:

  1. Set up initial addresses for the FFSC. If the FFSC has NVRAM available to it, then the initial addresses could be obtained from there. Otherwise, generate addresses as follows:

    Unique ID
    Always taken from the MAC address of the ethernet device. This address is never subject to change.

    IP Address
    Use some fixed class-A network address for the network portion, and some 24-bit value hashed from the unique ID for the host portion.

    Rack ID
    "Unassigned"

    Note that as long as no new modules are added to the system, each module should end up with the same addresses upon power-on that it had after successfully completing address negotiation the first time it was added to the system. This should be helpful during debugging. However, addresses are subject to change at the address daemon's whim and so should not be relied upon in general.

  2. Send out a "QUERY" message containing the current proposed addresses. Any other FFSC's should then respond with their own IDENTITY messages.
  3. Process any incoming "FFSC Address Daemon" messages:

    IDENTITY
    Save the information in the message for future reference. Note that the message may supersede a previous IDENTITY message for the same unique ID. If an IDENTITY message is received by an address daemon that contains its own unique ID, then something is terribly wrong; print appropriate error messages. The "master FFSC" (see below) may do additional work.

    QUERY
    Same processing as for IDENTITY messages, then send out an IDENTITY message for the local FFSC. Set a timer for some fixed period of time; when the timer goes off, examine the list of known FFSC's and do the following:

    • If an FFSC has not responded with an IDENTITY message since the last QUERY message was received, remove it from the list.
    • The FFSC with the lowest unique ID in the list is designated the "master FFSC".

    ASSIGN
    If the message specifies a different unique ID than the one belonging to the local FFSC, the message is ignored. Otherwise, reset the identifiers associated with the local FFSC to the values specified in the message. If NVRAM is available, the new identifiers should be stored there as well. Once everything has been changed, send out an IDENTITY message with the updated identifiers.

The Master FFSC

One FFSC in a configuration is designated the "master FFSC". In addition to the usual responsibilities, the master FFSC is responsible for resolving conflicting identifiers. If it sees two IDENTITY messages that have different unique IDs but match on some other identifier (e.g. IP address, rack ID) it arbitrarily selects one of the two unique IDs and updates the conflicting identifier(s) as follows:

IP address
Select some other random IP address that is not currently in use by another FFSC.

Rack ID
Print an appropriate error message, then reset the rack ID to "UNASSIGNED". (The general issue of dealing with FFSC's that have unassigned rack ID's is TBD.)

It then sends out an ASSIGN message with the updated information.

The selection of a master FFSC is done shortly after a QUERY message has been sent by any address daemon. The following steps are performed on receipt of a QUERY message:

It is assumed that all FFSC's will successfully receive all QUERY and IDENTITY messages that are broadcast, so each FFSC will reach the same conclusion as to the identity of the master FFSC. It may be necessary to revisit this problem if that turns out to be a bad assumption.


It is possible or even likely that an FFSC may not be powered up or initialized when other FFSC's broadcast their initial QUERY or IDENTITY messages, and so may miss some of them. Any FFSC can cause all of the other FFSC's to rebroadcast IDENTITY messages by simply broadcasting a QUERY message. This should probably be done routinely by the master FFSC once all of the FFSC's are likely to be powered up. Since the FFSC's are powered separately from the rest of the system, and presumably must all be powered on before the rest of the system can be powered on, a user "system power up" command might be a good time to do this. It probably wouldn't hurt for the master FFSC to send out a QUERY message every minute or so even during normal operation in order to drop any FFSC's that have been powered off or otherwise removed from service.

A user "query addresses" command might be useful as well. It should cause the broadcast of a QUERY message, then wait until no IDENTITY messages have been received for some fixed period of time. At that point, it should be safe for it to dump out the contents of the local identity table.


up a level

Send questions and comments to: Rob Bradshaw

Last modified: Sat Sep 7 14:40:47 PDT