Assigning IP Addresses to FFSC's
Rob Bradshaw
Version 1.4
Problem
The FFSC's in a multi-rack lego system are connected to each
other by a private ethernet.
VxWorks, the real time OS that runs on the FFSC, has support
for networking, but mainly at the IP level.
This implies that unique IP addresses need to be assigned to
each FFSC before the FFSC's can talk to each other.
While an SE or system administrator could in theory plug a
console into each FFSC and assign an IP address manually, this
is cumbersome and prone to errors.
Furthermore, because this private ethernet isn't really an
externally visible feature of the machine, it is likely to
cause a fair amount of confusion and resistance.
Therefore, the FFSC's in a lego system need to negotiate and
assign addresses among themselves, without intervention from
users.
Assumptions
-
All FFSC's in a complex are connected to each other on a
private ethernet.
This implies that an arbitrary network address can be used.
-
The FFSC has easy access to some sort of world-unique
identifier, such as a MAC address, serial number, etc.
-
It is possible to send and receive raw ethernet packets
across an FFSC network interface before an IP address has
been assigned to it.
-
Messages sent across the ethernet are received
"simultaneously" by all FFSC's, within some
reasonable tolerance.
This implies no WAN repeaters, SLIP connections, or other
slow links in the private ethernet.
-
The FFSC has no way of determining if a display is attached
to it, so there is no trivial mechanism for
identifying a "master" FFSC.
It is therefore necessary to elect a master FFSC.
Proposal
During initialization, each FFSC will start an "address
daemon" task that sends and receives several types of
messages:
- IDENTITY
-
Sent by an individual FFSC to describe the various
identifiers associated with it.
These identifiers include:
-
A world-unique ID (serial number/MAC address/etc.)
-
A rack ID (a small, user-friendly number to identify an
individual rack in the system - it's origin is TBD)
-
IP address (initially generated as a function of the
unique ID)
- QUERY
-
This message is identical to an IDENTITY message, except
that it also asks the receiver to identify itself, typically
by broadcasting an IDENTITY message.
- ASSIGN
-
Assigns a new set of identifiers to an FFSC.
This is generally done by the "master FFSC" (see
below) when it has detected two or more FFSC's that are
using the same identifier. In this case, it will
arbitrarily reassign addresses to avoid conflicts.
- REQUEST_RACKID
-
This message is used when an FFSC is requesting a specific
rack ID, presumably to satisfy a user request.
It is treated the same as an IDENTITY message except that
the master FFSC will always resolve rack ID conflicts in
favor of the sender.
All of these messages are sent as raw ethernet packets with a
fixed, non-standard protocol type, in order to distinguish
them from normal IP packets.
VxWorks has a mechanism for filtering incoming raw ethernet
packets.
When an ethernet packet with the special address negotiation
protocol is detected, it will be sent to the address daemon
rather than the normal networking code via a named pipe.
(Note that VxWorks does indeed allow writing to named pipes
from interrupt services routines.)
The general operation of the address daemon is as follows:
-
Set up initial addresses for the FFSC.
If the FFSC has NVRAM available to it, then the initial
addresses could be obtained from there.
Otherwise, generate addresses as follows:
- Unique ID
-
Always taken from the MAC address of the ethernet
device.
This address is never subject to change.
- IP Address
-
Use some fixed class-A network address for the network
portion, and some 24-bit value hashed from the unique ID
for the host portion.
- Rack ID
-
"Unassigned"
Note that as long as no new modules are added to the
system, each module should end up with the same addresses
upon power-on that it had after successfully completing
address negotiation the first time it was added to the
system.
This should be helpful during debugging.
However, addresses are subject to change at the address
daemon's whim and so should not be relied upon in general.
-
Send out a "QUERY" message containing the current
proposed addresses.
Any other FFSC's should then respond with their own IDENTITY
messages.
-
Process any incoming "FFSC Address Daemon"
messages:
- IDENTITY
-
Save the information in the message for future
reference.
Note that the message may supersede a previous IDENTITY
message for the same unique ID.
If an IDENTITY message is received by an address daemon
that contains its own unique ID, then something is
terribly wrong; print appropriate error messages.
The "master FFSC" (see below) may do
additional work.
- QUERY
-
Same processing as for IDENTITY messages, then send out an
IDENTITY message for the local FFSC.
Set a timer for some fixed period of time; when the
timer goes off, examine the list of known FFSC's and do
the following:
-
If an FFSC has not responded with an IDENTITY
message since the last QUERY message was received,
remove it from the list.
-
The FFSC with the lowest unique ID in the list is
designated the "master FFSC".
- ASSIGN
-
If the message specifies a different unique ID than the
one belonging to the local FFSC, the message is ignored.
Otherwise, reset the identifiers associated with the
local FFSC to the values specified in the message.
If NVRAM is available, the new identifiers should be
stored there as well.
Once everything has been changed, send out an IDENTITY
message with the updated identifiers.
The Master FFSC
One FFSC in a configuration is designated the "master
FFSC".
In addition to the usual responsibilities, the master FFSC is
responsible for resolving conflicting identifiers.
If it sees two IDENTITY messages that have different unique
IDs but match on some other identifier (e.g. IP address, rack
ID) it arbitrarily selects one of the two unique IDs and
updates the conflicting identifier(s) as follows:
- IP address
-
Select some other random IP address that is not currently in
use by another FFSC.
- Rack ID
-
Print an appropriate error message, then reset the rack ID
to "UNASSIGNED".
(The general issue of dealing with FFSC's that have
unassigned rack ID's is TBD.)
It then sends out an ASSIGN message with the updated information.
The selection of a master FFSC is done shortly after a QUERY message
has been sent by any address daemon.
The following steps are performed on receipt of a QUERY message:
-
A timestamp is obtained, which will be used later for
determining stale identity table entries.
-
The address daemon sets a timer for some fairly short period
of time (a second or two).
-
If a QUERY or IDENTITY message is received before the timer
has gone off, the timer is restarted with the original
value.
As a result, the timer should not go off until no QUERY
or IDENTITY messages have been received for the
selected period of time.
-
When the timer finally goes off, the table of identity
information is examined.
Any entries that were received prior to the timestamp that
was taken when the last QUERY message was encountered are
discarded.
The remaining entry with the lowest unique ID is deemed the
master FFSC.
It is assumed that all FFSC's will successfully receive all
QUERY and IDENTITY messages that are broadcast, so each FFSC
will reach the same conclusion as to the identity of the
master FFSC.
It may be necessary to revisit this problem if that turns out
to be a bad assumption.
It is possible or even likely that an FFSC may not be powered
up or initialized when other FFSC's broadcast their initial
QUERY or IDENTITY messages, and so may miss some of them.
Any FFSC can cause all of the other FFSC's to rebroadcast
IDENTITY messages by simply broadcasting a QUERY message.
This should probably be done routinely by the master FFSC once
all of the FFSC's are likely to be powered up.
Since the FFSC's are powered separately from the rest of the
system, and presumably must all be powered on before the rest
of the system can be powered on, a user "system power
up" command might be a good time to do this.
It probably wouldn't hurt for the master FFSC to send out a
QUERY message every minute or so even during normal operation
in order to drop any FFSC's that have been powered off or
otherwise removed from service.
A user "query addresses" command might be useful as
well.
It should cause the broadcast of a QUERY message, then wait
until no IDENTITY messages have been received for some fixed
period of time.
At that point, it should be safe for it to dump out the
contents of the local identity table.
up a level
Send questions and comments to:
Rob Bradshaw
Last modified: Sat Sep 7 14:40:47 PDT