Man Pages
CLSHMD(1M) CLSHMD(1M)
NAME
clshmd - Daemon for inter-partition shared-memory communication over
CrayLink.
SYNOPSIS
clshmd <unit>
DESCRIPTION
clshmd is a daemon which facilitates the setting up and tearing down of
shared-memory communication over CrayLink between different partitions of
a partitioned Origin-2000 machine. clshmd should not be started up
directly from the command-line. Rather, the startup function of
clshmctl(1m) should be used to set up this daemon.
Each partition of a partitioned Origin-2000 runs a distinct kernel.
CrayLink Shared-Memory allows low-latency shared-memory communication
between applications running on the distinct partitions. For such
communication to be effected, physical memory from one partition is
mapped to user buffers in a different partition. The different kernels
which are involved in such mappings have to exchange information for the
mapping to take place. clshmd allows the kernels to communicate over
CrayLink using an RPC-like mechanism. It should be noted that the kernels
need to communicate only when the inter-partition-mapped buffers are set
up, torn down, when a new partition comes up dynamically, or when an
existing partition is brought down for administrative maintainence or
other reasons. Neither clshmd, nor the different kernels are involved
during steady-state inter-partition shared-memory communication. The
clshmd daemon contains some state necessary to allow the xp_func(3p) user
library to have a system V-like interface which includes part_func(3p)
helper functions. It also has a shared memory segment that is shared
between it and the clshm kernel driver. It receives messages from the
driver in this area and sends messages to the driver through ioctl(2)
calls.
clshmd takes only one parameter. This parameter is the <unit> number of
the clshm driver. This should always be "0". Other numbers are reserved
for future expansion. All other controlling state of the daemon can be
set with the different parameters of the clshmctl(1m) command before the
clshmctl startup command is issued.
clshmd uses two hwgraph files. /hw/clshm/0/ctl and /hw/clshm/0/shm are
used by the clshm. The "0" in the paths are the unit number of the clshm
driver. ctl stands for "control" file which is used for setting up
shared memory between the driver and daemon and the shm stands for
"shared memory" file which is used by the user library to mmap shared
memory segments into user processes.
FILES
/hw/clshm/
SEE ALSO
clshmctl(1m), xp_func(3p), part_func(3p), clshm_diag(1m), mkpart(1m)
CLSHMCTL(1M) CLSHMCTL(1M)
NAME
clshmctl - Administration tool for CrayLink Shared Memory
SYNOPSIS
clshmctl startup
clshmctl shutdown
clshmctl getcfg
clshmctl maxpgs <num of pages>
clshmctl maxdpgs <num daem pages>
clshmctl timeout <daemon timeout>
DESCRIPTION
clshmctl is a system-administrator's tool for controlling the resources
associated with CrayLink-shared-memory (clshm) -- a means of cache-
coherent, shared-memory communication between partitions on an Origin-
2000. Each partition of an Origin-2000 has its own hostname, IP address
and operating-system, and behaves as a stand-alone host. clshm has a
system V shared memory interface in the form of a user library
(xp_func(3p) and part_func(3p)) that can be used directly by user-
applications.
clshmctl performs various functions needed to administer the software
devices and daemon associated with clshm and returns operational status
information.
clshmctl is invoked with one of the following function names, and the
associated parameters:
startup Configure clshm on the local partition, and start up the
clshm devices and clshmd(1m) daemon with default parameters.
The value of the defaults can be examined by using the
clshmctl getcfg function, as explained later. The startup
step is performed at system boot-up by /etc/init.d/sn0start
and is required before any inter-partition communication can
be effected over CrayLink Shared Memory. This startup at
bootup time must be enabled using the chkconfig(1m) command
for clshmd(1m). It also needs to be done if clshm has been
reset by a clshmctl shutdown command. clshmctl startup also
creates and initializes the software devices through which
shared-memory communication is performed.
shutdown Forces a reset of the clshm software on the local partition.
The clshmd(1m) daemon is made to exit and the resources
associated with the clshm software devices are released.
clshmctl shutdown should be used carefully since any user-
program using the clshm devices will be killed when this
function is called.
getcfg Returns the configuration of clshm on the local partition.
The parameters configurable through clshmctl are the maximum
number of 16K pages of shared-memory that are supported, the
number of 16K pages shared between the daemon and the kernel
driver, and the timeout time before the clshmd(1m) daemon
wakes up from sleeping and checks for a message from the
driver when there is no user activity going on.
maxpgs <num of pages>
Sets the maximum number of pages of shared memory that can be
associated with any of the channels on the local partition.
maxpgs <num of pages> will fail if it is called after
clshmctl startup has been called (without another clshmctl
shutdown). In order to set this parameter, the clshmd(1m)
daemon must be in the shutdown state.
clshmctl maxdpgs <num daem pages>
Sets the maximum number of shared memory pages that the clshm
driver can use to communicate with the clshmd(1m) daemon.
Like maxpgs, this call will fail if made after a clshmctl
startup call has been made.
clshmctl timeout <daemon timeout>
Sets the timeout value in milliseconds that the clshmd(1m)
daemon will sleep before checking for messages in the shared
memory segment from the driver. If there are messages coming
in on either other daemon or user sockets, then the driver
will not sleep the full amount of the timeout, but will
process all messages if fines on all sockets and in the
shared memory segment.
WARNINGS
If the clshmd(1m) daemon aborts abnormally a clshmctl shutdown must be
called before a clshmctl startup can be called again to restart the
shared-memory mechanism.
FILES
/etc/init.d/sn0start
SEE ALSO
clshmd(1m), chkconfig(1m), xp_func(3p), part_func(3p), clshm_diag(1m),
mkpart(1m)
CLSHM_DIAG(1M) CLSHM_DIAG(1M)
NAME
clshm_diag - Diagnostic tool for CrayLink Shared Memory
SYNOPSIS
clshm_diag
DESCRIPTION
clshm_diag is a system-administrator's tool for ensuring that the
daemons, devices and driver required for communication between partitions
using shared-memory over CrayLink (clshm), have been properly set up.
clshm_diag causes the exchange of information between the clshmd(1m)
daemon on the local partition, and its counterpart on every remote
partition known to the local partition. Services are known to be up if we
can send and receive hostname information from other partitions' daemons.
The daemons will not come up if the driver is not present. If the
exchange of hostnames can occur then all services are up and ready to
run. clshm_diag prints out the hostnames of all partitions that we can
successfully communicate and indicates all partitions that it has
knowledge, but cannot communicate with, so that the system-administrator
can investigate the problem further.
SEE ALSO
clshmctl(1m), clshmd(1m), xp_func(3p), part_func(3p), mkpart(1m)
XP_FUNC(3P) XP_FUNC(3P)
NAME
xp_func: xp_ftok, xp_shmget, xp_shmat, xp_shmdt, xp_shmctl - CrayLink
Shared Memory Functions
SYNOPSIS
#include <xp_shm.h>
key_t xp_ftok(const char *path, int id);
int xp_shmget(key_t key, size_t size, int shmflg);
void *xp_shmat(int shmid, const void *shmaddr, int shmflg);
int xp_shmdt(const void *shmaddr);
int xp_shmctl(int shmid, int cmd, .../*struct shmid_ds *buf*/);
DESCRIPTION
xp_func calls are a user library interface for CrayLink-shared-memory
(clshm) -- a means of cache-coherent, shared-memory communication between
partitions on an Origin-2000. The functions try to be very much "system
V-like" in their interface.
xp_ftok creates a key of type key_t given a path and an id much like the
ftok(3c) call. The path should either be /hw/clshm/partition/<2 digit
hex partition #> or NULL. The shared memory associated with this key
will be placed on the specified partition. If no partition was specified
it will simply be placed on the partition that the process is running.
The id should always be passed in a "1". Everytime this function is
called a different key will be returned.
xp_shmget must be passed a key that is created by xp_ftok in order to
create a shared memory segment and its associated shmid. The size that
is passed in, must be the same size as all other xp_shmget calls that
have been called with the same key. The shmflg parameter is ignored.
xp_shmat must be called with a shmid that was returned from a xp_shmget
call in order to attach a shared memory segment. The shmaddr and shmflg
parameters have the same semantics as the shmat(2) function.
xp_shmdt detaches the given shared memory segment for the given shmaddr.
The last detach does not automatically remove the shared memory segment
unless the IPC_AUTORMID command was sent to the xp_shmctl function.
xp_shmctl is much like the shmctl(2) call. But it only implements the
IPC_RMID, command which removes a shared memory segment associated with
the shmid passed in. It also implements the IPC_AUTORMID, command which
removes the shared memory segment associated with shmid after the last
xp_shmdt call. If the last xp_shmdt call has already been made, then the
shared memory segment is removed immediately.
All xp_func calls will fail and not perform any function if one of the
following are true:
ENOMEM A malloc(3c) call failed.
EPROTO This version of the library is not compatable with the
local clshmd(1m) daemon's version.
EIO There has been a communication problem with the local
clshmd(1m) daemon.
EINVAL The function was sent invalid arguments.
ENOSYS The argument passed in is not supported at this time.
EFAULT There has been an internal problem with the user library or
clshmd(1m) daemon.
ENAMETOOLONG The path is too long to put into the specified buffer.
These functions can also fail with the errno returned from various unix
calls that the library makes.
WARNINGS
If xp_shmat is called after the last xp_shmdt and the IPC_AUTORMID
command has been sent through xp_shmctl, then the behavior is undefined.
SEE ALSO
part_func(3p), clshmctl(1m), clshmd(1m), mkpart(1m)
DIAGNOSTICS
Upon successful completion, the return values are as follows:
xp_ftok returns a valid key_t type key.
xp_shmget returns a valid shmid.
xp_shmat returns the data segment start address of the attached
shared memory segment.
xp_shmdt returns 0.
xp_shmctl returns 0.
Otherwise, a value of -1 is returned and errno is set to indicate the
error.
PART_FUNC(3P) PART_FUNC(3P)
NAME
part_func: part_getid, part_getcount, part_getlist, part_gethostname,
part_getpart, part_setpmo, part_getnode - CrayLink Shared Memory Helper
Functions
SYNOPSIS
#include <xp_shm.h>
partid_t part_getid(void);
int part_getcount(void);
int part_getlist(partid_t *part_list, int max_parts);
int part_gethostname(partid_t part, char *name, int max_len);
partid_t part_getpart(int shmid);
int part_setpmo(key_t key, pmo_handle_t pmo_handle, pmo_type_t pmo_type);
int part_getnode(int shmid, char *hwpath, int maxlen);
DESCRIPTION
part_func calls are part of the xp_func(3p) user library interface for
Craylink shared memory (clshm). They are functions that do not map
directly to system V shared memory like calls that the xp_func(3p) calls
do.
part_getid gets the partition id number for the partition that the
process is running on.
part_getcount gets the number of partitions that are currently active
according to the partition that the process is running on.
part_getlist returns the number of partitions that are currently active.
The partition numbers of the active partitions are placed int the
part_list array whose lenth is passed in through the max_parts parameter.
part_gethostname takes a active partition number and places the hostname
of that partition in the name string. The maximum length this string can
be filled up to is passed in the max_len parameter.
part_getpart takes a shmid and returns the partition id of the partition
that hosts the memory associated with this shmid.
part_setpmo takes a key, pmo_handle, and pmo_type and associates the pmo
information with the key. The key must come from xp_ftok(3p) and must be
hosted by the partition that the current process is running on. This
call must be made before any xp_shmget(3p) calls are made to associate a
shmid with this key. See the mld(3) interface for how to use memory
locality domains to control the placement of memory for this key.
part_getnode takes a shmid and hwpath string with maximum length of
max_len to fill in the hwgraph(4) path of the node that the shmid shared
memory has been allocated on. This is only guaranteed to return a valid
string if both part_setpmo and xp_shmget both succeeded for the shmid and
the part_getnode function is being called from the partition that hosts
the shmid shared memory. However, if all memory associated with a shared
memory segment is placed on the same node without part_setpmo being
called, then this call will also return a valid hwgraph(4) string, given
that it is called from the correct partition.
All part_func calls will fail and not perform any function if one of the
following are true:
ENOMEM A malloc(3c) call failed.
EPROTO This version of the library is not compatable with the
local clshmd(1m) daemon's version.
EIO There has been a communication problem with the local
clshmd(1m) daemon.
EINVAL The function was sent invalid arguments.
ENOSYS The argument passed in is not supported at this time.
EFAULT There has been an internal problem with the user library or
clshmd(1m) daemon.
ENAMETOOLONG The path is too long to put into the specified buffer.
These functions can also fail with the errno returned from various unix
calls that the library makes.
SEE ALSO
xp_func(3p), mld(3), hwgarph(4), clshmctl(1m), clshmd(1m), mkpart(1m)
DIAGNOSTICS
Upon successful completion, the return values are as follows:
part_getid returns a valid partition id.
part_getcount returns the number of active partitions.
part_getlist returns the number of active partitions and fills in
the part_list array.
part_gethostname returns 0.
part_getpart returns the partition number of the partition hosting
the shmid memory.
part_setpmo returns 0.
part_getnode returns 0.
Otherwise, a value of -1 is returned and errno is set to indicate the
error.