MMT AO Software notes

This document is intended to be a high level overview of the software used to run the AO f/15 secondary at the MMT Observatory. These notes were begun in October of 2010, and as much as possible have been worked up from study of the source files themselves. In addition, information has been obtained from the AO Wiki, as well as from discussions with cognizant individuals, some of whom are:

Note that this discusses only the NGS system. The LGS system has some subsystems in common, but also has many unique components and issues.

Some goals of this writeup are:

Machines involved

Linux machines

Dedicated machines

Software on hacksaw

Software on ao-pcr

The pcr (aka reconstructor) software is written and maintained in its current form by Vidhya Vaitheeswaran. It is a large body of C code, and in many ways is the "guts" of the AO system.

DDS also runs on this machine. This is the "dataserver". When the PCR code was first being developed, it was partitioned into real time code (now the PCR code) and non-real time code (what is now DDS). Vidhya dreams of the day when DDS will go away.

At present (11/2010) DDS runs on ao-pcr, but it could and maybe should run on ao-server (but this has not been moved because no doubt unexpected dependencies exist and things would break and need to be sorted out).

The following is from pcr/src/include/ports.h

#define COMMAND_PORT     8001
#define STREAM_PORT       8002
#define TESTPORT          8003
#define ERROR_PORT        8004
#define TILT_PORT         8005
#define CLOSEDLOOP_PORT   8006
#define STORAGE_PORT      8009

#define TCOMMAND_PORT      8011
#define TSTREAM_PORT       8012
#define TERROR_PORT        8014
#define TTILT_PORT         8015
#define TCLOSEDLOOP_PORT   8016

Here is a brief outline of what goes on in the PCR code.

Notes on Microgate communication

The Microgate is a big piece of custom electronics which did not work. (At least it did not work to do the reconstruction calculations.) It is now used in a degenerate "pass through" mode to send commands to the DM. Communication with the Microgate is via a small dedicated ethernet.

Three hosts are connected to a DLink ethernet switch: ao-pcr, ao-server, and the microgate. When in open loop mode, xadsec (running on ao-server) sends commands to the DM, when the loop closes there is a handoff and the pcr code (running on ao-pcr) sends commands to the DM. Communication is done via "jumbo packets" (normal ethernet packets are limited to 1500 bytes, whereas jumbo packets allow up to 9000 bytes). This allows the command vector to be sent in a single packet. This requires a jumbo frame compatible switch (which the DLink is), as well as jumbo frame compatible NIC cards and drivers within linux.

The machine ao-server has two identical ethernet cards, lspci shows these to be:

06:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
06:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Under linux, this is supported by the e1000 driver. Jumbo packets are supported by setting the MTU size, which can be 2048, 4096, 8192, or 16384 bytes. This is done via a command like:
ifconfig eth1 mtu 9000 up
Alternately, on Red Hat systems, a setting such as:
can be added to the file /etc/sysconfig/network-scripts/ifcfg-eth1.

Note that some Intel gigabyte NIC cards do not support jumbo frames (those based on 82542 and 892573V/E chips).

Software on ao-server

Software on hoseclamp

IDL and IDL versions

Here is a quick rundown on IDL versions currently (11/2010) in use:

Reflective Memory

Some special hardware was purchased to implement reflective memory as a high performance communication link. A pair of PCI cards (one in the machine ao-pcr, the other in the machine ao-server) are connected by a pair of fibers. Apparently though this has never been placed into operation, and these machines communicate through regular network sockets.

Special user identities to support AO software

Several linux users exist and are essential in the current scheme of things to support the AO software system. There are a large number of environment variables which are set up by the user startup scripts which are interrogated by other scripts and software.

Software Startup

check_tss - this is a ruby script in /mmt/scripts that can be run on any of our linux machines that mount the /mmt directory. It communicates with the tss_server (which at the time of this writing runs on hacksaw) to check the status of the TSS, as well as the tss_server. Tom wrote this script.

tss_server - this is a special miniserver than communicates with the TSS. This is a perl script written by Tom. It is managed by the mmtserv system, and can be started (or restarted) via the command:

	mmtserv tss_server restart

reset_cyclades - the tss_server communicates with the TSS via port 3 on the cyclades terminal server "mmtts2" (and from the cyclades to a serial to fiber converter, then via fibers to the TSS which accepts fibers. The "reset_cyclades" script may be run by user mmtao on the machine ao-server to kick the cyclades in the butt, which is sometimes needed. This script is only on ao-server, which is not powered on right now, so I cannot investigate details.

dm_status - this is a GUI which is usually started from the Application pulldown menu (via MMT --> MMTAO GUIs --> DM Status GUI. This will try to run the script /home/mmtao/ao/bin/dm_status, so this will probably only work on the machine hoseclamp. This was an IDL script in the good old days, but now is a shell script which runs the script /mmt/scripts/dm_status. This script is a ruby/gtk GUI written by Tom. Note that there is also a web version of this script available: The note in the wiki about hogging of IDL licenses when running this script is no longer relevant.

loft neslab control - there is a web based interface to control the loft neslab available at:

The script: ao_gui

This script does support and housekeeping and is started up before the main AO gui, which is xadsec. This script is in /home/mmtao/ao/bin/ao_gui, and will typically be invoked by user mmtao on the machine hoseclamp. This script runs on hoseclamp. Note that the directories /home/mmtao/bin and /home/mmtao/ao/bin both exist and are both on the search path for that user. This script effectively consists of the two lines:
cd /home/mmtao/ao/tcl/gui
exec /home/mmtao/ao/wish/PCR_Wish -f gui_startup.tcl

The actual script relies on the setting of the PCR_HOME variable, followed by two lines which are verbatim as follows:

cd ${PCR_HOME}/tcl/gui
exec ${PCR_HOME}/wish/PCR_Wish -f gui_startup.tcl
Doug Miller is the original author of this script. Skip Schaller has had extensive experience with it and is the resident expert. The PCR_Wish interpreter is a specially compiled tcl/wish interpreter with C extensions added that are used by the tcl code which makes up the bulk of the script. The script gui_startup.tcl is a 361 line tcl script, most of which sources other tcl files (it is estimated that approximately 150 tcl files comprise this GUI).

The ao_gui process starts (or can start) other processes, including:

Much of the mechanism for controlling processes is handled by Tcl code in the directory /home/mmtao/ao/tcl/process, in particular /home/mmtao/ao/tcl/process/process_start.tcl This script invokes either a start, stop, or kill script in /home/mmtao/ao/bin for the process from the following list:

	-- echo data | nc `mmt/bin/mmtservdata mmtaodata` (Skip)

	-- run "local_pcr_ao_pcr" on mmtao@ao-pcr
	-- on that machine:  cd /NGSAO/latest/bin; ./pcr

	-- mmtserv ngs_info start
	-- runs /home/mmtao/ao/bin/data_server

	-- run "local_dds_ao-pcr" on mmtao@ao-pcr
	-- on that machine:  cd /NGSAO/latest/src/dds/src ; ./dds

	-- mmtserv ngs_topbox start
	-- runs /home/mmtao/ao/bin/topbox_server

	-- mmtserv science_server start
	-- runs /home/mmtao/ao/bin/science_server (ruby, by Skip)

	-- echo science | nc `mmt/bin/mmtservdata mmtaodata` (Skip)

	-- mmtserv tss_server start
	-- runs /mmt/tss_server/tss_server

Notes on the above:

pcr_server is also called ngs_info, and runs something called "data_server". This is not the same as DDS, which may also be called a data_server. Nobody should be confused by any of this, which is perhaps historical.

mmtsrvdata is a little bash script that does srv record lookups. mmtsrvdata mmtaodata yields: 7853.

There is a service (handled by mmtserv) called mmtaodata. (Beware, this may also be refered to as a data server, that makes 3 components of the AO system that may in different contexts be called "data server".) It runs: /home/mmtao/ao/bin/logtail.rb mmtaodata /home/mmtao/data
Skip wrote it.

The script: xadsec (aka adsec)

This is an IDL script and is the main GUI for the ao system. (ao_gui does support and housekeeping, and although not as central or complex, is also required to run the system). The adsec script is refered to as both adsec and xadsec. These days, we always run xadsec, which is the GUI version (using X windows) of adsec (aparently a command line adsec was once used). Calling it an IDL script is a bit of an understatement, it is actually a large body of IDL code consisting of many separate files. It runs on ao-server, with remote displays on whatever machine was used to start it (typically hoseclamp). Vidhya has done the bulk of the work on this code. Guido and Armando did early work on the adsec code.

It is started as follows:

		ssh adsec@ao-server
		cd LISEC
		-- choose "BCU" communications
		-- answer yes to all subsequent questions

The savedom "data saver" script

The wiki states that this program may be run if desired by issuing the following commands on hoseclamp:
		ssh adsec@ao-server
		cd savebox/bin
This is a compiled program written in C, and written by Vidhya. It is approximately 1000 lines of code. The source code for this program is in:

This program is only useful when the CLIO instrument is running. Software running on CLIO sends a time packet to this code, which responds by returning the packet with the filename of a telemetry file filled in.

This program listens for UDP packets on two ports: 8008 and 8009. Port 8008 is the time port, 8009 is the data port. This program uses pthreads to listen on both ports.

The time thread (listening to port 8008), when it gets data from CLIO, attempts to log what it is doing to /home/observer/log/mmmddyyyy_clio.log, but the directory /home/observer/log does not exist the machine on ao-server, so this fails. The time thread also responds via UDP to whatever machine is sending it messages. It is not at all clear what software on what machine sends these messages.

Port 8009 (CLOSEDLOOP_PORT in savedom and dds code, STORAGE_PORT in the pcr code) comes from the PCR code on machine ao-pcr. Note that the PCR code is always and continually sending UDP packets, which do no harm if savedom is not running to listen to them.

The following lines of code are found in the PCR source in the file pcr/src/ngs/I_MainServer.c:

    printf (" Help: \n ");
    printf (" PORT: 8001 - Control Commands \n ");
    printf (" PORT: 8002 - WFSC Stream \n ");
    printf (" PORT: 8004 - Interrupt Error Reports \n ");
    printf (" PORT: 8005 - Science camera Tip/Tilt signals \n ");
    printf (" PORT: 8007 - Accelerometer Tip/Tilt signals \n ");
    printf (" PORT: 8009 - UDP port to savebox \n\n ");
Note that CLOSEDLOOP_PORT is 8006 in the pcr code.

When data is received, files are written in directory /home/observer/Oct242010 (replace the specific mmmddyyyy with the current date) of the form:

Each time data is received, 6 files are written, with the prefixes: cmd_, cur_, pos_, rtr_, slopes_, and spots_. After the prefix, the filename consists of the day in mmmddyyyy form, the time within the day in milliseconds, and a random value to ensure unique filenames.