This document is intended to be a high level overview of the software used to run the AO f/15 secondary at the MMT Observatory. These notes were begun in October of 2010, and as much as possible have been worked up from study of the source files themselves. In addition, information has been obtained from the AO Wiki, as well as from discussions with cognizant individuals, some of whom are:
Note that this discusses only the NGS system. The LGS system has some subsystems in common, but also has many unique components and issues.
Some goals of this writeup are:
DDS also runs on this machine. This is the "dataserver". When the PCR code was first being developed, it was partitioned into real time code (now the PCR code) and non-real time code (what is now DDS). Vidhya dreams of the day when DDS will go away.
At present (11/2010) DDS runs on ao-pcr, but it could and maybe should run on ao-server (but this has not been moved because no doubt unexpected dependencies exist and things would break and need to be sorted out).
The following is from pcr/src/include/ports.h
#define COMMAND_PORT 8001 #define STREAM_PORT 8002 #define TESTPORT 8003 #define ERROR_PORT 8004 #define TILT_PORT 8005 #define CLOSEDLOOP_PORT 8006 #define ACCELEROMETER_PORT 8007 #define STORAGE_PORT 8009 #define TCOMMAND_PORT 8011 #define TSTREAM_PORT 8012 #define TERROR_PORT 8014 #define TTILT_PORT 8015 #define TCLOSEDLOOP_PORT 8016 #define TACCELEROMETER_PORT 8017
Here is a brief outline of what goes on in the PCR code.
Three hosts are connected to a DLink ethernet switch: ao-pcr, ao-server, and the microgate. When in open loop mode, xadsec (running on ao-server) sends commands to the DM, when the loop closes there is a handoff and the pcr code (running on ao-pcr) sends commands to the DM. Communication is done via "jumbo packets" (normal ethernet packets are limited to 1500 bytes, whereas jumbo packets allow up to 9000 bytes). This allows the command vector to be sent in a single packet. This requires a jumbo frame compatible switch (which the DLink is), as well as jumbo frame compatible NIC cards and drivers within linux.
The machine ao-server has two identical ethernet cards, lspci shows these to be:
06:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01) 06:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)Under linux, this is supported by the e1000 driver. Jumbo packets are supported by setting the MTU size, which can be 2048, 4096, 8192, or 16384 bytes. This is done via a command like:
ifconfig eth1 mtu 9000 upAlternately, on Red Hat systems, a setting such as:
MTU=9000can be added to the file /etc/sysconfig/network-scripts/ifcfg-eth1.
Note that some Intel gigabyte NIC cards do not support jumbo frames (those based on 82542 and 892573V/E chips).
check_tss - this is a ruby script in /mmt/scripts that can be run on any of our linux machines that mount the /mmt directory. It communicates with the tss_server (which at the time of this writing runs on hacksaw) to check the status of the TSS, as well as the tss_server. Tom wrote this script.
tss_server - this is a special miniserver than communicates with the TSS. This is a perl script written by Tom. It is managed by the mmtserv system, and can be started (or restarted) via the command:
mmtserv tss_server restart
reset_cyclades - the tss_server communicates with the TSS via port 3 on the cyclades terminal server "mmtts2" (and from the cyclades to a serial to fiber converter, then via fibers to the TSS which accepts fibers. The "reset_cyclades" script may be run by user mmtao on the machine ao-server to kick the cyclades in the butt, which is sometimes needed. This script is only on ao-server, which is not powered on right now, so I cannot investigate details.
dm_status - this is a GUI which is usually started from the Application pulldown menu (via MMT --> MMTAO GUIs --> DM Status GUI. This will try to run the script /home/mmtao/ao/bin/dm_status, so this will probably only work on the machine hoseclamp. This was an IDL script in the good old days, but now is a shell script which runs the script /mmt/scripts/dm_status. This script is a ruby/gtk GUI written by Tom. Note that there is also a web version of this script available: https://hacksaw.mmto.arizona.edu/ssl/dm_status/ The note in the wiki about hogging of IDL licenses when running this script is no longer relevant.
loft neslab control - there is a web based interface to control the loft neslab available at: https://hacksaw.mmto.arizona.edu/ssl/loft_neslab_control/
cd /home/mmtao/ao/tcl/gui exec /home/mmtao/ao/wish/PCR_Wish -f gui_startup.tcl
The actual script relies on the setting of the PCR_HOME variable, followed by two lines which are verbatim as follows:
PCR_HOME=/home/mmtao/ao
cd ${PCR_HOME}/tcl/gui
exec ${PCR_HOME}/wish/PCR_Wish -f gui_startup.tcl
Doug Miller is the original author of this script.
Skip Schaller has had extensive experience with it and is the resident expert.
The PCR_Wish interpreter is a specially compiled tcl/wish interpreter with C extensions
added that are used by the tcl code which makes up the bulk of the script.
The script gui_startup.tcl is a 361 line tcl script, most of which sources other tcl
files (it is estimated that approximately 150 tcl files comprise this GUI).
The ao_gui process starts (or can start) other processes, including:
Much of the mechanism for controlling processes is handled by Tcl code in the directory /home/mmtao/ao/tcl/process, in particular /home/mmtao/ao/tcl/process/process_start.tcl This script invokes either a start, stop, or kill script in /home/mmtao/ao/bin for the process from the following list:
start_data_server_log stop_data_server_log -- echo data | nc `mmt/bin/mmtservdata mmtaodata` (Skip) start_pcr stop_pcr kill_pcr -- run "local_pcr_ao_pcr" on mmtao@ao-pcr -- on that machine: cd /NGSAO/latest/bin; ./pcr start_pcr_server stop_pcr_server kill_pcr_server -- mmtserv ngs_info start -- runs /home/mmtao/ao/bin/data_server start_dds stop_dds kill_dds -- run "local_dds_ao-pcr" on mmtao@ao-pcr -- on that machine: cd /NGSAO/latest/src/dds/src ; ./dds start_topbox_server stop_topbox_server kill_topbox_server -- mmtserv ngs_topbox start -- runs /home/mmtao/ao/bin/topbox_server start_science_server stop_science_server kill_science_server -- mmtserv science_server start -- runs /home/mmtao/ao/bin/science_server (ruby, by Skip) start_science_server_log stop_science_server_log -- echo science | nc `mmt/bin/mmtservdata mmtaodata` (Skip) start_tss_server stop_tss_server kill_tss_server -- mmtserv tss_server start -- runs /mmt/tss_server/tss_server
pcr_server is also called ngs_info, and runs something called "data_server". This is not the same as DDS, which may also be called a data_server. Nobody should be confused by any of this, which is perhaps historical.
mmtsrvdata is a little bash script that does srv record lookups. mmtsrvdata mmtaodata yields: ao-server.mmto.arizona.edu 7853.
There is a service (handled by mmtserv) called mmtaodata.
(Beware, this may also be refered to as a data server, that makes 3 components
of the AO system that may in different contexts be called "data server".)
It runs: /home/mmtao/ao/bin/logtail.rb mmtaodata /home/mmtao/data
Skip wrote it.
It is started as follows:
ssh adsec@ao-server cd LISEC idlde startup.pro -- choose "BCU" communications -- answer yes to all subsequent questions
ssh adsec@ao-server cd savebox/bin ./savedomThis is a compiled program written in C, and written by Vidhya. It is approximately 1000 lines of code. The source code for this program is in:
ao-server:/home/adsec/savebox/src
This program is only useful when the CLIO instrument is running. Software running on CLIO sends a time packet to this code, which responds by returning the packet with the filename of a telemetry file filled in.
This program listens for UDP packets on two ports: 8008 and 8009. Port 8008 is the time port, 8009 is the data port. This program uses pthreads to listen on both ports.
The time thread (listening to port 8008), when it gets data from CLIO, attempts to log what it is doing to /home/observer/log/mmmddyyyy_clio.log, but the directory /home/observer/log does not exist the machine on ao-server, so this fails. The time thread also responds via UDP to whatever machine is sending it messages. It is not at all clear what software on what machine sends these messages.
Port 8009 (CLOSEDLOOP_PORT in savedom and dds code, STORAGE_PORT in the pcr code) comes from the PCR code on machine ao-pcr. Note that the PCR code is always and continually sending UDP packets, which do no harm if savedom is not running to listen to them.
The following lines of code are found in the PCR source in the file pcr/src/ngs/I_MainServer.c:
printf (" Help: \n ");
printf (" PORT: 8001 - Control Commands \n ");
printf (" PORT: 8002 - WFSC Stream \n ");
printf (" PORT: 8004 - Interrupt Error Reports \n ");
printf (" PORT: 8005 - Science camera Tip/Tilt signals \n ");
printf (" PORT: 8007 - Accelerometer Tip/Tilt signals \n ");
printf (" PORT: 8009 - UDP port to savebox \n\n ");
Note that CLOSEDLOOP_PORT is 8006 in the pcr code.
When data is received, files are written in directory /home/observer/Oct242010 (replace the specific mmmddyyyy with the current date) of the form:
cmds_Oct242010_220112_736101.fits cmds_Oct242010_220150_674102.fits cmds_Oct242010_220228_626103.fits cmds_Oct242010_220306_564104.fitsEach time data is received, 6 files are written, with the prefixes: cmd_, cur_, pos_, rtr_, slopes_, and spots_. After the prefix, the filename consists of the day in mmmddyyyy form, the time within the day in milliseconds, and a random value to ensure unique filenames.