July 5, 2019

Grinder - Keeping a FITS file mirror

At this time, we are controlling the oven with Crater, and the current set of data files (in old IRAF format) are on the Crater disk.

I have been keeping a mirror of these files on Dorado in FITS format, and I keep the mirror up to date by hand. I use a perl script to generate a CL script (I could do this all with the CL if I was fluent). Then I run the CL script which runs WFITS on each file to both convert and copy it. The following is a cookbook for what needs to be done periodically to keep this set of files up to date. In the future we will simply write FITS files and none of this will be necessary.

What needs to be done on Dorado

Dorado accesses the files on the Crater disk via NFS. What follows could be done equally well on Crater, but doing it on Dorado reduces any (minimal anyway) load on Crater and possible impact on the pilot. Note though that running as user pilot on Dorado can have ugly impact if mistakes are made. 
ssh tom@dorado
bash
cd /d0/Archive
 -- the mirror is in "Current", which is a link to Casting_190227
ls *.cl
 -- pick a number not used in the current set of scripts
cd Sun_scripts
./mk_cur > ../wget_cur99.cl
-- I used to edit this file and trim out all the lines for
-- files that had already been copied, but this is a foolish
-- waste of time.  WFITS will simply refuse to overwrite
-- existing files and quickly move on.
su - pilot
cd /d0/Archive/Current
-- delete the partial last file from the prior mirroring
rm *190619*
cd
cl
cd /d0/Archive
cl < wget_cur99.cl

What needs to be done on the linux box

This machine mounts the /d0 disks on both Crater and Dorado read-only via NFS. This is very easy: 
ssh tom@linuxpilot
cd /u1/Archive
./rsync_archive
cd Current
tar cvf /home/tom/cur_fits.tar *.fits
This places the current set of files in /u1/Archive/Current The rather big tarball is available as a single file to move them all to some other machine.

Using rsync is by far preferable, but is impossible to the suns, and awkward to the linux pilot machine due to firewall proxy issues.

Have any comments? Questions? Drop me a line!

Tom's home page / tom@mmto.org