Do not panic and switch too hastily. Switching workstations makes a mess of the oven data by breaking image files into several pieces. If you make a mistake, the database files could also be damaged.
Neither Pilot nor Pilot2 should be running oven on the other workstation (crater or dorado) at the same time. This is not impossible, but it is rather dangerous --- you could destroy the other database if you make a mistake.
Make sure the other workstation is functional before you get too involved in this process.
Note that pilot's home directory is physically on crater!/d0. So that if crater dies, pilot cannot login to any machine anywhere. Similarly pilot2's home directory is physically on dorado!/d0, so pilot2 is able to login and run the oven from dorado if crater dies. (Also remember that pilot has a non-home directory on dorado!/d0/ and vice-versa that pilot2 has a non-home directory on crater!/d0/.)
If a workstation has died, or even appears to have died, you need lots of help. Start with J. Hill and S. Schaller.
This is assuming that crater is not already dead. Try to kill the oven daemons if you can; otherwise they may continue to read the error messages.
Kill whatever is running on dorado; then follow the normal instructions in pilot_login or pilot_short. Be careful with the oven parameters because they might be set to "readonly". This login will possibly involve creating shared memory and possibly killing old daemons. Be careful not to load database from disk until step 4.
The last updated backup database file (with stale clocks) should be located in "dorado!/d0/pilot/". In the blue window on dorado, "copy /d0/pilot/database /home/pilot2/database". Also in the blue window on dorado, "copy database database_save" to make sure you don't lose the old database file when reading database from oven.
This should give you the latest database file with current clocks. Do NOT do this step unless you are sure that the problem was with crater and not with the on-board computers.
You should now be piloting the oven again. If not, call for more help. Some additional supplementary information is given below, but it should not be needed under normal circumstances.
If the IRAF environment variable "backup" contains a valid path, the oven program should write a backup copy of database in that directory whenever a change to the database is made. For pilot running on crater, the command to set the backup path is "set backup = dorado!/d0/pilot/" (usually found in the loginuser.cl file). This provides a copy of the database on another machine in case of a disk or CPU problem on crater. Copies of the last 9 changes to the database are always maintained in the pilot's working directory. These automatic backups can get overwritten very quickly, so the pilot should always know the location of a "safe copy" of the database. The logbook should also carry sufficient information for the pilot to be able to reconstruct the clocks at any time (but don't change the clock time manually unless you have qualified assistance).
Usually we run the oven VxWorks software from EPROM. The EPROM version of the software can reboot without the Sun, and can get oven database information from other on-board computers or from non-volatile RAM. The typical boot setup looks like this:
> bootChange '.' = clear field; '-' = go to previous field; ^D = quit boot device : ln processor number : 0 host name : crater file name : /opt/vxworks/oven0v1/vxWorks inet on ethernet (e) : 184.108.40.206:ffffff00 t (g) : user (u) : vwuser ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (tn) : oven0v1 startup script (s) : /opt/vxworks/oven0v1/startup.cmd other (o) : value = 0 = 0x0
But, if you are using VxWorks which boots from disk, you will also need to change the boot parameters of the on-board computers before they can successfully reboot. To change the boot parameters, interrupt the auto-boot sequence and type "c" for change. If you are changing the network address of the VME computer, be sure to reboot it again and verify the name before connecting it to the network.
The typical network boot parameters look like:
-> bootChange '.' = clear field; '-' = go to previous field; ^D = quit boot device : ln processor number : 0 host name : dorado file name : /opt/vxworks/oven1v0/vxWorks inet on ethernet (e) : 220.127.116.11:ffffff00 inet on backplane (b): 0 host inet (h) : 18.104.22.168 gateway inet (g) : user (u) : vwuser ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (tn) : oven1v0 startup script (s) : /opt/vxworks/oven1v0/startup.cmd other (o) : value = 0 = 0x0
The relevant network computers are: (from /etc/hosts)
22.214.171.124 dorado.as.arizona.edu dorado 126.96.36.199 libra.as.arizona.edu libra 188.8.131.52 crater.as.arizona.edu crater 184.108.40.206 ncdxtb4.as.arizona.edu ncdxtb4 220.127.116.11 ncdxte3.as.arizona.edu ncdxte3 18.104.22.168 vwlog4.as.arizona.edu vwlog4 22.214.171.124 vw5.as.arizona.edu vw5 126.96.36.199 afone.as.arizona.edu afone 188.8.131.52 aftwo.as.arizona.edu aftwo 184.108.40.206 oven0v0.as.arizona.edu oven0v0 220.127.116.11 oven0v1.as.arizona.edu oven0v1 18.104.22.168 oven0v2.as.arizona.edu oven0v2 22.214.171.124 oven1v0.as.arizona.edu oven1v0 126.96.36.199 mmtcell.as.arizona.edu mmtcell 188.8.131.52 xcell.as.arizona.edu xcell 184.108.40.206 lw4.as.arizona.edu lw4 220.127.116.11 mlpc10.as.arizona.edu mlpc10 18.104.22.168 mlpc11.as.arizona.edu mlpc11 22.214.171.124 mlpc12.as.arizona.edu mlpc12 126.96.36.199 mlpc13.as.arizona.edu mlpc13 188.8.131.52 mlpc14.as.arizona.edu mlpc14 184.108.40.206 mlpc15.as.arizona.edu mlpc15