May 24, 2022

Analyze data from the David Gesswein MFM emulator

The last thing I did was this, and I have these two files on my linux desktop now. 
./mfm_read --transitions_file ../callan_raw1 --drive 1 --heads 8 --cylinders 320
./mfm_read --transitions_file ../callan_raw2 --drive 1 --heads 8 --cylinders 320
These runs seemed to go nicely with no errors reported. It has been suggested to me that the mfm_util program can be used to analyze transition files and extract data from them.

Build mfm_util

I clone the files from Github and try "make mfm_util". I get the error that "libiberty.h" is missing. This is an ancient and historical file that the FSF used to achieve portability in their programs. They say modern linux systems supply most or all of what it once addressed.

I comment it out and see what happens. Of course there is also a library to deal with. Getting rid of it will reveal what if any functions are actually getting called.

One function is being called, and it is "buildargv". As much as I am tempted to eradicate this dependency, the quick and lazy approach is to figure out how to install this on Fedora and move on. It is apparently in this package, which I install: 

su
dnf install binutils-devel
exit
make clean ; make mfm_util
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/mfm_util.o mfm_util.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/mfm_decoder.o mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/wd_mfm_decoder.o wd_mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/xebec_mfm_decoder.o xebec_mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/crc_ecc.o crc_ecc.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/msg.o msg.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/parse_cmdline.o parse_cmdline.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/emu_tran_file.o emu_tran_file.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/corvus_mfm_decoder.o corvus_mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/northstar_mfm_decoder.o northstar_mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/analyze.o analyze.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/deltas_read_file.o deltas_read_file.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/drive_file.o drive_file.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/tagged_mfm_decoder.o tagged_mfm_decoder.c
c99  -I inc/ -I /usr/include/libiberty/ -O3 -g -Wall -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -c -o obj/perq_mfm_decoder.o perq_mfm_decoder.c
c99  obj/mfm_util.o obj/mfm_decoder.o obj/wd_mfm_decoder.o obj/xebec_mfm_decoder.o obj/crc_ecc.o obj/msg.o obj/parse_cmdline.o obj/emu_tran_file.o obj/corvus_mfm_decoder.o obj/northstar_mfm_decoder.o obj/analyze.o obj/deltas_read_file.o obj/drive_file.o obj/tagged_mfm_decoder.o obj/perq_mfm_decoder.o  -lm -lrt -liberty -o mfm_util
cp mfm_util /usr/local/bin
It builds clean without warnings and I copy it to /usr/local/bin so I can use it from whatever directory I want.
The source is in /u1/Projects/Callan/Gesswein/Github_mfm/mfm in case I forget and need to go study the source.

How do we use it?

There is no usage message. But there is a PDF file "mfm_read_util_doc.pdf" that is much the same as a man page. After several iterations, I have settled on the following command.
Since I don't like typing long commands over and over when I am doing trial and error iteration, I put my command in a makefile. 
make
mfm_util --analyze --sector_length 256 --sectors 32,0 --heads 8 --cylinders 320 --transitions_file callan_raw1
Original decode arguments: --heads 8 --cylinders 320 --sector_length 512 --retries 50,4 --drive 1
Matches count 34 for controller WD_1006
Header CRC: Polynomial 0x1021 length 16 initial value 0xffff
Sector length 512
Data CRC: Polynomial 0x140a0445 length 32 initial value 0xffffffff
Number of heads 8 number of sectors 17 first sector 0
Interleave (not checked): 0 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8

Command line to read disk:
--format WD_1006 --sectors 17,0 --heads 8 --cylinders 320 --header_crc 0xffff,0x1021,16,0 --data_crc  0xffffffff,0x140a0445,32,6 --sector_length 512 --retries 50,4 --drive 1
make: *** [Makefile:4: run] Error 1
The "WD_1006" format in the analysis is interesting. The controller uses a flock of six small WD chips -- WD1100-01, WD1100-03, WD1100-05, WD1100-06, WD1100-07, WD1100-12.

What I would like to know for sure is how many sectors per track and what the sector size is. The fact that I got about a million messages like this when trying to read the disk made me think it is/was 256 bytes rather than 512, but who can say without more information? 

Expected sector size 512 header says 256 cyl 0 head 0 sector 0
Of course what I want to do ulimately is to extract the sectors from this transition file and have a disk image that I can examine, archive, and pull files from.

The make error alarmed me, making me think this failed, but did it? Maybe not. It gives me a long series of options to try, so let's give it a go. I settle on this command and try it: 

mfm_util --format WD_1006 --sectors 17,0 --heads 8 --cylinders 320 --header_crc 0xffff,0x1021,16,0 --data_crc  0xffffffff,0x140a0445,32,6 --sector_length 512  --extracted_data_file callan.img --transitions_file callan_raw1
I get a billion errors like the following, again suggesting that the sector size may be 256 rather than 512,
then it ends as shown. 
Expected sector size 512 header says 256 cyl 63 head 6 sector 7
Mismatch cyl 319,63 head 6,6 index 8
Expected sector size 512 header says 256 cyl 63 head 6 sector 16
Mismatch cyl 319,63 head 6,6 index 9
....
....
....
Found cyl 0 to 255, head 0 to 7, sector 0 to 16
Expected cyls 320 doesn't match cyls found 256
Expected 43520 sectors got 34814 good sectors, 8704 bad header, 2 bad data
I don't know why it is arguing with me about the number of cylinders. I do get an extracted data file of about the right size (22282240 bytes), but my "acid test" of running "strings" on the file just gives me gibberish like: 
*/T ih ssia l tilt eibazrr,eb tuw ah tsib iegnd no esit  oeresvr e aifex
d    l netg hifle dnit ehs attro  faehcf aremf rot ehI  Da(tuoh)rf eidla dn
    ifllw ah tewd  oon tsu eiwhts apec.sT ka eaceri  foy uhcnaegt ih,sa dn
    olkoc olesls ytat ehc do eniv sdamnic.a dnv sduo.t chttam napiluta
e    t ehv raailbse" tpir"da dn" elin"d

David was actually kind enough to answer email and give a huge amount of help.

It turns out this is just byte swapped (but it is easy to get confused since swapping doesn't necessarily start at the start of words in sentences like this. This is easy to fix by: 

dd if=callan_data1 of=data1_swap conv=swab
Then we see things like this: 
/* these values are for the 40inch plotter (ECP42 machine) at CDI
	running at 200nibs per inch - note that every nib in a line
	must be explicitly defined (no short lines allowed) */
It would seem that the data extraction has actually done just fine, up to cylinder 255.

David points out that after cylinder 255, something unexpected goes on about the format. The upper byte of the cylinder number has been moved to some new location and for cylinder 319, we would have 256 + 63. This is what David concluded from these messages: 

Expected sector size 512 header says 256 cyl 63 head 6 sector 7
Mismatch cyl 319,63 head 6,6 index 8
So it makes sense. It expects cylinder 319, but sees 63. Where did the upper 256 get off to? This disk is going to need a custom format.
Feedback? Questions? Drop me a line!

Tom's Computer Info / tom@mmto.org