I began using the NGS TOPO! program running it under wine on my linux machine. I grew frustrated with various aspects of that and began writing my own viewer, the gtopo program. Along the way, it was necessary to comprehend the file layout and file format used by NGS TOPO!. This document primarily describes the file layout, rather than the format of the files themselves. To understand the latter, the best approach at present is to look at the gtopo source code. If people keep asking for it, I may someday write a description of the TPQ file format.
My work began with the Arizona map set (TOPO version 2.6.9, circa 2001), which was later augmented by the California set (TOPO version 2.7.6, circa 2002) and then by the Nevada (TOPO version 4.2, circa 2006), Utah, and New Mexico sets.
The first thing to understand is that TOPO! handles 5 scales of maps as follows:
What I do is to copy the contents of the TOPO! cdroms to a nice sized part of my hard drive (in my case in /u1/topo). I have resisted the temptation to reorganize the information in any way. I keep each CD image in a directory with a name like AZ_D01. The gtopo program just needs to be told that the archive is in /u1/topo and it builds its own datastructures without making any changes in the archive. (This should make it easy to run gtopo off of actual CDROMS).
When I was running TOPO! under wine, I would make a symbolic link from d: to this partition in ~/.wine/dosdevices like so:
d: -> /u1/topo
Here is what the /u1/topo directory looks like with just the 8 Arizona CD's loaded up:
dr-xr-xr-x 10 tom tom 4096 Sep 26 2002 AZ_D01 dr-xr-xr-x 17 tom tom 4096 Sep 26 2002 AZ_D02 dr-xr-xr-x 15 tom tom 4096 Sep 26 2002 AZ_D03 dr-xr-xr-x 15 tom tom 4096 Sep 26 2002 AZ_D04 dr-xr-xr-x 13 tom tom 4096 Sep 26 2002 AZ_D05 dr-xr-xr-x 13 tom tom 4096 Sep 26 2002 AZ_D06 dr-xr-xr-x 16 tom tom 4096 Sep 26 2002 AZ_D07 dr-xr-xr-x 15 tom tom 4096 Jul 24 2001 AZ_D08
We see a group of directories, one from each of the CD's that gets shipped with TOPO. The first disk (AZ_D01) holds levels 1 through 4, but no information at level 5. The other disks have subdirectories that each hold information for a 1x1 degree area.
If we look at the AZ_D02 directory, we see:
dr-xr-xr-x 3 tom tom 4096 Sep 26 2002 AZ1_GNIS dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 AZ1_MAP1 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 AZ1_MAP2 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 AZ1_MAP3 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 AZ1_OVER dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D34113 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D34114 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D35113 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D35114 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D36113 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D36114 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D37113 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 D37114 dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 INFO dr-xr-xr-x 2 tom tom 4096 Sep 26 2002 LEGENDS
The main thing of interest are the D directories, which hold information for levels 4 and 5. The directory AZ_D02/D34113 holds information for the area from latitude 34 to 35 degrees north and longitude 113 to 114 west. If we are not at the edge of the state (where some stuff will be missing) there will be 66 files in this directory as follows:
K34113A1.TPQ K34113E1.TPQ Q34113A1.TPQ ... Q34113H8.TPQ
There are 64 "Q" files, corresponding to the 64 7.5 minute quadrangles contained in this 1x1 degree area. The last two characters of the filename (the "A1" "E1" or "H8") tell you which quadrangle. A thru H moves from south to north and 1 thru 8 moves from east to west. In other words A1 is in the extreme southeast, and H8 is in the extreme northwest. A2 is directly west of A1 and B1 is directly above A1. These 64 files give the 7.5 minute 1:24,000 data.
The K files hold information for level 4. This is identical to the information collected for the entire state on the first disk. There are two of these files per square degree. The A1 file covers the south half and the E1 file the north half.
Some references that were handy in the next stage:
At some point in time, certain NGS TOPO! distributions actually consisted of data in "maplets" which were literal GIF files. The PyTopo program is an open source program written in python that will display these files. GIF files are limited to 256 colors per file (from a 24 bit color palette).
The latest versions of TOPO boast that the scanned files have "millions of colors" which would tell us that they are no longer stored in a GIF format, and indeed the fact that the TPQ file holds JPEG format fragments is consistent with this. The NGS also says that they use NED (National Elevation Dataset) information at 1-arc-second resolution (approximately 30 meter resolution.
What does a .TPQ file look like? Well, there is a header, followed by an index to "maplets" which follow (which are stored as a bunch of JPEG images concatenated end to end). These are in lossless JPEG format. For a .TPQ file representing a 7.5 minute quadrangle there are 50 of these JPEG "maplets". These make up the bulk of the file. At the end of the file are several PNG images that I have yet to make any sense out of, they may have the information for the shaded relief overlay.
The 50 JPEG files cover a 7.5 minute sheet with 5 files east to west and 10 files north to south. They are ordered as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 10 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
For the O'Neill Hills quadrangle (q32113a3.tpq) these are all 436 by 256 pixel files. This quadrangle is in extreme southwest Arizona at latitude 32 north. For the North Palisade quadrangle q37118a5.tpq) these are all 410 by 256 pixel files. This quadrangle is in east central California at latitude 37 north.
For O'Neill Hills, we have 2560 pixels N-S and 2180 E-W.
For North Palisade, we have 2560 pixels N-S and 2050 E-W.
At this point you should refer to my
discussion of map scales.
At a scale of 1:24,000 the N-S size of a 7.5 minute map
should be 22.82 inches (covering 45,640 feet on the ground).
So each pixel covers 17.83 feet N-S. For North Palisade
the E-W map size is 36449.7 feet on the ground, so each
pixel covers 17.78 feet E-W. Similarly for
O'Neill Hills, the E-W map size is 38705.9 feet on the
ground, so each pixel covers 17.75 feet E-W. All of this
makes sense and yields square pixels in terms of ground
coverage.
What if we want to paint these pixels back on paper and get the original 1:24000 scale? 1:24,000 is a scale of 1 inch = 2000 feet. With 17.8 feet per pixel, we need 112 pixels per inch for this to print right (a bit odd, but we seem stuck with it).
Uncle Tom's Computer Info / tom@mmto.org