LIMS Programs ver 5 =================== agj: 29Apr99 The LIMS programs developed and maintained by Alan Jones consist of two main codes for processing, and ancillary codes. The data must be in TS (time series) format, and for the codes to work correctly, the directory/file structure described below must be followed. -------------------------------------------------------------------------------- Modification history: -------------------------------------------------------------------------------- Directory/file structure ------------------------ The data are assumed to have a specific directory structure, based on a 3-character project-name (e.g. tbt), and a 3-numeric site-number (e.g. 001). The top level database directory has the same name as the project, and within that directory is a separate directory for each site, given by a 6-character code consisting of the project-name + site-number. tbt/ tbt001/ all files for site 001 for project tbt tbt002/ all files for site 002 for project tbt etc. Each time series should be in "ts" format, and should be named with an 8-character name, which is the project-name + site-number + window-identifier where the window-identifier is a 2-character alphanumeric. tbt/ tbt001/ tbt001a1 window a1 tbt001a2 window a2 tbt001b1 window b1 tbt001ss window ss There are two required auxiliary files. One is a windows file that contains information about each window. This is used by a number of codes to speed access and to determine possible remote-reference combinations. The second file contains information about the instrument used, and is called the hardware file. The extension of each file is the project-name. These two files must be in the database main directory, in this case tbt/ -------------------------------------------------------------------------------- In the top-level directory of the database (e.g. tbt in above example), there must be three files for the programs and scripts to work correctly. These are the windows file (e.g., windows.tbt), the hardware file (e.g., hardware.tbt), and the processing configuration file (process.cfg). Windows file format: -------------------- The windows file format is as follows:- 8-character window-identifier space 3-numeric instrument-number space 1-numeric number-of-channels recorded space 19-numeric window start time, in format yyyy-mm-dd hh:mn:ss space or hyphen 19-numeric window end time, in format yyyy-mm-dd hh:mn:ss space 5-numeric digitizing rate in Hz space comment field e.g. windows.tsnobt sno101as 052 5 1996-08-08 21:15:00-1996-08-18 16:10:00 0.200000 sno102as 059 5 1996-08-06 20:00:00-1996-09-04 05:41:00 0.200000 sno103as 064 5 1996-08-08 23:45:00-1996-08-25 19:10:00 0.200000 sno104as 060 5 1996-08-09 20:20:00-1996-08-18 19:48:00 0.200000 sno106as 063 5 1996-08-11 19:00:00-1996-08-21 17:31:00 0.200000 sno107as 062 5 1996-08-12 19:15:00-1996-08-25 20:56:00 0.200000 sno108as 067 5 1996-08-14 00:20:00-1996-08-21 18:58:00 0.200000 sno109as 068 5 1996-08-15 17:20:00-1996-08-21 20:34:00 0.200000 sno110as 057 5 1996-08-16 17:45:00-1996-08-21 15:02:00 0.200000 sno111as 058 5 1996-08-17 17:50:00-1996-08-26 19:41:00 0.200000 sno112as 060 5 1996-08-19 20:15:00-1996-09-07 16:54:00 0.200000 The convention used at the GSC for window-identifiers is that those ending in a numeric are original data, whereas those ending in a character are derived data. The windows file can be built automatically using the program "wtime". If you have LiMS data in .1mp format and convert to TS format using mp2ts, an entry is automatically written into the windows file. Hardware file format: --------------------- The hardware file format is as follows:- 3-numeric instrument-number then one line for each active channel: 1-character channel (H, D, Z, N, E are recognized for MT) space final calibration value space low-pass filter -3 dB point in seconds space high-pass filter -3 dB point in seconds e.g. hardware.tbt 001 'H' 1.0 0.1 0.0 'D' 1.0 0.1 0.0 'Z' 1.0 0.1 0.0 'N' 1.0 0.1 30000.0 'E' 1.0 0.1 30000.0 002 'H' 1.0 0.1 0.0 'D' 1.0 0.1 0.0 'Z' 1.0 0.1 0.0 'N' 1.0 0.1 30000.0 'E' 1.0 0.1 30000.0 This file is not required by tscascade, as no filter corrections are applied for those runs. The corrections are applied to the final cross-power estimates in tsrestack. process configuration file -------------------------- One more file is required to run tscascade and tsrestack. You can either construct this by hand from this example, or have tscascade make it for you when it runs the first time and does not find this file. It is names "process.cfg" and is in the project database directory. Any line that begins with a hash sign (#) is treated as a comment and not read. The active lines contain a 6-character keyword, followed by a parameter. This is an example of the process.cfg file that I use:- # process.cfg: version: v5.0 # # This file contains parameters that may be # altered for running tscascade and tsrestack # Parameters listed below are the default # values in the tscascade executable. # # A hash (#) in the first column comments the line # out. That variable is requested on execution # VER :v5.0 cfg file version NUMSTK:20 number of estimates per substack 64 64 48 48 24 24 16 16 12 12 8 8 8 8 8 8 4 4 4 4 VARTOL:0.0100 tolerance for minimum variance # Other parameters are allowed (remove # to use) OWRITE: T overwrite files without asking PROTYP: MT processing type: MT, GDS or HSG REMOTE: H type of remote: H or E (not requested if absent) #DECLIN: 0 sets declination #ROTANG: 0 sets rotation co-ord system MAG_OR: T correct magnetometer orientation using baselines P_NORM: T normalize spectra by horizontal magnetic field power STACK : T .TRUE. substacking JJANAL: T .TRUE. Jones-Joedicke processing MAXTYP: JKN max type: JKN, COH or VAR NTYPE : 1 param to minimize WEIGHT: T apply weighting COHMIN: 4*0.95 4*0.9 4*0.8 4*0.7 4*0.6 coherence minimum COHMAX: 0.999 coherence maximum UNITS : SI impedance units: SI or FIELD You will be prompted for any values not set in the process.cfg file. Note that if you wish to use a different value for one site compared to another, such as declination or rotation, then start that line with the hash sign (#) to comment it out. -------------------------------------------------------------------------------- tscascade: ---------- Once you have the data file in the correct location, and the hardware and windows file constructed (and optionally the process.cfg configuration files written), then you can run "tscascade". You can set a print level from -1 (no output at all), 0 (questions and a few comments), to 3 (very verbose - for testing and bug finding). Here is an example run for window "tbt101ss", i.e., file tbt101ss.ts in directory tbt101 {1}juandefuca:/em/lims/tbt% tscascade Give print level (-1/0/1/2/3) [default: 0 ] > Opened processing configuration file process.cfg Read in parameters from process.cfg file Remote-reference processing (y/n)? [default: Y ] >N Give window (extension assumed) [default: tbt101ss ] >tbt101ss Opening input file for window tbt101ss Are data formatted (f) or UNFORMATTED (u) [default: u ] > File opened in UNFORMATTED mode=>tbt101/tbt101ss.ts Opened data file =>tbt101/tbt101ss.ts Give the required event start time >yymmddhhmmss Default is [default: 950511090000 ] > Give the event duration >33170600 (ddhhmmss) > Total number of data requested [default: 582552 ] > Co-ordinates of acquisition ==> Hx = 45 Ex = 45 Ey = 135 Rotation angles to be applied ==> Ex = -45.0000 Ey = -45.0000 Hx = -45.0000 Opening cascade file: tbt101/101ss101.cas Opening output file: tbt101/101ss101.out7 number of points read = 25000 number of points read = 50000 number of points read = 75000 number of points read = 100000 number of points read = 575000 Total number of data entries found NPT= 582552 stack7: called with m=-1 - replacing spectra There are two output files. The names of the files are the site-number + window-identifier + remote-site-number. For example, window tbt101ss with local processing will yield output files: 101ss101.out7 ASCII output file with interim response estimates 101ss101.cas cascade file for input into tsrestack If site tbt201 was used as a remote reference, then the files would be: 101ss201.out7 101ss201.cas -------------------------------------------------------------------------------- tsrestack: --------- For processing, you have a choice whether to splice all your windows together then use tscascade on the spliced window, or to process then independently with lismcas7, then restack the cascade estimates together using tsrestack. There will be no difference in the way that periods less than 1/32nd of the window lengths are treated - the cascade estimates will be the same. But for estimates at long periods, splicing will yield more estimates. Once you have performed tscascade on your windows, then you restack the cascade estimates together using "tsrestack". This program also uses the process configuration file "process.cfg". If the parameters are set in the configuration file, then very few questions are asked. Here is an example run of tsrestack: {297}juandefuca:/em/jones/tmp/tmp% tsrestack Set error message print level (-1/0/1/2) [default: 0 ] > Opened processing configuration file process.cfg Read in parameters from process.cfg file Give station [default: nod001 ] >tbt101 Error ***** ==> file tbt101/tbt101.out already exists. Being overwritten Remote-reference processing (y/n)? [default: Y ] >N Give input file, * for all .cas [default: * ] > Cascade file with local data >tbt101/101ss101.cas angle= 0 All data read in; proceeding with analysis Cascade data co-ordinate system is 0.0 degrees from TRUE NORTH Spectra to be rotated by angle = 0.0 All estimates in co-ordinate system of angle = 0.0 N, period, nacpt, ndeg = 1 20.0000 1223 12888 N, period, nacpt, ndeg = 2 26.6667 1454 15700 N, period, nacpt, ndeg = 3 40.0000 843 9340 N, period, nacpt, ndeg = 4 53.3333 898 10182 N, period, nacpt, ndeg = 5 80.0000 455 5272 N, period, nacpt, ndeg = 6 106.667 450 5244 N, period, nacpt, ndeg = 7 160.000 234 2742 N, period, nacpt, ndeg = 8 213.333 232 2668 N, period, nacpt, ndeg = 9 320.000 234 2688 N, period, nacpt, ndeg = 10 426.667 213 2434 N, period, nacpt, ndeg = 11 640.000 122 1392 N, period, nacpt, ndeg = 12 853.333 109 1256 N, period, nacpt, ndeg = 13 1280.00 54 586 N, period, nacpt, ndeg = 14 1706.67 56 614 N, period, nacpt, ndeg = 15 2560.00 26 274 N, period, nacpt, ndeg = 16 3413.33 25 294 N, period, nacpt, ndeg = 17 5120.00 15 166 N, period, nacpt, ndeg = 18 6826.67 12 132 N, period, nacpt, ndeg = 19 10240.00 5 58 N, period, nacpt, ndeg = 20 13653.3 5 46 Number of accepted periods = 20 Note that the MT and GDS estimates are derived independently by using the jackknife-based iterative rejection scheme. So that there will be two spectra estimate files, and the GDS estimates derived from the MT spectra will not be the same as those from the GDS optimized spectra. There are a number of output files generated. All have the file name project-name + site-number, with extension to show their contents. tbt101.out output file with all estimates etc. tbt101.xl spectra file for MT estimation optimization tbt101.xlg spectra file for GDS estimation optimization tbt101l.dat J-format MT & GDS estimates and jackknife errors The ".xl" and ".xlg" denotes "local" processing. If the data had been remote-reference processed, then the file names would be ".xr" and ".xrg". -------------------------------------------------------------------------------- x2edi ----- The spectra file can be converted into an EDI file using "x2edi". The first time you run this, it sets up its own configuration file (x2edi.cfg). Subsequently, you are not asked any questions about the survey, only the site number. -------------------------------------------------------------------------------- j2edi ----- The data file can be converted into an EDI file using "j2edi" The first time you run this, it sets up its own configuration file (j2edi.cfg). Subsequently, you are not asked any questions about the survey, only the site number.