Repeating Formats and Brackets

QUESTION: I have a very simple file to read, but I am having great difficulty. Can you help?

Here is the data file I wish to read:

|020922 10:00:00    |   339|   348|    63|   356|   400|    88|
|020922 11:00:00    |   345|   349|   111|   422|   482|   140|
|020922 12:00:00    |   339|   340|   113|   431|   533|   137|
|020922 13:00:00    |   335|   333|    99|   410|   492|   127|

And here is the simple IDL program I use to read it:

   FUNCTION FileReadTest

   filename = 'data.txt'
   data= INTARR(10, 4)

   OPENR, 1, filename
   READF, 1, data, FORMAT='(1x,i6,3(1x,i2),4x,6(1x,i6))'
   CLOSE,1
   RETURN, data
   END

But when I run the program, I get this result:

IDL> data = FileReadTest()
% READF: End of file encountered. Unit: 1, File: C:\IDL\research\testdata.txt
% Execution halted at: FILEREADTEST        7 C:\IDL\research\testdata.pro
%                      FILEREADTEST        6 C:\IDL\research\testdata.pro
%                      $MAIN$

And if I print the data out at the point where I crash, I find this result:

IDL> Print, data
   20922      10       0       0     339     348      63     356     400      88
   20922      11       0     345     349     111   20922      12       0     339
     340     113   20922      13       0     335     333      99       0       0
       0       0       0       0       0       0       0       0       0       0

Does that make any sense at all to you?

ANSWER: Uh, no, not to me. :-(

But, fortunately for you, it does make sense to Christoper Lee who put us all on the right track with his answer to this strange problem in the IDL newsgroup. As it happens, this is a well-known (for FORTRAN programmers, anyway) behavior of the way brackets or parentheses are handled in format statements.

Here is Christoper's answer:

The answer to this lies with our old friend Google....
http://www.hef.kun.nl/~pfk/teaching/fortran/ftn-AB.html

"The rule for rescanning formats is complicated by the presence of brackets surrounding groups of format codes. The full rule for rescan is as follows. If there are no nested brackets, rescan returns to the beginning of the format. Otherwise, the rescan returns to the left bracket which matches the right bracket nearest to the closing bracket of the format. If this left bracket is preceded by a repeat count, this count takes effect during the rescan."

so...
IDL> Print, findgen(6), FORMAT='(i," A",i," B",i," C")'
           0 A           1 B           2 C
           3 A           4 B           5 C
IDL> Print, findgen(6), FORMAT='(i," A",(i," B"),i," C")'
           0 A           1 B           2 C
           3 B           4 C
           5 B
If you want the read to include the terms before the first real brackets in a repeat, you need an extra set of opening brackets.

So, the solution to this problem is to write your FORMAT statement with two extra parentheses or brackets, like this:

   FUNCTION FileReadTest

   filename = 'data.txt'
   data= INTARR(10, 4)

   OPENR, 1, filename
   READF, 1, data, FORMAT='( (1x,i6,3(1x,i2),4x,6(1x,i6)) )'
   CLOSE,1
   RETURN, data
   END

Extra Brackets Apply to Output, Too

Note that extra brackets or parentheses apply to formatted output, too. Here is a recent question on the IDL newsgroup.

QUESTION: I have an array with 101 columns and want to output the first column in integer format and the next 100 columns in float format. I've tried something like this.

   openw, 1, 'test.txt'
   printf, 1, array, FORMAT='(I0,100(2X,:,F0))'
   close, 1

However, because I used the group specification, format control reverts to the group repeat specification whose opening parenthesis matches the next-to-last closing parenthesis of the format string. So, the data are output to float format except the first line. Would you please give me some advice for fixing this problem?

ANSWER: Throw in an extra set of parentheses.

   printf, 1, X, FORMAT='((I0,100(2X,:,F0)))'