Common Programming Errors

QUESTION: I was curious to know what other IDL professionals thought were the most common IDL programming errors, so I asked them. Here is their list, in no particular order, other than massaged slightly to accommodate my experience in teaching IDL programming classes. I have to admit, I have never run into some of these “common ”problems in nearly 20 years of working with IDL. I'm either very lucky or the word common was somehow conflated with annoying in the minds of the IDL professionals I asked. But I leave that for you to decide.

IDL Programs Named Incorrectly

By far the most common IDL error I see are IDL programs that are named incorrectly. If you have to compile an IDL program first to get it to run, it is named incorrectly. Think of IDL programs as IDL commands that you are constructing. The IDL module (procedure or function) that corresponds to the command name should be the last module in the file. If there are other modules in the file, they should be placed in front (toward the top of the file) of the command module and should be utility programs for the command module, not commands of their own. The names of these utility modules should reflect their utility status by having the command name prepended to their names. If a utility module gains status and importance beyond its modest service to the command module, it should be removed from the command module file, renamed, and promoted to a command module. The name of the file should be exactly the same as the name of the command module, and should be spelled with all lowercase letters, since that is what IDL looks for when it is auto-compiling code on all platforms.

KEYWORD_SET Used For Non-Binary Keywords

Programmers often want to know if a keyword was used in their program or not. Unfortunately, determining if a keyword is used is extremely difficult and I have never seen it done in IDL programs. Rather, we usually determine if a keyword variable is undefined or not. The IDL routine used to do this is N_ELEMENTS, which returns a zero if its argument is undefined. KEYWORD_SET returns a Boolean value of 0, if its argument is 0 or undefined, or a 1 if its argument is anything else at all. If KEYWORD_SET is used to check whether a keyword variable is used or not, it becomes impossible to set that keyword variable value to zero.

Hit-or-Miss Graphics in IDL Widget Programs

If you write non-blocking IDL widget programs, it is essential that you remember there is only one current graphics window in your IDL session. If you issue a graphics command, it will show up in the current graphics window. A great many IDL widget programmers forget this, and find their graphics commands showing up in the strangest places. Be sure you always set the current graphics window (with WSET) to the right window in any event handler that will issue graphics commands. The graphics index number of the “right window” can be found by asking the draw widget for its value after the draw widget has been realized on the display. Normally, this value is stored in the info structure along with all the other information required to run the widget program.

Problems Caused by Short Integers

IDL is old. Grandfatherly, even. When it was first written, you had a really souped-up computer if you had added an additional 10 MByte hard drive to go with the standard 1 MByte of disk space. So, you can maybe understand why the default integer size in IDL is 16-bits, or two bytes. This is a short integer by any current standards. This causes problems because a 16-bit signed integer can only represent the values -32768 to 32768. Occasionally, you need an integer larger than that, and even throwing in fingers and toes doesn't do the job.

One place where it causes problems is when short integers are used as counters in FOR loops.

     FOR j=0,n DO ... 

If you want your loop to go beyond 32768 (loop values will become negative it they go beyond their bounds), you will have to make your loop variable a long integer, like this.

    FOR j=0L,n DO .... 

Another place where short integers can cause confusion is in mathematical expressions. Here is an example.

    IDL> Print, 30000 + 30000          -5536

In this case, you would need to make at least one of these integers a long integer so that IDL could promote the expression to a long integer.

   IDL> Print, 30000 + 30000L          60000 

A better way to make your short default integers into long integers is to use the IDL compiler option DEFINT32. Put the following piece of IDL code into your IDL start-up file, so all default integers at the IDL command line will be long integers, and into every IDL program module you write, usually as the first line of code after the procedure or function definition statement. It will force any default integer, created in a location where the compile option is in effect, to be a long integer and not a short integer.

   Compile_Opt DEFINT32 

In practice, most IDL programmers prefer both this option and the option of enforcing square bracket subsetting for IDL arrays (Compile_Opt STRICTARR). The shorthand way of specifying both of these options is like this.

    Compile_Opt idl2 

Integer Arithmetic

IDL promotes the expression on the left-hand side of an equal sign to the the size and type of the expression on the right-hand side in a way that preserves the most accuracy for the expression. Normally, this does exactly what you want it to do. But you can have problems when you are dealing with integers. Consider this expression.

    IDL> variable = 3 / 2    IDL> Help, variable
        VARIABLE     INT    =     1 

This may or may not be what you had in mind. If you expected a floating point number, you will have to promote one of the values in your expression to a float, either by using a decimal point in the expression, or my explicitly casting the value with the FLOAT function.

   IDL> variable = 3. / 2  ;; Or, variable = Float(3) / 2
        VARIABLE     FLOAT  =    1.50000 

Integer division is a tough problem to track down in a program, so be extra careful when you are working with integer data.

Excessive Use of FOR Loops

Ok, an argument can be made that IDL is a loopy language, but it is decidedly not a looping language. It is an array processing language. There are those who say FOR loops are the embodiment of pure evil and those who say they are not so bad. But, whatever your religious conviction, you do not want to write code like this.

   s = Size(array, /DIMENSIONS)    newArray = FltArr(s[0],s[1])
   FOR j=0, s[0]-1 DO BEGIN       FOR k=0,s[1]-1 DO BEGIN
         newArray[j,k] = Sin(array[j,k])       ENDFOR    ENDFOR 

This kind of code will always be extremely slow. And extremely unnecessary. You can replace all code like that shown above with code like this.

   newArray = Sin(array) 

Failure to Check WHERE Count

The WHERE function in IDL is generally considered one of the best and most useful of all the IDL functions. But, occasionally, and when you least expect it, the WHERE function will not find anything true in its Boolean search. When that happens, the WHERE function returns a -1. If you don't check for that, and you use the return value as a index into another array, then your program causes an error.

    IDL> array_1 = [3, 4, 5]    IDL> array_2 = [6, 7, 8]
   IDL> indices = Where(array_1 EQ 2)
   IDL> array_2[indices] = array_1[indices]
   Attempt to subscript ARRAY_1 with INDICES is out of range. 

The proper way to write this code is to use the count parameter to WHERE to find out how many indices WHERE found that met the criteria.

   IDL> array_1 = [3, 4, 5]    IDL> array_2 = [6, 7, 8]
   IDL> indices = Where(array_1 EQ 2, count)
   IDL> IF count GT 0 THEN array_2[indices] = array_1[indices] 

What is Black and White, and Red All Over?

No, not a newspaper. The answer is an IDL programmer who forgets that the default color mode in IDL is to use decomposed color. In this mode, any 2D image is displayed in gray scale, and any IDL graphic that tries to use color indices to specify colors comes out red.

There are two solutions. Set IDL up to use indexed color mode (Device, DECOMPOSED=0) and become a 1970s Luddite with respect to color and expensive 24-bit graphics cards, or learn how to write your IDL programs without thinking about the current color mode by using programs that are color-mode aware.

Getting PostScript Color Output

It is sometimes difficult to get color PostScript output. Mostly this is a problem with the PostScript device not being set up correctly. You must configure the PostScript device with both the COLOR and BITS_PER_PIXEL keywords to get accurate color output.

   Set_Plot, 'PS'    Device, COLOR=1, BITS_PER_PIXEL=8 

Also, be aware that, until recently, the PostScript device was an 8-bit device that uses indexed color mode. Moreover, it flips the foreground and background colors so it will draw black on white, rather than the default IDL graphics window colors of white on black. (Saves toner this way!). If you have changed the normal color table in IDL graphics windows, you can sometimes be drawing white-on-white in your PostScript file, which is hard for journal editors to see. For this reason, I strongly recommend that you not specify drawing colors on either end of the color table. That is, do not use index 0 of 255 to specify your drawing colors. Use any other index and you will probably be fine. See the Coyote Plot Gallery for examples of how to create color PostScript files automatically from your graphics programs.

It probably goes without saying, but I will say it nonetheless, that you need a color PostScript printer to print color PostScript output. You would be surprised how often this little detail is overlooked.

Landscape PostScript Output

It is not uncommon, when switching from Portrait to Landscape mode in PostScript output, to have your graphics output completely disappear. This is because...Oh, heck, it's too hard to explain. Just read the article. And I would investigate IDL tools that that act more intuitively.

Problems with CATCH Error Handling

There are two common problems with CATCH error handling. The first, and the most pernicious, is to use silent CATCH error handlers.

    Catch, theError
   IF theError NE 0 THEN BEGIN       RETURN    ENDIF 

This is not usually a problem in your code, of course, but in code you have been recruited to support that was written by someone else. The only solution is to hunt down the mangy excuse of a dog who wrote the code and summarily execute him. Of course, you will have to do this after spending days and weeks trying to find out what the hell is going on in the program! This is especially aggravating when you find it in ITTVIS-supplied code in the IDL lib directory. At the very least, let the user know something has happened.

    Catch, theError
   IF theError NE 0 THEN BEGIN
      void = Dialog_Message('I discovered an error I prefer not to tell you about.')
      RETURN    ENDIF 

The second problem, IDL going into an infinite loop, occurs when--heaven forbid--you have errors in your error handing code and you haven't thought to cancel the CATCH in your error handling code. Now, I understand you are an expert typist, but a recent study has determined that on a national average, programmers type about 1.7 consecutive letters in a word correctly. What I am saying is, there is a pretty good chance you might have an error in your error handling code. To save wear and tear on your computer, you just might think of cancelling the CATCH as the very first line in the CATCH error handler.

    Catch, theError    IF theError NE 0 THEN BEGIN        Catch, /CANCEL
       Prent, 'I in the errrof hamdler'        RETURN    ENDIF 

I think you will find this gives you a lot of peace of mind.

Getting Rows Mixed up with Columns in Matrix Operations

IDL's arrays are specified as [column, row]. Linear algebra's arrays are specified as [row, column]. IDL's # matrix multiply operator computes array elements by multiplying the columns of the first array by the rows of the second array. The ## matrix multiply operator computes array elements by multiplying the rows of the first array by the columns of the second array. In linear algebra a matrix multiply computes array elements by multiplying the rows of the first array by the columns of the second array. Ergo, to make IDL arrays act like arrays in linear algebra, use the ## operator to perform array multiplication. But in practice, confusion just reigns supreme. There is no solution, as far as I know, beyond total befuddlement. My advice it to check your code at least three times with known quantities before you show it to your boss. You don't want this satellite ending up in the Indian Ocean instead of in orbit.

Calling a Function as a Procedure, and Visa Versa

There are three types of IDL program modules: a main program, a procedure, and a function. A main program is neither a procedure or a function, rather it is just a bunch of IDL commands, followed by an END statement that can be compiled and run (usually with the .RUN command in IDL). Consider an IDL program module with three IDL statements added to a file and named mystatements.pro.

    statement_1    statement_2    statement_3    END 
   IDL> .Run mystatments 

An IDL procedure, requires an IDL procedure definition statement in which various positional and keyword arguments are defined.

 
   PRO MYPROCEDURE, arg1, arg2, KEYWORD1=keyword1, KEYWORD2=keyword2
      statement_1       statement_2       statement_3    END 

Procedures are called in IDL by specifying the name of the procedure and by specifying the positional and keyword arguments to the procedure to the right of the procedure name, and separated by commas.

   IDL> MyProcedure, coyote, deer, KEYWORD1=puma, KEYWORD2=elk 

In IDL functions are nearly identical to procedures, except they are defined with a function definition statement, and the must return one IDL variable (of any type). A function without a specific RETURN statement returning an IDL variable will return a 0.

     FUNCTION MYFUNCTION, arg1, arg2, KEYWORD1=keyword1, KEYWORD2=keyword2
      statement_1       statement_2       statement_3
      RETURN, arg1 * arg2 / (keyword1 / keyword2)    END 

When you call a function in IDL, you must put the positional and keyword arguments inside of parentheses and you must provide a receptacle (IDL variable) to hold the return value.

    animalMishMash = MyFunction(coyote, deer, KEYWORD1=puma, KEYWORD2=elk)

Sometimes, you will find IDL procedures and functions combined into one IDL expression.

    IDL> TV, Bytscl(image) 

Here, the return value of the BYTSCL function is used as the first positional parameter of the TV procedure.

Using Keyword Shortcuts Incorrectly

Some keyword are binary. They are either on or off, yes or no, true or false, etc. These keywords have values of 0 or 1. A shorthand way of specifying KEYWORD=1 is to type /KEYWORD. Or, another way of saying this, is that /KEYWORD means KEYWORD=1. It is incorrect, however, to type a slash in front of a keyword and try to set its value at the same time. This kind of syntax will cause a syntax error in IDL.

   IDL> scaledImage = BytScl(image, /TOP=200)
        scaledImage = BytScl(image, /TOP=200)
                                           ^         Syntax error. 

Inability to Find the ? Key on the Keyboard

It is hard to believe in this day and age, but a great many people cannot find the ? key on their keyboards. If they could, they would find that nearly every document the good folks at ITTVIS ever wrote about IDL is immediately (well, relatively speaking in the case of IDL 7) available to them in the form of on-line help. The IDL Reference Guide lists every IDL command that is distributed with IDL and includes its syntax, and the arguments and keywords that are defined for that command.

    IDL> ? 

Now, it is possible that you might not always understand what the documentation is telling you (and if this is the case, there is always the IDL newsgroup, whose members are happy to enlighten you about that), but this is always the place to start when you have a question about IDL. If you are using IDL commands that are not distributed with IDL, but come from one of the known IDL libraries, open up the command file itself. There will almost certainly be an IDL header in the program file that describes how to use the IDL program.

Rolling Your Own Over and Over Again

There are probably a large, but finite, number of things you can realistically use IDL to do. And a great many of these things have already been discovered in IDL's 20 year life time. So, it is not true that you have to start from scratch every time you want to build a program in IDL. Many of these programs are available for the asking. In fact, a great many of them exist in your current IDL distribution, if you just knew about them. And a small percentage of them have even been written using the Principles of the IDL Way, which means they don't have the three nested FOR loops that your program utilizes. How can you find out about them?

First, use the on-line help (type ? at the IDL prompt). The Functional List of IDL Routines is particularly helpful in identifying IDL routines you may already have on your computer.

Second, check around. There are dozens of good IDL web pages on the Internet. Many of these provide IDL code libraries that you can use. Two of the best are the Johns Hopkins University Applied Physics Lab IDL web page and the IDL Astronomy User's Library at NASA Goddard Space Flight Center. The Coyote Library, I would suggest, is another place to get well-documented and general-purpose IDL code. There are other, more specialized, sites as well. Ten minutes spent looking can save you hours of IDL programming time.

Third, ask the experts on the IDL Newsgroup for help. There is no better place than this friendly, helpful newsgroup to get your IDL questions answered quickly and to learn more about IDL. But don't waste the expert's time by not doing your homework first. Come prepared and curious, and you will find your IDL understanding growing exponentially. It's always a good idea to check the Google archives to see if your question has already been answered before you ask the question again. (See the Search feature in the link above.)

Fourth, Coyote's Guide to IDL Programming serves as the de facto IDL FAQ for most IDL users. Often, a search of this site will turn up an answer to your IDL questions. Or, if you ask the site coordinator politely, he might be interested enough in your question to write an article about it.

Fifth, buy a good book about IDL. There are a number available now, including my own. Any one of them will put IDL in context for you, allowing you to concentrate on the things you really need to know while you are learning IDL, and leaving aside for later the more arcane topics that you might still need to learn on your own.

Closing Files Incorrectly

There are two pools of numbers, from which you take get logical unit numbers to read and write files in IDL. The larger pool goes from 1 to 99. You select a logical unit number out of this pool to open a file, for example, then close the logical unit number with the CLOSE command.

   OpenR, 5, 'myfile.dat'    ReadF, 5, data    Close, 5 

There is another pool of numbers, 100-128, that IDL manages for you. You select a number out of this pool by using the GET_LUN command or by setting the GET_LUN keyword on an OPEN statement. These logical unit numbers are closed and made available for re-use with the FREE_LUN command. (There is no need to use the CLOSE command with these logical unit numbers, although it doesn't hurt anything.)

    OpenR, lun, 'myfile.dat', /GET_LUN
   ReadF, lun, data    Free_Lun, lun 

But it is a mistake to use the GET_LUN pool and use CLOSE to close the logical unit number. While it is true, the logical unit number is closed, it is not released under these conditions and over time you will run out of logical unit numbers in your IDL session.

    OpenR, lun, 'myfile.dat', /GET_LUN    ReadF, lun, data
   Close, lun ;;;;; MISTAKE! File is closed but not released.

Leaking Memory

Pointers and objects are heap variables. That is, they are variables that are stored on the heap, an area in global memory. Heap variables are persistent in the IDL session. That means they exist until you explictly delete them or you exit the IDL session. When you create a heap variable, you are returned (via the heap variable creation function) a reference to that variable. (This is called a pointer or object reference, such as ptr in the code below.)

   ptr = Ptr_New(image) 

The heap reference is like any other IDL variable, and since IDL is a weakly typed language, you can reassign the heap reference to be something else entirely.

   ptr = 5 

If you do this, you will no longer be able to access the memory allocated on the heap, and that memory will no longer be available for the IDL process to use. We say that memory has leaked away. That is a bad thing, and we try to avoid it by explicitly deleting our heap variables when we are done with them, using either PTR_FREE or OBJ_DESTROY, depending on whether the heap variable is a pointer or an object.

A common way to create leaking memory is to create a pointer inside an IDL procedure or function, and then fail to explicitly delete it before you exit the procedure or function. In this case, the pointer reference is a local variable, which is cleaned up (deleted, essentially) when the procedure or function is exited. But the memory that local variable pointed to on the heap is not cleaned up, and will leak away from your IDL process. Be absolutely sure you destroy your pointers and functions when you are finished with them.

    ptr = Ptr_New(image)    ...
   Ptr_Free, ptr 

A good way to tell if you are leaking memory is to exit all your programs and then look at the heap. If there are things still on the heap, you are leaking memory! From personal experience, I recommend you do this before showing your code to your boss or posting programs to the IDL newsgroup to avoid unnecessary embarrassment.

    IDL> Help, /HEAP 

Infrequent Update of Libraries

The number one problem with third-party IDL libraries is that people won't update them! Every third-party IDL library I know is a dynamic library. It grows, it changes, bugs are fixed, and functionality is added. Eighty percent of the problems reported to me concerning Coyote Library routines are solved by updating the Coyote Library. If your problem is with a third-party library, the first thing you should do is update your library. Be sure you restart IDL after you do, and before you check the program again.

Version of IDL used to prepare this article: IDL 7.0.1.