Findfile Oddity in UNIX IDL

QUESTION: I tried to use Findfile to identify all the files in my directory, but I notice that it only returns a subset of the files matching the criteria. It seems as though there is an upper limit to the number of files it can return of about 3500. What is going on?

ANSWER: This is a limitation that has been noticed by UNIX users of IDL through IDL 5.2. Several people have written workaround programs and have offered them to IDL newsgroup readers.

Here is a description of a program written original by Stein Vidar Haugen at the University of Oslo and included in the SOHO library of IDL routines maintained at Goddard Space Flight Center and offered by Bill Thompson (thompson@orpheus.nascom.nasa.gov).

The builtin FINDFILE() function has problems on some unixes whenever a lot of files are matching the file specification. This is due to the fact that filename expansion is done by the shell before interpreting a command. Too many files cause too long commands, which are not accepted. This causes FINDFILE() to return an empty list of candidates.

FIND_FILE tries the builtin function first, and whenever the returned list of files is empty, it tries to recheck through spawning a "find" command.

Since FINDFILE doesn't discriminate between directories, links and files, this function will not do this either.

Under UNIX, however, calls like FINDFILE("*") returns the unfiltered output of the shell commmand "ls *", including colon-terminated lines for each subdirectory matching the specification and empty lines separating each subdirectory listing. Such silly effects are not implemented in the "find" version. Be warned, however, that these effects are present when the builtin function does not "fail" due to a too long file list.

It is possible (under UNIX) to use the "find" method as default by setting the keyword /USEFIND (no effect under other operating systems).

You can download the version of Find_File that was offered to the IDL newsgroup here.

David Foster also solved this problem and a program with the same name Find_File can be found on his anonymous ftp site.

[Editor's Note: As of IDL 5.5, RSI recommends using File_Search instead of Findfile. File_Search is apparently more platform-independent and provides greater flexibility and functionality.]