FB II Compiler







Disk I/O














Made with FB


Follow a roadmap for a watched folder

It has been my experience that many processes that create/write files to a folder do so in a series of chunks spread out over time. That is, a file is opened, appended to, and closed several times rather than being written in one single continuous open, write, close operation. Also files being written over a network are a different kind of beast that a file being written locally.

What I do:

Before attempting to process, the user must identify:

a) the Source folder.
Create a done folder (where successfuly processed source files are placed after processing.)
Create an error folder (where unsuccessfuly processed source files are placed after a processing attempt.)

b) the target folder
(where successfuly processed output files are placed after processing.)
(I keep an alias to these folders and validity check them on startup.)

Step #1:
Is the files to be processed list empty? If so, make a list of files in the source folder. Keep track of the file list creation dateTime. Wait x seconds before doing anything. This wait allows sometimes slow network writes to complete. If not, use the first file ('selected file') on the list.

Step #2:
Does the selected file meet rudimentary filetype/creator/visibility/ignore processing criteria?
If not, remove from the list.

Step #3:
Is the file busy? If so, remove it from the list. Step #4:
Check the file's total size (resource plus data fork) until it has been stable (remained the same) for n seconds.
If not, remove it from the list.
If stable, wait a further y seconds. (x, n and y are user defineable)

Step #5:
Perform additional more rigorous checks on the file to see if it meets processing criteria. Files from non-mac systems may have unexpected file types or creator type. This usually involves opening the file's data fork and checking for expected values at known offsets.
Remove the file from the list, if it does not pass muster.(See Note A) Step #6:
Is the file busy? If so, remove it from the list. Step #7:
Process the file. (See Note A)

Step #8:
Does the file already exist in the done, error or target folders?
If so, uniquely rename the source and target files before placement in any of the target, done or error folder. (See Note A)

Step #9:
Write the processed file to the target folder.

Step #10:
If Step#9 successful, move or copy (as appropriate) the source file to the done folder or delete according to a user setting.
Otherwise move the source file to the error folder (See Note A)

Step #11:
Go to step 1

Note A:
If a system file error is generated, chances are 9 out of 10 that the file was not ready to be processed, remove the file from the list. If a particular file has triggered many system file errors (usually means that for some reason moving the file to the error folder has failed many times) put it on a permanent ignore list. (A placement on the permanent ignore list times out after 24 hours.)

Each of the steps (and often sub-steps) is done only after receiving a user-setable number of null events. This allows other applications to run robustly on the same machine but, in my experience, a dedicated machine works best...

This is not absolutely everything that I do but I believe it covers all of the important operations. I hope this helps and I look forward to any comments...

Michael Evans