Batch For loop doesn't refresh the file it's pulling from

Question

So I have a for loop that does an iteration of a SQL stored procedure for every line in a file queue.txt, now that all works great, what DOESNT however is that if i

Answer 1

You are absolutely correct - A FOR /F loop waits for the command in the IN() clause to finish and buffers the result prior to processing the 1st line. The same is true if you read from a file within the IN() clause instead of executing a command.

Your proposed strategy of counting the number of lines in the queue prior to the FOR loop, and then recounting after the FOR loop has completed could just about work if you stop mucking with the queue contents within the FOR loop. If the final count is greater than the original you could GOTO a :label before the FOR loop and skip the original line count in the FOR loop so you only process the appended lines. But you would still have a concurrency issue if a process writes to the queue while you are getting the line count or if it appends to the queue after you get the final count but before you delete the queue.

There are ways to serialize events within batch when dealing with multiple processes. The key to doing this is to take advantage of the fact that only one process can have a file open for write access.

Code like the following can be used to establish an exclusive "lock". As long as every process uses the same logic, you can guarantee you have exclusive control over one or more file system objects until you release the lock by exiting the block of code.

:getLock
2>nul (
  >lockName.lock (
    rem ::You now have an exclusive lock while you remain in this block of code
    rem ::You can safely count the number of lines in a queue file,
    rem ::or append lines to the queue file at this time.
  )
)||goto :getLock

I demonstrated how this could work at Re: parallel process with batch. After pressing the link, scroll up to see the original question. It seems like a very similar problem to yours.

You might want to consider using a folder as a queue instead of a file. Each unit of work can be it's own file within the folder. You can use a lock to safely increment a sequence number in a file to be used in naming each unit of work. You can guarantee the unit of work has been completely written by preparing it in a "preperation" folder and only move it to the "queue" folder after it is complete. The advantage to this strategy is that each unit of work file can be moved to an "inProcess" folder while the processing is happening, and then it can be deleted or moved to an archive folder when finished. If the processing fails, you can recover because the file still exists in the "inProcess" folder. You are in a position to know which units of work are unstable (the dead ones in the "inProcess" folder), as well as which units of work have yet to be processed at all (those still in the "queue" folder).