File Access Strategy in a Multi-Threaded Environment (Web App)

Question

I have a file which is an XML representation of some data that is taken from a Web service and cached locally within a Web Application. The idea being is that this data is <

Answer 1

How about using AutoResetEvent to communicate between threads? I created a console app which creates roughly 8 GB file in createfile method and then copy that file in main method

 static AutoResetEvent waitHandle = new AutoResetEvent(false);
    static string filePath=@"C:\Temp\test.txt";
    static string fileCopyPath=@"C:\Temp\test-copy.txt";
    static void Main(string[] args)
    {
        Console.WriteLine("in main method");
        Console.WriteLine();
        Thread thread = new Thread(createFile);
        thread.Start();

        Console.WriteLine("waiting for file to be processed ");
        Console.WriteLine();
        waitHandle.WaitOne();
        Console.WriteLine();

        File.Copy(filePath, fileCopyPath);
        Console.WriteLine("file copied ");

    }


    static void createFile()
    {

        FileStream fs= File.Create(filePath);            
        Console.WriteLine("start processing a file "+DateTime.Now);
        Console.WriteLine();
        using (StreamWriter sw = new StreamWriter(fs))
        {
            for (long i = 0; i < 300000000; i++)
            {
                sw.WriteLine("The value of i is " + i);

            }
        }
        Console.WriteLine("file processed " + DateTime.Now);
        Console.WriteLine();

        waitHandle.Set();
    }

Answer 2

If you're locking on a object stored as a static then the lock should work for all threads in the same Application Domain, but perhaps you need to upload a code sample so we can have a look at the offending lines.

That said, one thought would be to check if IIS is configured to run in Web Garden mode (i.e. more than 1 process executing your application) which would break your locking logic. While you could fix such a situation with a mutex it'd be easier to reconfigure your application to execute in a single process, although you'd be wise to check the performance before and after messing with the web garden settings as it can potentially affect performance.

Answer 3

OK, I have been working on this and ended up creating a stress-test module to basically hammer the crap out of my code from several threads (See Related Question).

It was much easier from this point on to find holes in my code. It turns out that my code wasn't actually far off, but there was a certain logic path that it could enter in to which basically caused read/write operations to stack up, meaning if they didn't get cleared in time, it would go boom!

Once I took that out, ran my stress test again, all worked fine!

So, I didn't really do anything special in my file access code, just ensured I used lock statements where appropriate (i.e. when reading or writing).

Answer 4

Here is the code that I use to make sure a file is not locked by another process. It's not 100% foolproof, but it gets the job done most of the time:

    /// <summary>
    /// Blocks until the file is not locked any more.
    /// </summary>
    /// <param name="fullPath"></param>
    bool WaitForFile(string fullPath)
    {
        int numTries = 0;
        while (true)
        {
            ++numTries;
            try
            {
                // Attempt to open the file exclusively.
                using (FileStream fs = new FileStream(fullPath,
                    FileMode.Open, FileAccess.ReadWrite, 
                    FileShare.None, 100))
                {
                    fs.ReadByte();

                    // If we got this far the file is ready
                    break;
                }
            }
            catch (Exception ex)
            {
                Log.LogWarning(
                   "WaitForFile {0} failed to get an exclusive lock: {1}", 
                    fullPath, ex.ToString());

                if (numTries > 10)
                {
                    Log.LogWarning(
                        "WaitForFile {0} giving up after 10 tries", 
                        fullPath);
                    return false;
                }

                // Wait for the lock to be released
                System.Threading.Thread.Sleep(500);
            }
        }

        Log.LogTrace("WaitForFile {0} returning true after {1} tries",
            fullPath, numTries);
        return true;
    }

Obviously you can tweak the timeouts and retries to suit your application. I use this to process huge FTP files that take a while to be written.

Answer 5

You could maybe create the file with a temporary name ("data.xml_TMP"), and when it's ready change the name to what it is supposed to be. That way, no other process will be accessing it before it is ready.