Find and Replace in a Large File

Question

I want to find a piece of text in a large xml file and want to replace with some other text. The size of the file is around ( 50GB). I want to do this in command line. I am look

Answer 1

Aside from worrying about reading the file in chunks to avoid loading it into memory, you need to dump to disk often enough that you aren't storing the entire contents of the resulting file in memory.

Get-Content sourcefile.txt -ReadCount 10000 | 
    Foreach-Object {
        $line = $_.Replace('http://example.com', 'http://another.example.com')
        Add-Content -Path result.txt -Value $line
    }

The -ReadCount sets the number of lines to read at a time. Then the ForEach-Object writes each line as it is read. For a 30GB file filled with SQL Inserts, I topped out around 200MB of memory and 8% CPU. While, piping it all into Set-Content at hit 3GB of memory before I killed it.