In Powershell, what's the most efficient way to split a large text file by record type?

Question

I am using Powershell for some ETL work, reading compressed text files in and splitting them out depending on the first three characters of each line.

If I were just fi

Answer 1

Given the size of input files, you definitely want to process a line at a time. I wouldn't think the re-opening/closing of the output files would be too huge a perf hit. It certainly makes the implemation possible using the pipeline even as a one-liner - really not too different from your impl. I wrapped it here to get rid of the horizontal scrollbar:

gc foo.log | %{switch ($_.Substring(0,3)) {
    '001'{$input | out-file output001.txt -enc ascii -append} `
    '002'{$input | out-file output002.txt -enc ascii -append} `
    '003'{$input | out-file output003.txt -enc ascii -append}}}