Merging multiple CSV files into one using PowerShell

Question

Hello I\'m looking for powershell script which would merge all csv files in a directory into one text file (.txt) . All csv files have same header which is always stored in

Answer 1

Get-ChildItem *.csv|select -First 1|Get-Content|select -First 1|Out-File -FilePath .\input.csv -Force #Get the header from one of the CSV Files, write it to input.csv
Get-ChildItem *.csv|foreach {Get-Content $_|select -Skip 1|Out-File -FilePath .\Input.csv -Append} #Get the content of each file, excluding the first line and append it to input.csv

Answer 2

This will append all the files together reading them one at a time:

get-childItem "YOUR_DIRECTORY\*.txt" 
| foreach {[System.IO.File]::AppendAllText
 ("YOUR_DESTINATION_FILE", [System.IO.File]::ReadAllText($_.FullName))}

# Placed on seperate lines for readability

This one will place a new line at the end of each file entry if you need it:

get-childItem "YOUR_DIRECTORY\*.txt" | foreach
{[System.IO.File]::AppendAllText("YOUR_DESTINATION_FILE", 
[System.IO.File]::ReadAllText($_.FullName) + [System.Environment]::NewLine)}

Skipping the first line:

$getFirstLine = $true

get-childItem "YOUR_DIRECTORY\*.txt" | foreach {
    $filePath = $_

    $lines =  $lines = Get-Content $filePath  
    $linesToWrite = switch($getFirstLine) {
           $true  {$lines}
           $false {$lines | Select -Skip 1}

    }

    $getFirstLine = $false
    Add-Content "YOUR_DESTINATION_FILE" $linesToWrite
    }

Answer 3

I found the previous solutions quite inefficient for large csv-files in terms of performance, so here is a performant alternative.

Here is an alternative which simply appends the files:

cmd /c copy  ((gci "YOUR_DIRECTORY\*.csv" -Name) -join '+') "YOUR_OUTPUT_FILE.csv" 

Thereafter, you probably want to get rid of the multiple csv-headers.

Answer 4

type *.csv >> folder\combined.csv

Answer 5

This is pretty trivial in PowerShell.

$CSVFolder = 'C:\Path\to\your\files';
$OutputFile = 'C:\Path\to\output\file.txt';

$CSV = Get-ChildItem -Path $CSVFolder -Filter *.csv | ForEach-Object { 
    Import-Csv -Path $_
}

$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;

Only drawback to this approach is that it does parse every file. It also loads all files into memory, so if we're talking about 4000 files that are 100 MB each you'll obviously run into problems.

You might get better performance with System.IO.File and System.IO.StreamWriter.