How to export data as CSV format from SQL Server using sqlcmd?

Question

I can quite easily dump data into a text file such as:

sqlcmd -S myServer -d myDB -E -Q \"select col1, col2, col3 from SomeTable\" 
     -o \"MyData.txt\"


        

Answer 1

A note for anyone looking to do this but also have the column headers, this is the solution that I used an a batch file:

sqlcmd -S servername -U username -P password -d database -Q "set nocount on; set ansi_warnings off; sql query here;" -o output.tmp -s "," -W
type output.tmp | findstr /V \-\,\- > output.csv
del output.tmp

This outputs the initial results (including the ----,---- separators between the headers and data) into a temp file, then removes that line by filtering it out through findstr. Note that it's not perfect since it's filtering out -,-—it won't work if there's only one column in the output, and it will also filter out legitimate lines that contain that string.

Answer 2

You can run something like this:

sqlcmd -S MyServer -d myDB -E -Q "select col1, col2, col3 from SomeTable" 
       -o "MyData.csv" -h-1 -s"," -w 700

• -h-1 removes column name headers from the result
  
• -s"," sets the column seperator to ,
  
• -w 700 sets the row width to 700 chars (this will need to be as wide as the longest row or it will wrap to the next line)

Answer 3

This answer builds on the solution from @iain-elder, which works well except for the large database case (as pointed out in his solution). The entire table needs to fit in your system's memory, and for me this was not an option. I suspect the best solution would use the System.Data.SqlClient.SqlDataReader and a custom CSV serializer (see here for an example) or another language with an MS SQL driver and CSV serialization. In the spirit of the original question which was probably looking for a no dependency solution, the PowerShell code below worked for me. It is very slow and inefficient especially in instantiating the $data array and calling Export-Csv in append mode for every $chunk_size lines.

$chunk_size = 10000
$command = New-Object System.Data.SqlClient.SqlCommand
$command.CommandText = "SELECT * FROM <TABLENAME>"
$command.Connection = $connection
$connection.open()
$reader = $command.ExecuteReader()

$read = $TRUE
while($read){
    $counter=0
    $DataTable = New-Object System.Data.DataTable
    $first=$TRUE;
    try {
        while($read = $reader.Read()){

            $count = $reader.FieldCount
            if ($first){
                for($i=0; $i -lt $count; $i++){
                    $col = New-Object System.Data.DataColumn $reader.GetName($i)
                    $DataTable.Columns.Add($col)
                }
                $first=$FALSE;
            }

            # Better way to do this?
            $data=@()
            $emptyObj = New-Object System.Object
            for($i=1; $i -le $count; $i++){
                $data +=  $emptyObj
            }

            $reader.GetValues($data) | out-null
            $DataRow = $DataTable.NewRow()
            $DataRow.ItemArray = $data
            $DataTable.Rows.Add($DataRow)
            $counter += 1
            if ($counter -eq $chunk_size){
                break
            }
        }
        $DataTable | Export-Csv "output.csv" -NoTypeInformation -Append
    }catch{
        $ErrorMessage = $_.Exception.Message
        Write-Output $ErrorMessage
        $read=$FALSE
        $connection.Close()
        exit
    }
}
$connection.close()

Answer 4

Usually sqlcmd comes with bcp utility (as part of mssql-tools) which exports into CSV by default.

Usage:

bcp {dbtable | query} {in | out | queryout | format} datafile

For example:

bcp.exe MyTable out data.csv

---

To dump all tables into corresponding CSV files, here is the Bash script:

#!/usr/bin/env bash
# Script to dump all tables from SQL Server into CSV files via bcp.
# @file: bcp-dump.sh
server="sql.example.com" # Change this.
user="USER" # Change this.
pass="PASS" # Change this.
dbname="DBNAME" # Change this.
creds="-S '$server' -U '$user' -P '$pass' -d '$dbname'"
sqlcmd $creds -Q 'SELECT * FROM sysobjects sobjects' > objects.lst
sqlcmd $creds -Q 'SELECT * FROM information_schema.routines' > routines.lst
sqlcmd $creds -Q 'sp_tables' | tail -n +3 | head -n -2 > sp_tables.lst
sqlcmd $creds -Q 'SELECT name FROM sysobjects sobjects WHERE xtype = "U"' | tail -n +3 | head -n -2 > tables.lst

for table in $(<tables.lst); do
  sqlcmd $creds -Q "exec sp_columns $table" > $table.desc && \
  bcp $table out $table.csv -S $server -U $user -P $pass -d $dbname -c
done

Answer 5

Since following 2 reasons, you should run my solution in CMD:

1. There may be double quotes in the query

2. Login username & password is sometimes necessary to query a remote SQL Server instance

   sqlcmd -U [your_User]  -P[your_password] -S [your_remote_Server] -d [your_databasename]  -i "query.txt" -o "output.csv" -s"," -w 700