Postgres CSV COPY from/import is not respecting CSV headers

Question

I\'m trying to import data from CSV into the table. The issue is that even with CSV HEADER, the CSV is being imported based on the column index, not on the head

Answer 1

Here's a single line example for importing users using the header row of a csv:

echo "\copy users ($(head -1 users.csv)) FROM 'users.csv' DELIMITER ',' CSV HEADER" | psql

Or with gzip:

echo "\copy users ($(gzip -dc users.csv.gz | head -1)) FROM PROGRAM 'gzip -dc users.csv.gz' DELIMITER ',' CSV HEADER" | psql

Answer 2

Just to answer Jonathan's comment under the accepted answer - if you want to load the data from CSV "respecting" the column order (I had a few dumps with different schema migration history, or missing columns, which I wanted to import).

If you want to use the CSV headers to import it in Bash: (my table's name is alarms)

#!/bin/bash

if [ -z "$1" ] ; then
    echo "Usage: $0 <alarms_dump_file.csv>"
    exit
fi

columns=$(head -n1 $1)
echo "Using columns:"
if ! echo $columns | grep '^id,' ; then
    echo "Missing id in header. No header present? See below:"
    echo $columns
    exit
fi

sudo -u postgres psql YOUR_DATABASE <<EOF
\copy alarms ( $columns ) FROM '$1' DELIMITER ',' CSV HEADER;
EOF

Answer 3

The COPY command by default copies columns from a CSV file in the default order of the columns in the table. The HEADER option on input is ignored, it basically only informs the backend to ignore the first line on input. If the order of the columns in the CSV does not match the order of the columns in the table, you can explicitly specify the column order to match the layout of the CSV file:

COPY churches (id,denomination_id,name,address_id)
FROM '$PWD/data/Data - Churches.csv'
WITH DELIMITER ',' CSV HEADER;