Building SQL update statements using a Bash script

Question

For a recent project I need to loop through rows of a csv (ignore.csv) with 3 columns:

acctnum, errcode, date ← in that order…

(Column naming doesn\

Answer 1

You're overwriting the file every time you echo inside the loop. Move the output redirection to the end of the loop.

while IFS=, read -r field1 field2 field3;
do
   echo "UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T' WHERE BUS_DT='$field3' and ETL_ERR_CD='$field2' AND ACCT_KEY in (SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO WHERE ACCT_SRCH_NBR='$field1');"
done < ignore.csv > sqlfile.txt

Answer 2

As Barmar pointed-out:

You're overwriting the file every time you echo inside the loop.

You can do it this way with including space into the field separators of the CSV.

With some important caution:

• Shell cannot reliably parse CSV fields witch may include quotes, escaping of quotes. You should consider using a command able to parse CSV correctly and return values as arguments or null delimited fields like csvtool.
• If your SQL database engine knows about PREPARE statements, it would be safer to PREPARE the SQL UPDATE request, and feed it with arguments. You'd prepare the request before the loop, and then feed the arguments and execute the prepared statement in the loop.

#!/usr/bin/env bash

while IFS=', ' read -r acctnum errcode date; do

  cat <<SQL
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='$date' and ETL_ERR_CD='$errcode' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='$acctnum');
SQL
done < ignore.csv > sqlfile.txt

Parsing and converting your exclude.csv into SQL queries using csvtool:

excludeCSV2sql:

#!/usr/bin/env bash

# This script uses csvtool to parse exclude.csv CSV data from stdin
# and output SQL queries to stdout

# Convert arguments from csvtool call, into an SQL query
to_sql ()
{
  # Double single-quote for SQL string values if any
  local -- \
    acct_num="${1//\'/\'\'}" \
    err_code="${2//\'/\'\'}" \
    date="${3//\'/\'\'}"

  cat <<SQL
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='$date' AND ETL_ERR_CD='$err_code' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='$acct_num'
  );
SQL
}

# Export for use in csvtool call
export -f to_sql

# Process CSV from stdin
csvtool call to_sql -

Testing:

Make the above script executable:

chmod +x excludeCSV2sql

Create a sample test_exclude.csv:

cat >test_exclude.csv <<CSV
foo,bar,baz
here,it's using a single quote, string
this, "has a double-quoted string
with a newline", in it
10100000012, "R4242, has comma", 20200524
10100000042, R1337, 20200525
CSV

Run the test:

./excludeCSV2sql <test_exclude.csv >test.sql

Check the result:

test.sql

UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='baz' AND ETL_ERR_CD='bar' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='foo'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='string' AND ETL_ERR_CD='it''s using a single quote' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='here'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='in it' AND ETL_ERR_CD='has a double-quoted string
with a newline' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='this'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='20200524' AND ETL_ERR_CD='R4242, has comma' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='10100000012'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='20200525' AND ETL_ERR_CD='R1337' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='10100000042'
  );