load data infile, dealing with fields with comma

Question

How do we deal with field with comma when using load data infile? i have this query:

$sql = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE sales_per_pgs 
        FIELDS TERMINATED BY ','
        LINES TERMINATED BY '\n'
        IGNORE 1 LINES
        (@user_id, @account_code, @pg_code, @sales_value)
        SET
        user_id = @user_id, 
        account_code = @account_code,
        product_group_code = @pg_code,
        sales_value = REPLACE(@sales_value, ',', ''),
        company_id = {$company_id},
        year = {$year},
        month = {$month}";

and a line from the csv looks like this:

139, pg89898, op89890, 1,000,000.00

where 1,000,000.00 is a sales value.

Currently, what is inserted in my database is only "1.

EDIT

The user downloads a form with columns like:

user id, account id, pg id, sales value

where the first three columns user id, account id, pg id, were populated and the sales value column is blank because the user has to fill it up manually... the user uses MS excel to do that...

after the form is completed, he will now upload it, in which i am using the load data infile command...

Answer 1

Your content should really look like:

"139", "pg89898", "op89890", "1,000,000.00"

Then you could add the following to the command:

ENCLOSED BY '"' ESCAPED BY "\\"

And you won't have an issue.

Also, somethign you could try if you don't have any paragraphs or strings with , in them:

FIELDS TERMINATED BY ', '

Answer 2

You will have to alter the CSV file that is being input or alter the output that generates the CSV file - sounds the same but it isn't.

You can modify the data coming in by encapsulating fields with quotes and update your command so that it recognizes that fields are encapsulated with them using a command like ENCLOSED BY '"'

or

alter your output so that it formats the number as 1000000 rather than 1,000,000

Answer 3

In a CSV, comas separate "columns". Since your last value is 1,000,000.00 it is regarded as 3 different columns instead one just one (as intended).

You can either quote each value(column) or change the number format, by removing the commas (,).

Answer 4

if your entire file is exactly as you wrote, then maybe you could use fields terminated by ', ' (comma + space), if and only if you don't have that string within any individual value. If you are using Linux (or any other Unix like system) and your field separator is comma + space, you can use sed to replace this separator with something else:

sed 's/, /|/g' myfile.csv > myfile.txt

However, I would recommend what has already been said: modify your input file enclosing each value with quotes or double quotes and use fields terminated by ',' optionally enclosed by '"'.

Remember that your field termination character must be unique, and must not be contained within any individual value.

Answer 5

As a workaround, try this one -

LOAD DATA INFILE
...
FIELDS TERMINATED BY ', '
...