问题
I have a ton of files which are delimited by |, however, they have | as values in the fields as well. the | in the data has been escaped with \ but I don't think BQ is picking it up, is this something I can fix without having to open every single file, and updating? there are 2-3000 files and are all zipped, so doing it one by one is not at all practical.
回答1:
- Read each row as a whole line (CSV, with a weird character delimiter).
- Parse in BigQuery - either via REGEX or JavaScript UDF.
I describe a similar approach here:
- https://medium.com/google-cloud/bigquery-lazy-data-loading-ddl-dml-partitions-and-half-a-trillion-wikipedia-pageviews-cd3eacd657b6
来源：https://stackoverflow.com/questions/51429988/escaping-delimiter-in-bq