How to Import Big XML File (~10GB) into PostgreSQL

问题

I have a XML file of about 10GB, I dont know the content of the file but I would like to import it into my database to make it easy to view.

How can I import an xml file to my PostgreSQL database? (Is this even possible with such a large file?)

I Hope you guys can help me out :)

回答1:

1. Convert XML file into CSV file. Also, when converting, split it to 100Mb-1Gb parts for easier batching.

2. Create the table with columns you defined in the CSV file.

3. Upload the file(s) into Postgres with COPY command. It is the fastest way to upload a large amount of data I know. BTW, it could be done from Java too, with CopyManager class.

Depending on the kind of queries you will perform, you will like to create indexes:

1. It will be the most time-consuming part. However, you may use CREATE INDEX CONCURRENTLY. It will allow you to work with your table while the index is created in the background.

2. If you repeat the import process and already have the table and indexes created, drop the indexes before issuing the COPY command and recreate them later. It will save you much time.

3. If you are still unhappy with the speed of your queries or the speed of indexes creation, maybe it will be a good idea to use Clickhouse instead. However, it depends on what kind of queries you perform.

来源：https://stackoverflow.com/questions/54109811/how-to-import-big-xml-file-10gb-into-postgresql