Amazon redshift: bulk insert vs COPYing from s3

Question

I have a redshift cluster that I use for some analytics application. I have incoming data that I would like to add to a clicks table. Let\'s say I have ~10 new \'cl

Answer 1

It might be worth implementing micro batching while performing bulk uploads to Redshift. This article may be worth reading as it does also contain other techniques to be followed for better performance of the COPY commmand.

http://blogs.aws.amazon.com/bigdata/post/Tx2ANLN1PGELDJU/Best-Practices-for-Micro-Batch-Loading-on-Amazon-Redshift

Answer 2

I mean COPYing the data as soon as new .csv files are added into s3 ?

Yes use can use AWS Lambda for this , which can be triggered when you have a new file uploaded

Answer 3

My test results differ a bit. I was loading CSV file to Redshift from OS Windows desktop.

• Row insert was the slowest.
• Multi-row insert was 5 times faster than row inset.
• S3+COPY was 3 times faster than multi-row insert.

What contributed to faster bulk S3+COPY insert.

• The fact that you do not have to parse insert statement from CSV line.
• Stream was compressed before multipart upload to S3.
• COPY command was extremely fast.

I compiled all my findings into one Python script CSV_Loader_For_Redshift

Answer 4

S3 copy works faster in case of larger data loads. when you have say thousands-millions of records needs to be loaded to redshift then s3 upload + copy will work faster than insert queries.

S3 copy works in parallel mode.

When you create table and do insert then there is limit for batch size. The maximum size for a single SQL is 16 MB. So you need to take care size of SQL Batch ( depends on size of each insert query)

The S3 copy automatically applies encoding ( compression) for your table. When your create table and do sample load using copy then you can see compression automatically applied.

But if you are using insert command for beginning you will notice no compression applied which will result more space for table in redshift and slow query process timing in some cases.

If you wish to use insert commands, then create table with each column has applied encodings to save space and faster response time.

Answer 5

Redshift is an Analytical DB, and it is optimized to allow you to query millions and billions of records. It is also optimized to allow you to ingest these records very quickly into Redshift using the COPY command.

The design of the COPY command is to work with parallel loading of multiple files into the multiple nodes of the cluster. For example, if you have a 5 small node (dw2.xl) cluster, you can copy data 10 times faster if you have your data is multiple number of files (20, for example). There is a balance between the number of files and the number of records in each file, as each file has some small overhead.

This should lead you to the balance between the frequency of the COPY, for example every 5 or 15 minutes and not every 30 seconds, and the size and number of the events files.

Another point to consider is the 2 types of Redshift nodes you have, the SSD ones (dw2.xl and dw2.8xl) and the magnetic ones (dx1.xl and dw1.8xl). The SSD ones are faster in terms of ingestion as well. Since you are looking for very fresh data, you probably prefer to run with the SSD ones, which are usually lower cost for less than 500GB of compressed data. If over time you have more than 500GB of compressed data, you can consider running 2 different clusters, one for "hot" data on SSD with the data of the last week or month, and one for "cold" data on magnetic disks with all your historical data.

Lastly, you don't really need to upload the data into S3, which is the major part of your ingestion timing. You can copy the data directly from your servers using the SSH COPY option. See more information about it here: http://docs.aws.amazon.com/redshift/latest/dg/loading-data-from-remote-hosts.html

If you are able to split your Redis queues to multiple servers or at least multiple queues with different log files, you can probably get very good records per second ingestion speed.

Another pattern that you may want to consider to allow near real time analytics is the usage of Amazon Kinesis, the streaming service. It allows to run analytics on data in delay of seconds, and in the same time prepare the data to copy into Redshift in a more optimized way.