问题
OK! So I decided to use Parquet as storage format for hive tables and before I actually implement it in my cluster, I decided to run some tests. Surprisingly, Parquet was slower in my tests as against the general notion that it is faster then plain text files.
Please be noted that I am using Hive-0.13 on MapR
Follows the flow of my operations
Table A
Format - Text Format
Table size - 2.5 Gb
Table B
Format - Parquet
Table size - 1.9 Gb
[Create table B stored as parquet as select * from A]
Table C
Format - Parquet with snappy compression
Table size - 1.9 Gb
[Create table C stored as parquet tblproperties ("parquet.compression"="SNAPPY") as select * from A]
Now I ran some tests on above mentioned tables and follows the details.
- Row count operation
Table A
Map - 15
Reduce - 1
Cumulative CPU - 123.33 sec
Time taken - 59.057 seconds
Table B
Map - 8
Reduce - 1
Cumulative CPU - 204.92 sec
Time taken - 50.33 seconds
- Single Row Selection
Table A
Map - 15
Reduce - 0
Cumulative CPU - 51.18 sec
Time taken - 25.296 seconds
Table B
Map - 8
Reduce - 0
Cumulative CPU - 117.08 sec
Time taken - 27.448 seconds
- Multi Row Selection Using Where clause [1000 rows fetched]
Table A
Map - 15
Reduce - 0
Cumulative CPU - 57.55 sec
Time taken - 20.254 seconds
Table B
Map - 8
Reduce - 0
Cumulative CPU - 113.97 sec
Time taken - 27.678 seconds
- Multi Row Selection [with only 4 columns] Using Where clause [1000 rows fetched]
Table A
Map - 15
Reduce - 0
Cumulative CPU - 57.55 sec
Time taken - 20.254 seconds
Table B
Map - 8
Reduce - 0
Cumulative CPU - 113.97 sec
Time taken - 27.678 seconds
- Aggregation operation [Using sum function on a given column]
Table A
Map - 15
Reduce - 1
Cumulative CPU - 127.85 sec
Time taken - 29.68 seconds
Table B
Map - 8
Reduce - 1
Cumulative CPU - 255.2 sec
Time taken - 41.025 seconds
You can see that in almost all the operations that I have applied on both the tables, Parquet is lagging behind in terms of time taken to execute the query with an exception of row count operation.
I also used table C to perform the aforementioned operations but the results were almost on similar lines with TextFile format again was snappier of the two.
Can some one please let me know what I am doing wrong?
Thanks!
EDIT
I added ORC to the list of storage formats and ran the tests again. Follows the details.
Row count operation
Text Format Cumulative CPU - 123.33 sec
Parquet Format Cumulative CPU - 204.92 sec
ORC Format Cumulative CPU - 119.99 sec
ORC with SNAPPY Cumulative CPU - 107.05 sec
Sum of a column operation
Text Format Cumulative CPU - 127.85 sec
Parquet Format Cumulative CPU - 255.2 sec
ORC Format Cumulative CPU - 120.48 sec
ORC with SNAPPY Cumulative CPU - 98.27 sec
Average of a column operation
Text Format Cumulative CPU - 128.79 sec
Parquet Format Cumulative CPU - 211.73 sec
ORC Format Cumulative CPU - 165.5 sec
ORC with SNAPPY Cumulative CPU - 135.45 sec
Selecting 4 columns from a given range using where clause
Text Format Cumulative CPU - 72.48 sec
Parquet Format Cumulative CPU - 136.4 sec
ORC Format Cumulative CPU - 96.63 sec
ORC with SNAPPY Cumulative CPU - 82.05 sec
Does that mean ORC is faster then Parquet? Or there is something that I can do to make it work better with query response time and compression ratio?
Thanks!
来源：https://stackoverflow.com/questions/32350664/why-is-parquet-slower-for-me-against-text-file-format-in-hive