Pig & Cassandra & DataStax Splits Control

Question

I have been using Pig with my Cassandra data to do all kinds of amazing feats of groupings that would be almost impossible to write imperatively. I am using DataStax\'s int

Answer 1

You should set pig.noSplitCombination = true. You can do this in one of three places.

When invoking the script:

dse pig -Dpig.noSplitCombination=true /path/to/script.pig

In the Pig script itself:

SET pig.noSplitCombination true
table = LOAD 'cfs://ks/cf' USING CqlStorage();

Or permanently in /etc/dse/pig/pig.properties. Uncomment:

pig.noSplitCombination=true

Otherwise, Pig may set your total input paths (combined) to process: 1.