does anyone find Cascading for Hadoop Map Reduce useful?

Question

I\'ve been trying Cascading, but I cannot see any advantage over the classic map reduce approach for writing jobs.

Map Reduce jobs gives me more freedom and Cascading se

Answer 1

I used Cascading with Bixo to write the complete anti-spam link classification pipeline for a large social network.

The Cascading pipeline resulted in 27 MR jobs, which would have been very difficult to maintain in plain MR. I have written MR jobs before, but using something like Cascading feels like switching from Assembler to Java (insert_fav_language_here).

One of the big advantages over Hive or Pig IMHO is that Cascading is a single jar, which you bundle with your job. Pig and Hive have more dependencies (e.g. MySQL) or are not as easy to embed.

Disclaimer: While I know Chris Wensel personally, I really think Cascading is kick a**. Considering its complexity it is extremely impressive that I haven't found a single bug using it.

Answer 2

I teach the Hadoop Boot Camp course for Scale Unlimited, and also make extensive use of Cascading in Bixo and for building web mining apps at Bixo Labs - so I think I've got a good appreciation for both approaches.

The biggest single advantage I see in Cascading is that it allows you to think about your data processing workflow in terms of operations on fields, and to (mostly) avoid worrying about how to transpose this view of the world onto the key/value model that's intrinsically part of any map-reduce implementation.

The biggest challenge with Cascading is that it is a different way of thinking about data processing workflows, and there's a corresponding conceptual "hump" you need to get over before it all starts making sense. Plus the error messages can remind one of the output from lex/yacc ("conflict in shift/reduce") :)

-- Ken

Answer 3

Cascading is a wrapper around Hadoop that provides Taps and Sinks to and from Hadoop.

Writing Mappers and Reducers for all your tasks is going to be tedious. Try writing one Cascading job and then you're all set to avoiding writing any mappers and reducers.

You also want to look at cascading Taps and Schemes (this is how you input data into your cascading processing job).

With these two, i.e. Ability to avoid writing ad-hoc Hadoop Mappers with Reducers and the ability to consume a wide variety of data sources, you can solve a lot of your data processing very fast and effective.

Cascading is more than just a simple wrapper around hadoop, I am trying to keep the answer simple. For example, I've ported a huge mysql database containing terabytes of data to log files using cascading jdbc tap

Answer 4

Cascading allows you to use simple field names and tuples in place of the primitive types offered by Hadoop which, "... tend to be at the wrong level of granularity for creating sophisticated, highly composable code that can be shared among different developers" (Tom White, Hadoop The Definitive Guide). Cascading was designed to solve those problems. Keep in mind, some of the applications like Cascading, Hive, Pig, etc, were developed in parallel and sometimes do the same thing. If you don't like Cascading or find it confusing, maybe you would be better of using something else?

I'm sure you already have this, but here is the user guide: http://www.cascading.org/1.1/userguide/pdf/userguide.pdf. It provides a decent walk through of the flow of data in a typical Cascading application.

Answer 5

Keeping in mind I'm the author of Cascading...

My suggestion is to use Pig or Hive if they make sense for your problem, Pig especially.

But if you are in the business of data, and not just poking around your data for insights, you will find the Cascading approach makes much more sense for most problems than raw MapReduce.

Your first obstacle with raw MapReduce will be thinking in MapReduce. Trivial problems are simple in MapReduce, but its much easier to develop complex applications if you can work with a model that more easily maps to your problem domain (filter this, parse that, sort those, join the rest, etc).

Next you will realize that a normal unit of work in Hadoop consists of multiple MapReduce jobs. Chaining jobs together is a solvable problem but it should not leak into your application domain level code, it should be hidden and transparent.

Further, you will find refactoring and creating re-usable code much harder if you have to continually move functions between mappers and reducers. or from mappers to the previous reducer to get an optimization. Which leads to the issue of brittleness.

Cascading believes in failing fast as possible. The planner attempts to resolve and satisfy dependencies between all those field names before the Hadoop cluster is even engaged in work. This means 90%+ of all issues will be found before waiting hours for your job to find it during execution.

You can alleviate this in raw MapReduce code by creating domain objects like Person or Document, but many applications don't need all the fields down stream. Consider if you needed the average age of all males. You do not want to pay the IO penalty of passing a whole Person around the network when all you need is a binary gender and numeric age.

With fail fast semantics and lazy binding of sinks and sources, it becomes very easy to build frameworks on Cascading that themselves create Cascading flows (which become many Hadoop MapReduce jobs). A project I'm currently involved with ends up with 100's of MapReduce jobs per run, many created on the fly mid run based on feedback from the data being processed. Search for Cascalog to see an example of a Clojure based framework for simply creating complex processes. Or Bixo for a web mining toolkit and framework that's far easier to customize than Nutch.

Finally Hadoop is never used alone, that means your data is always pulled from some external source and pushed to another after processing. The dirty secret about Hadoop is that it is a very effective ETL framework (so its silly to hear ETL vendors talk about using their tools to push/pull data onto/from Hadoop). Cascading eases this pain somewhat by allowing you to write your operations, applications, and unit tests independent of the integration end-points. Cascading is used in production to load systems like Membase, Memcached, Aster Data, Elastic Search, HBase, Hypertable, Cassandra, etc. (Unfortunately not all the adapters have been released by their authors.)

If you will, please send me a list of the issues your are experiencing with the interface. I am constantly looking for better ways to improve the API and documentation, and the user community is always around to help.

Answer 6

I worked on cascading for couple of years and below are useful things in cascading.

1. code testability
2. easy integration with other tools
3. easily extensibile
4. you will focus only on business logic not on keys and values
5. proven in production and used by even twitter.

I recommend people use cascading most of the times.