Hadoop Map Reduce reference static objects

Question

I have a static object in my map reduce job class that I want to initialize once (in the main method), then call a function on it in every mapping. So I have this object, M

Answer 1

Your main() doesn't get invoked on every node, it only runs where you start up the job. In order to have access to your static object, it needs to be initialized at the instantiation of the mapper. That way the initialization will happen on every node that runs a map task.

But there may be another way to do what you're trying to accomplish, so the question is, what does this static object do?

Answer 2

static object resides in memory. now your system is distributed one so object you had created is in memory of node on which your jobtracker is running not on other systems.

now you cannot pass object from job to mapper because config is written as xml, but there is a workaround, Serialize your object into JSON and then put it as string in your configuration and in mappers deserialize this json object

for job

job.getConfiguration().set("some key", "json string")

for mapper

Configuration conf = context.getConfiguration();
conf.get("some key");

Answer 3

Since my object was really loading a library, I ended up using the distributed cache and just instantiating the object in the M/R methods.