Apache Giraph - Cannot run in split master / worker mode since there is only 1 task at a time

Question

I ran Giraph 1.0.0 with hadoop 2.2.0 using the PageRank Benchmark example here.

Suddenly I got this error result:

Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only one worker since only 1 task at a time! at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:151) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225) at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:90) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

When I changed the number of workers to 1, I got:

Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time! at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:157) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225) at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:90) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Any solutions?

Answer 1

Hi I assume you are not running on a cluster? If I run in our demo VMs I get the same error.

You can disable the split master worker behaviour in giraph-site.xml

giraph.SplitMasterWorker=false

If you just want to disable this during a one-shot exeuction you can also pass it as a command-line parameter to your program.

-ca giraph.SplitMasterWorker=false

For instance I run a demo for my Big Data lecture like this:

#!/bin/bash

yarn jar /root/giraph-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.GiraphRunner at.jku.tk.steinbauer.bigdata.giraph.MaxInDegreeComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hue/graph/tinygraph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat  -op /user/hue/graph/degree -w 1 -ca giraph.SplitMasterWorker=false