How to run WordCountTopology from storm-starter in Intellij

Question

I work with Storm for a while already, but want to get started with development. As suggested, I am using IntelliJ (up to now, I was using Eclipse and did only write topolog

Answer 1

I managed to run it on my virtualbox, storm version 1.2.2:

just download https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py and put it into any folder you want, for example: /apache-storm-1.2.2/examples/storm-starter/multilang/resources/ , and then change the main function:

public static void main(String[] args) throws Exception {

    SplitSentence pythonSplit = new SplitSentence();
    Map env = new HashMap();
    env.put("PYTHONPATH", "/apache-storm-1.2.2/examples/storm-starter/multilang/resources/");
    pythonSplit.setEnv(env);

    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("spout", new RandomSentenceSpout(), 5);

    builder.setBolt("split",pythonSplit, 8).shuffleGrouping("spout");
    builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));

    Config conf = new Config();
    conf.setDebug(true);

    if (args != null && args.length > 0) {
      conf.setNumWorkers(3);

      StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
    }
    else {
      conf.setMaxTaskParallelism(3);

      LocalCluster cluster = new LocalCluster();
      cluster.submitTopology("word-count", conf, builder.createTopology());

      Thread.sleep(600000);

      cluster.shutdown();
    }
  }

the full instructions can be found on my blog which includes other issues encountered when running it in Local Mode and Local Cluster mode: https://lyhistory.com/storm/

Answer 2

Unfortunately, AFAIK you can't run multilang feature with LocalCluster without having packaged file.

ShellProcess relies on codeDir of TopologyContext, which is used by supervisor. Workers are serialized to stormcode.ser, but multilang files should extracted to outside of serialized file so that python/ruby/node/etc can load it.

Accomplishing this with distribute mode is easy because there's always user submitted jar, and supervisor can know it is what user submitted.

But accomplishing this with local mode is not easy cause supervisor cannot know user submitted jar, and users can run topology to local mode without packaging.

So, Supervisor in local mode finds resource directory ("resources") from each jars (which ends with "jar") in classpath, and copy first occurrence to codeDir.

storm jar places user topology jar to the first of classpath, so it can be run without issue.

So normally, it's natural for ShellProcess to not find "splitsentence.py". Maybe your working directory or PYTHONPATH did the trick.

Answer 3

I struggled with a similar issue, not with the sample topology, but with my own using a Python bolt.

Also experienced the "AttributeError: 'module' object has no attribute 'BasicBolt'" exception - in local mode and when submitting to the cluster.

There are very few resources on this, I found your question and little else discussing this issue.

In case someone else has the same problem: Make sure you include the correct Maven "multilang-python" dependency in your pom file. This will package the correct run time dependencies into the JAR file needed to run your topology.