入门大数据---Storm搭建与应用

空扰寡人 提交于 2020-03-06 03:57:38

1.Storm在Linux环境配置

主机名 tuge1 tuge2 tuge3
部署环境 Zookeeper/Nimbus Zookeeper/Supervisor Zookeeper/Supervisor

​ (部署一览图)

1.1 配置Zookeeper环境(三台机器都要配置,可以先配置一台,然后分发)

  • 去官网下载apache-zookeeper-3.5.5-bin.tar.gz,然后上传到Linux的/opt/zookeeper目录下。(如果没有创建下。)

  • 解压

    tar -xvf apache-zookeeper-3.5.5-bin.tar.gz

  • 配置环境

    vim /etc/profile

    export ZK_HOME=/opt/zookeeper/apache-zookeeper-3.5.5-bin
    
    export PATH=$ZK_HOME/bin:$PATH
  • 配置Zookeeper日志自动清理
    通过配置 autopurge.snapRetainCountautopurge.purgeInterval这两个参数能够实现定时清理了。
    这两个参数都是在zoo.cfg中配置的,将其前面的注释去掉,根据需要修改日志保留个数:

    autopurge.purgeInterval 这个参数指定了清理频率,单位是小时,需要填写一个1或更大的整数,默认是0,表示不开启自己清理功能。

    autopurge.snapRetainCount 这个参数和上面的参数搭配使用,这个参数指定了需要保留的文件数目。默认是保留3个。

1.2 配置Java环境(三台机器都要配置)

  • 去官网下载jdk-8u221-linux-x64.tar.gz,然后上传到/opt/java目录下。(如果没有创建下,根据官网要下载Java8+版本)

  • 解压

    tar -xvf jdk-8u221-linux-x64.tar.gz

  • 配置环境

    vim /etc/profile

    export JAVA_HOME=/opt/java/jdk1.8.0_221
    export PATH=$JAVA_HOME/bin:$ZK_HOME/bin:$PATH

1.3 配置Storm环境(三台机器都要配置)

  • 配置全局环境

    tar -xvf apache-storm-2.1.0.tar.gz

    • 配置环境

    vim /etc/profile

    export STORM_HOME=/opt/storm/apache-storm-2.1.0
    export PATH=$STORM_HOME/bin:$JAVA_HOME/bin:$ZK_HOME/bin:$PATH
    • 重新加载配置文件

    source /etc/profile

  • 配置storm.yaml文件

    vim /opt/storm/apache-storm-2.1.0/conf/storm.yaml

    • 配置Zookeeper服务器:
    # storm.zookeeper.servers:
    #     - "server1"
    #     - "server2"

    改为

    storm.zookeeper.servers:
    - "tuge1"
    - "tuge2"
    - "tuge3"
  • 创建一个状态目录:

    创建 storm-local 目录,并修改权限

    mkdir -p /opt/storm/apache-storm-2.1.0/status

    storm.local.dir: "/opt/storm/apache-storm-2.1.0/status"
  • 配置主控节点地址

    nimbus.seeds: ["tuge1"]
  • 配置Worker计算机数量(实际生产环境根据执行的任务来配置,我这里学习参照官网配置四个先)

    添加几个端口,最多就能分配 几个Worker。这里配置四个先。

    在storm.yaml里面添加如下配置:

    supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

1.4启动Storm

  • 先启动Zookeeper,三台都启动,具体启动步骤参考之前博客

  • 启动Nimbus和UI,在tuge1上执行

    ./storm nimbus >./logs/nimbus.out 2>&1 &
    ./storm ui >>./logs/ui.out 2>&1 &
  • 启动Supervisor,在tuge2,tuge3上运行

    ./storm supervisor >>./logs/supervisor.out 2>&1 &

    PS: >dev>null 2>&1的意思是,将错误输入到标准输出,再将标准输出输入到文件dev和null里面,最后的&意思是后台执行。

    访问ui页面: http://tuge1:8080/

    (PS:如果有什么异常的话,看下zookeeper是否启动,另外使用jps看下numbus和supervisor是否启动)

    如果报异常:Could not find leader nimbus from seed hosts [tuge1]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?

    请进入到zookeeper的bin目录下运行: zkCli.sh,进入到zookeeper控制台,然后删除Storm节点:

    注意:delete只能删除不包含子节点的节点,如果要删除的节点包含子节点,使用rmr命令

    重启zookeeper节点:
    bin/zkServer.sh restart

    没问题就可以看到如下界面啦~

2.Storm本地运行

下面是一个单词追加内容的小案例:

创建一个Maven项目,然后添加如下类结构:

代码如下(思路可以参考上一篇的架构捋顺):

App.java(入口类):
package Demo.Storm;

import java.util.Map;

import javax.security.auth.login.AppConfigurationEntry;
import javax.security.auth.login.Configuration;

import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.LocalCluster.LocalTopology;
import org.apache.storm.generated.StormTopology;
import org.apache.storm.thrift.TException;
import org.apache.storm.topology.TopologyBuilder;
import Demo.Storm.TestWordSpout;

/**
 * Hello world!
 *
 */
public class App {
    public static void main(String[] args) {
        try {

            TopologyBuilder builder = new TopologyBuilder();
            builder.setSpout("words", new TestWordSpout(), 6);//6个spout同时运行
            builder.setBolt("exclaim1", new ExclamationBolt1(), 2).shuffleGrouping("words");//2个bolt同时运行
            builder.setBolt("exclaim2", new ExclamationBolt2(), 2).shuffleGrouping("exclaim1");//2个bolt同时运行

            LocalCluster lc = new LocalCluster();//设置本地运行
    
            lc.submitTopology("wordadd", new Config(), builder.createTopology());//提交topology

        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}
ExclamationBolt1.java(Bolt1):
package Demo.Storm;

import java.util.Map;

import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;

public class ExclamationBolt1 extends BaseRichBolt {
    
    OutputCollector _collector;

    public void prepare(Map<String, Object> topoConf, TopologyContext context, OutputCollector collector) {
        // TODO Auto-generated method stub
        _collector=collector;
    }

    public void execute(Tuple input) {
        // TODO Auto-generated method stub
        String val=input.getStringByField("words")+"!!!";
        _collector.emit(input, new Values(val));//input用来标识是哪个bolt
        _collector.ack(input);//确认bolt
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub
        declarer.declare(new Fields("exclaim1"));
    }

}
ExclamationBolt2.java(Bolt2):
package Demo.Storm;

import java.util.Map;

import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Tuple;

public class ExclamationBolt2 extends BaseRichBolt {

    OutputCollector _collector;
    
    public void prepare(Map<String, Object> topoConf, TopologyContext context, OutputCollector collector) {
        // TODO Auto-generated method stub
        this._collector=collector;
    }

    public void execute(Tuple input) {
        // TODO Auto-generated method stub
        String str= input.getStringByField("exclaim1")+"~~~";
        System.err.println(str);
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub

    }

}
TestWordSpout.java(源源不断传送数据):
package Demo.Storm;
import java.util.Map;
import java.util.Random;
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;
import org.apache.storm.utils.Utils;
public class TestWordSpout extends BaseRichSpout {
    SpoutOutputCollector _collector;

    public void open(Map<String, Object> conf, TopologyContext context, SpoutOutputCollector collector) {
        // TODO Auto-generated method stub
        _collector=collector;

    }

    public void nextTuple() {
        // TODO Auto-generated method stub
        Utils.sleep(100);
        final String[] words = new String[] { "你好啊", "YiMing" };
        final Random rand = new Random();
        final String word = words[rand.nextInt(words.length)];//司机发送字符串
        _collector.emit(new Values(word));

    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub
        declarer.declare(new Fields("words"));
    }

}
pom.xm(maven配置文件,如果问题参考后面介绍):
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>Demo</groupId>
  <artifactId>Storm</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Storm</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.storm/storm-client -->
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-client</artifactId>
        <version>2.1.0</version>
    
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.storm/storm-server -->
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-server</artifactId>
        <version>2.1.0</version>      
    </dependency>
    
  </dependencies>
</project>

运行效果如下:

遇到的问题:

问题一:LocalCluster这个类明明存在,却引入不了?

解决: 首先看下和能引入的jar包的区别是,这个jar包是灰色的,一脸懵,估计就是这的事情。然后网上搜了搜为什么有的包是灰色的,果然有答案,原来是pom里面带的 test 导致的,将它删除就OK了,如果还有灰色的,那就干脆一点,将*** 都去掉,要不这是个坑。。。(参考来源

Storm的八种Grouping策略

1)shuffleGrouping(随机分组)

2)fieldsGrouping(按照字段分组,在这里即是同一个单词只能发送给一个Bolt)

3)allGrouping(广播发送,即每一个Tuple,每一个Bolt都会收到)

4)globalGrouping(全局分组,将Tuple分配到task id值最低的task里面)

5)noneGrouping(随机分派)

6)directGrouping(直接分组,指定Tuple与Bolt的对应发送关系)

7)Local or shuffle Grouping

8)customGrouping (自定义的Grouping)

3.Storm在Linux集群上运行

下面是一个统计单词数量的小案例:

在上面项目基础上继续添加如下类文件,结构如下:

代码如下:

WordCountApp.java(入口类)
package Demo.Storm;

import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.StormSubmitter;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.tuple.Fields;


public class WordCountApp {

      
      

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        TopologyBuilder builder = new TopologyBuilder();
        
        builder.setSpout("words", new WordCountSpout(), 8);//8个Spout同时执行
        
        builder.setBolt("wordSplit", new WordCountSplitBolt(), 3).shuffleGrouping("words");//3个Bolt同时执行
        builder.setBolt("wordSum", new WordCountSumBolt(), 3).fieldsGrouping("wordSplit", new Fields("word"));//3个Bolt同时执行

        if (args.length > 0) {//如果有参数,走集群执行
            try {
                StormSubmitter.submitTopology(args[0], new Config(), builder.createTopology());

            } catch (Exception ex) {
                ex.printStackTrace();
            }

        } else {//没有参数走本机执行

            try {
                LocalCluster lc = new LocalCluster();
                lc.submitTopology("wordCount", new Config(), builder.createTopology());

            } catch (Exception ex) {
                ex.printStackTrace();
            }
        }

    }

}
WordCountSpout.java(源源不断的提供数据)
package Demo.Storm;

import java.util.Map;
import java.util.Random;

import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;
import org.apache.storm.utils.Utils;


public class WordCountSpout extends BaseRichSpout {

    SpoutOutputCollector _collector;
    public void open(Map<String, Object> conf, TopologyContext context, SpoutOutputCollector collector) {
        // TODO Auto-generated method stub
        this._collector=collector;
    }

    public void nextTuple() {
        // TODO Auto-generated method stub
        Utils.sleep(1000);
        String[] words=new String[] {
                "hello YiMing",
                "nice to meet you"
        };
        Random r=new Random();
        _collector.emit(new Values(words[r.nextInt(words.length)]));//随机传递一个字母
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub
        declarer.declare(new Fields("words"));
    }

}
WordCountSplitBolt.java(分割类)
package Demo.Storm;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;

public class WordCountSplitBolt extends BaseRichBolt {

    OutputCollector _collector;

    public void prepare(Map<String, Object> topoConf, TopologyContext context, OutputCollector collector) {
        // TODO Auto-generated method stub
        this._collector = collector;
    }

    //传递分割后的字母
    public void execute(Tuple input) {
        // TODO Auto-generated method stub
        String line=input.getString(0);
        String[] lineGroup= line.split(" ");
        for(String str:lineGroup) {
            List list=new Values(str);
            _collector.emit(input, list);
            _collector.ack(input);
        }       
    }
    //声明传递的字母名称为 word,下一个bolt可以通过此名称获取
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub
        declarer.declare(new Fields("word"));
    }

}
WordCountSumBolt.java(归纳统计类)
package Demo.Storm;

import java.util.HashMap;
import java.util.Map;

import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Tuple;

public class WordCountSumBolt extends BaseRichBolt {

    OutputCollector _collector;
    Map<String, Integer> map = new HashMap<String, Integer>();

    public void prepare(Map<String, Object> topoConf, TopologyContext context, OutputCollector collector) {
        // TODO Auto-generated method stub
        this._collector = collector;
    }
    //归纳统计
    public void execute(Tuple input) {
        // TODO Auto-generated method stub
        String word = input.getString(0);

        if (map.containsKey(word)) {
            map.put(word, (map.get(word) + 1));
        } else {
            map.put(word, 1);
        }
        System.err.println("单词:" + word + ",出现:" + map.get(word) + "次");

    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        // TODO Auto-generated method stub

    }

}
弄好后,编译打包,然后上传到Linux上面。

进入到/opt/storm/apache-storm-2.1.0/bin执行

[root@tuge1 bin]# ./storm jar /opt/data/storm/WordCount.jar Demo.Storm.WordCountApp wc

运行官方示例:

storm jar all-my-code.jar org.apache.storm.MyTopology arg1 arg2

结束任务

storm kill wc(也就是topology名称)

要想获取结果请参考: https://blog.csdn.net/cuihaolong/article/details/52684396

PS:运行过程中,Task不可以改变,但是Worker和Executer可以改变。

zj。。。

系列传送门

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!