Storm Trident

序 本文主要研究一下storm trident batch的分流与聚合 实例 TridentTopology topology = new TridentTopology(); topology.newStream("spout1", spout) .partitionBy(new Fields("user")) .partitionAggregate(new Fields("user","score","batchId"),new OriginUserCountAggregator(),new Fields("result","aggBatchId")) .parallelismHint(3) .global() .aggregate(new Fields("result","aggBatchId"),new AggAgg(),new Fields("agg")) .each(new Fields("agg"),new PrintEachFunc(),new Fields()) ; 这里最后构造了3个bolt，分别为b-0、b-1、b-2 b-0主要是partitionAggregate，它的parallelismHint为3 b-1主要是处理CombinerAggregator的init，它的parallelismHint为1，由于它的上游bolt有3个task

序 本文主要研究一下storm trident的coordinator 实例 代码示例 @Test public void testDebugTopologyBuild(){ FixedBatchSpout spout = new FixedBatchSpout(new Fields("user", "score"), 3, new Values("nickt1", 4), new Values("nickt2", 7), new Values("nickt3", 8), new Values("nickt4", 9), new Values("nickt5", 7), new Values("nickt6", 11), new Values("nickt7", 5) ); spout.setCycle(false); TridentTopology topology = new TridentTopology(); Stream stream1 = topology.newStream("spout1",spout) .each(new Fields("user", "score"), new BaseFunction() { @Override public void execute(TridentTuple tuple, TridentCollector collector)

序 本文主要研究一下storm trident的state StateType storm-2.0.0/storm-client/src/jvm/org/apache/storm/trident/state/StateType.java public enum StateType { NON_TRANSACTIONAL, TRANSACTIONAL, OPAQUE } StateType有三种类型，NON_TRANSACTIONAL非事务性，TRANSACTIONAL事务性，OPAQUE不透明事务 对应的spout也有三类，non-transactional、transactional以及opaque transactional State storm-2.0.0/storm-client/src/jvm/org/apache/storm/trident/state/State.java /** * There's 3 different kinds of state: * * 1. non-transactional: ignores commits, updates are permanent. no rollback. a cassandra incrementing state would be like this 2. * repeat-transactional:

序 本文主要研究一下storm trident spout的_maxTransactionActive MasterBatchCoordinator storm-core-1.2.2-sources.jar!/org/apache/storm/trident/topology/MasterBatchCoordinator.java TreeMap<Long, TransactionStatus> _activeTx = new TreeMap<Long, TransactionStatus>(); public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { _throttler = new WindowedTimeThrottler((Number)conf.get(Config.TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS), 1); for(String spoutId: _managedSpoutIds) { _states.add(TransactionalState.newCoordinatorState(conf, spoutId)); } _currTransaction =

序 本文主要研究一下storm trident的operations function filter projection Function storm-core-1.2.2-sources.jar!/org/apache/storm/trident/operation/Function.java public interface Function extends EachOperation { /** * Performs the function logic on an individual tuple and emits 0 or more tuples. * * @param tuple The incoming tuple * @param collector A collector instance that can be used to emit tuples */ void execute(TridentTuple tuple, TridentCollector collector); } Function定义了execute方法，它发射的字段会追加到input tuple中 Filter storm-core-1.2.2-sources.jar!/org/apache/storm/trident/operation/Filter.java public