Does Storm topology need to synchronize shared member variables in the Execute or nextTuple method?

Here is a simple Storm Topology code, which is executed in stand-alone mode and can be used to observe the execution sequence of each module in the Storm topology.
It can run with Storm-0.9.x. Use an IDE, such as idea, to create a Maven project, and then add the following dependencies in pom.xml:

       <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <version>0.9.3</version>
            <scope>compile</scope>
        </dependency>
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;

import java.util.Map;
import java.util.Random;

/**
 * This is a basic example of a Storm topology.
 *
 * 探究对象的变量在被多个线程共享时,Storm是否已经对它进行同步了?
 * 比如ExclamationBolt中的成员变量i,当设置其并行度为13时,其execute方法的打印结果
 * 是同一个i值会被打印13次。
 *
 * 结论:execute()方法中调用的公共方法或共享变量都会被Storm同步。
 */
public class TestCountTopology {
    
    

    public static class ExclamationBolt extends BaseRichBolt {
    
    
        private static final long serialVersionUID = 6618890434446310020L;

        OutputCollector _collector;
        TopologyContext _context;
        int i = 0; //这个变量被多个线程共享时,是否被Storm同步了?

        @Override
        public void prepare(Map conf, TopologyContext context,
                OutputCollector collector) {
            _collector = collector;
            System.out.println("prepare in BOLT is called!");
            _context = context;
        }

        @Override
        public void execute(Tuple tuple) {
            System.out.println("execute in BOLT is called! " + i);
            System.err.println("Worker_port: " + _context.getThisWorkerPort()
                    + " tasks: " + _context.getThisWorkerTasks()
                    + " task_id: " + _context.getThisTaskId() + " BOLT receive: "
                    + new Values(tuple.getString(0)) + " i: " + i);
            inc();
            _collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
            _collector.ack(tuple);
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("word"));
        }

        private void inc() {
            i++;
            System.err.println("task_id: " + _context.getThisTaskId() + " inc() was called");
        }
    }

    public static class WordSpout extends BaseRichSpout {
    
    
        private static final long serialVersionUID = 6962615351294880911L;

        SpoutOutputCollector _collector;
        int i = 0;

        @Override
        public void open(Map conf, TopologyContext context,
                SpoutOutputCollector collector) {
            _collector = collector;
            System.err.println("open in SPOUT is called!");
        }

        @Override
        public void nextTuple() {
            System.out.println("nextTuple in SPOUT is called! " + i);
            Utils.sleep(100);
            final String[] words = new String[] { "nathan", "mike", "jackson",
                    "golda", "bertels" };
            final Random rand = new Random();
            final String word = words[rand.nextInt(words.length)];
            _collector.emit(new Values(word));
            System.out.println("SPOUI emit: " + new Values(word) + " i: " + i);
            i++;
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("word"));
            System.out.println("declareOutputFields in SPOUT is called!");
        }

        public void ack(Object msgId) {
            System.out.println("ack in SPOUT is called!");
        }

        public void fail(Object msgId) {
            System.out.println("fail in SPOUT is called!");
        }

        public void close() {
            System.out.println("close in SPOUT is called!");
        }

    }

    public static void main(String[] args) throws Exception {
        TopologyBuilder builder = new TopologyBuilder();

        // builder.setSpout("word", new TestWordSpout(), 10);
        builder.setSpout("word", new WordSpout(), 1);
        builder.setBolt("exclaim1", new ExclamationBolt(), 13).shuffleGrouping("word");
        //builder.setBolt("exclaim2", new ExclamationBolt(), 3).shuffleGrouping("exclaim1");

        Config conf = new Config();
        conf.setDebug(false);// TOPOLOGY_DEBUG
        conf.setNumWorkers(2);

        if (args != null && args.length > 0) {
            conf.setNumWorkers(3);// TOPOLOGY_WORKERS

            StormSubmitter.submitTopologyWithProgressBar(args[0], conf,
                    builder.createTopology());
        } else {

            LocalCluster cluster = new LocalCluster();
            cluster.submitTopology("test", conf, builder.createTopology());
            Utils.sleep(10000);
            cluster.killTopology("test");
            cluster.shutdown();
            /*
             * comment out the cluster.killTopology() and cluster.shutdown()
             * methods, to avoid the file deletion error.
             */
        }
    }
}

The topology of two nodes, WordSpout method nextTuple repeatedly executed, each time selecting a random word stream sent to the words from the array of characters; the ExclamationBolt reads the word from the stream, and behind it Add three exclamation marks "!!!" before sending to the stream.

For example, WordSpout produces the word "nathan", then ExclamationBolt will produce "nathan!!!".

In the above code, the parallelism of ExclamationBolt Bolt is set to 13, which means that Storm will generate 13 threads, and each thread will execute the Bolt execute method repeatedly , and the value of variable i in this method will be accessed , And then increment by 1.

Q: Since this variable will be shared and modified by multiple threads, do we need to synchronize it?

A: No. Run Topology to find out that the execute method of *ExclamationBolt will add "!!!" to each word received and print it 13 times. The variable i will be printed with the same value 13 times before it increments by 1. This shows that Storm helped us complete the synchronization.

Guess you like

Origin blog.csdn.net/lx1848/article/details/69663755