How to use map function in my code below instead of collect function?

独自空忆成欢 提交于 2019-12-11 16:24:19

问题


Kafka producer tries to send bulk message (10k) at a time but after transferring 3k it's start throwing error. I have used collect function in below code that might be a problem so I want to replace that with map.

//Memberjson is JavaRDD<String>

for (String outputMbrJson : memberjson.collect()) {

    try {

        Producer kafkaProducer = new Producer(configFile);

        kafkaProducer.runProducer(Arrays.asList(outputMbrJson).iterator());
        } 
        catch (Throwable e) {
            e.printStackTrace();
        }
    }
---------------------------------------
//runProducer method 

public Iterator<String> runProducer(Iterator<String> jsons) throws Exception {

        final Producer<Long, String> producer = createProducer();

        ArrayList<String> ret = new ArrayList<String>();

        try {
            while (jsons.hasNext()) {
                String jsonobj = jsons.next();

                ret.add(EMPTY_STRING);
                try {
                    ProducerRecord<Long, String> record = new ProducerRecord<Long, String>(conf.getProperty("producer.topic"), jsonobj.toString());

                    producer.send(record);

                } catch (Exception e) {
                    System.out.println(e.getMessage());
                }
            }
        } 
        catch (Exception e) {
            e.printStackTrace();
        } 
        finally {
            producer.flush();
            producer.close();
        }
        return ret.iterator();
}

来源:https://stackoverflow.com/questions/56744282/how-to-use-map-function-in-my-code-below-instead-of-collect-function

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!