问题
Kafka producer tries to send bulk message (10k) at a time but after transferring 3k it's start throwing error. I have used collect function in below code that might be a problem so I want to replace that with map.
//Memberjson is JavaRDD<String>
for (String outputMbrJson : memberjson.collect()) {
try {
Producer kafkaProducer = new Producer(configFile);
kafkaProducer.runProducer(Arrays.asList(outputMbrJson).iterator());
}
catch (Throwable e) {
e.printStackTrace();
}
}
---------------------------------------
//runProducer method
public Iterator<String> runProducer(Iterator<String> jsons) throws Exception {
final Producer<Long, String> producer = createProducer();
ArrayList<String> ret = new ArrayList<String>();
try {
while (jsons.hasNext()) {
String jsonobj = jsons.next();
ret.add(EMPTY_STRING);
try {
ProducerRecord<Long, String> record = new ProducerRecord<Long, String>(conf.getProperty("producer.topic"), jsonobj.toString());
producer.send(record);
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
catch (Exception e) {
e.printStackTrace();
}
finally {
producer.flush();
producer.close();
}
return ret.iterator();
}
来源:https://stackoverflow.com/questions/56744282/how-to-use-map-function-in-my-code-below-instead-of-collect-function