Spark Streaming not performing operations on read blocks

问题

I am newbie to Spark Streaming concept and have been stuck from last two days trying to understand Spark streaming from socket. I see Spark is able to read blocks passed to the socket. However it does not perform any operation on the read blocks.

Here is the Spark code

package foo;
import java.io.File;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;

import scala.Tuple2;

public class AppSocket {

public static void main(String[] args)
{

    SparkConf conf = new SparkConf().setAppName("KAFKA").setMaster("local");

    JavaStreamingContext jssc = new JavaStreamingContext(conf, new org.apache.spark.streaming.Duration(1000));

    JavaReceiverInputDStream<String> inputStream = jssc.socketTextStream("localhost", 33333);


    JavaPairDStream<String, Integer> mappedStream = inputStream.mapToPair(
        new PairFunction<String, String, Integer>() {

          public Tuple2<String, Integer> call(String i) {
             System.out.println(i);
            return new Tuple2<String, Integer>(i , 1);
          }
        });

    JavaPairDStream<String, Integer> reducedStream = mappedStream.reduceByKey(
      new Function2<Integer, Integer, Integer>() {

        public Integer call(Integer i1, Integer i2) {
          return i1 + i2;
        }
    });

    reducedStream.print();
    System.out.println("Testing........"+reducedStream.count());
    jssc.start();
    jssc.awaitTermination();


}

}

I am running netcat to create output stream on specified port

nc -lk 33333

I have tried to create output stream. Here is my java code

    ServerSocket serverSocket = null;
    int portNumber = 33333;
    serverSocket = new ServerSocket(portNumber);

    System.out.println("Server Waiting.................");

    Socket clientSocket = serverSocket.accept();

    System.out.println("Server Connected!!!!!!!!!!!!!!!");


    // Wait for a message
    int countflag = 0;
    PrintWriter out = null;
    out = new PrintWriter(clientSocket.getOutputStream(), true);
    while(true)
    {
    Message message = consumer.receive(1000);

    if (message instanceof TextMessage) {
        TextMessage textMessage = (TextMessage) message;
        String text = textMessage.getText();
        System.out.println("Received: " + text);
        list.add(text);
        System.out.println(++countflag);
        if(list.size() > 50)
        {

        for(int i = 0; i < list.size() ; i++)
        {
       System.out.print(i);
        out.write(text);
        out.write("\n");
        out.flush();
        }

       list.clear();
        }

    } else {
        count++;

    }
    if(count > 100) break;
    }
    out.close();
    consumer.close();
    session.close();
    connection.close();

Spark consumes the block sent on the stream however it does not perform any action required on the streamed blocks.

Spark Output Console

14/11/26 15:32:14 INFO MemoryStore: ensureFreeSpace(12) called with curMem=3521, maxMem=278302556
14/11/26 15:32:14 INFO MemoryStore: Block input-0-1417015934400 stored as bytes in memory (estimated size 12.0 B, free 265.4 MB)
14/11/26 15:32:14 INFO BlockManagerInfo: Added input-0-1417015934400 in memory on ip-10-0-1-56.ec2.internal:57275 (size: 12.0 B, free: 265.4 MB)
14/11/26 15:32:14 INFO BlockManagerMaster: Updated info of block input-0-1417015934400
14/11/26 15:32:14 WARN BlockManager: Block input-0-1417015934400 already exists on this machine; not re-adding it
14/11/26 15:32:14 INFO BlockGenerator: Pushed block input-0-1417015934400
14/11/26 15:32:15 INFO ReceiverTracker: Stream 0 received 1 blocks
14/11/26 15:32:15 INFO JobScheduler: Added jobs for time 1417015935000 ms

I would appreciate your help. Thanks in advance

回答1:

Set master as "local[n]" with n > 1 . Receivers take a task slot to run and with "local" if have only one task slot. So the receiver is running in that slot, leaving no task slot to process the data.

I recommended reading the "Points to remember" in the following section in my programming guide. http://spark.apache.org/docs/latest/streaming-programming-guide.html#input-dstreams

来源：https://stackoverflow.com/questions/27153247/spark-streaming-not-performing-operations-on-read-blocks

标签

java

apache-spark

spark-streaming