spark-graphx

I have data which starts from 1st Jan 2017 to 7th Jan 2017 and it is a week wanted weekly aggregate. I used window function in following manner val df_v_3 = df_v_2.groupBy(window(col("DateTime"), "7 day")) .agg(sum("Value") as "aggregate_sum") .select("window.start", "window.end", "aggregate_sum") I am having data in dataframe as DateTime,value 2017-01-01T00:00:00.000+05:30,1.2 2017-01-01T00:15:00.000+05:30,1.30 -- 2017-01-07T23:30:00.000+05:30,1.43 2017-01-07T23:45:00.000+05:30,1.4 I am getting output as : 2016-12-29T05:30:00.000+05:30,2017-01-05T05:30:00.000+05:30,723.87 2017-01-05T05:30:00

I need some help to be confirm my choice... and to learn if you can give me some information. My storage database is TitanDb with Cassandra. I have a very large graph. My goal is to use Mllib on the graph latter. My first idea : use Titan with GraphX but I did not found anything or in development in progress... TinkerPop is not ready yet. So I have a look to Giraph. TinkerPop, Titan can communique with Rexster from TinkerPop. My question is : What are the benefit to use Giraph ? Gremlin seems to do the same think and is distributed. Thank you very much to explain me. I think I don't really

my input dataframe is df valx valy 1: 600060 09283744 2: 600131 96733110 3: 600194 01700001 and I want to create the graph treating above two columns are edgelist and then my output should have list of all vertices of graph with its membership . I have tried Graphframes in pyspark and networx library too, but not getting desired results My output should look like below (its basically all valx and valy under V1 (as vertices) and their membership info under V2) V1 V2 600060 1 96733110 1 01700001 3 I tried below import networkx as nx import pandas as pd filelocation = r'Pathtodataframe df csv'

I am new to GraphX and have a Spark dataframe with four columns like below: src_ip dst_ip flow_count sum_bytes 8.8.8.8 1.2.3.4 435 1137 ... ... ... ... Basically I want to map both src_ip and dst_ip to vertices and assign flow_count and sum_bytes as edges attribute. As far as I know, we can not add edges attributes in GraphX as only vertex attributes are permitted. Hence, I am thinking about adding flow_count as edge weight: //create edges val trafficEdges = trafficsFromTo.map(x =Edge(MurmurHash3.stringHash(x(0).toString,MurmurHash3.stringHash(x(1).toString,x(2)) However, can I add sum_bytes

I am moving from MPI based systems to Apache Spark. I need to do the following in Spark. Suppose, I have n vertices. I want to create an edge list from these n vertices. An edge is just a tuple of two integers (u,v), no attributes are required. However, I want to create them in parallel independently in each executor. Therefore, I want to create P edge arrays independently for P Spark Executors. Each array may be of different sizes and depends on the vertices, therefore, I also need the executor id from 0 to n-1 . Next, I want to have a global RDD Array of edges. In MPI, I would create an

I am very new to spark but I want to create a graph from relations that I get from a Hive table. I found a function that is supposed to allow this without defining the vertices but I can't get it to work. I know this isn't a reproducible example but here is my code : import org.apache.spark.SparkContext import org.apache.spark.graphx._ import org.apache.spark.rdd.RDD val sqlContext= new org.apache.spark.sql.hive.HiveContext(sc) val data = sqlContext.sql("select year, trade_flow, reporter_iso, partner_iso, sum(trade_value_us) from comtrade.annual_hs where length(commodity_code)='2' and not