distributed-cache

问题 I am looking for a functionality similar to the distributed cache of Hadoop in Spark. I need a relatively small data file (with some index values) to be present in all nodes in order to make some calculations. Is there any approach that makes this possible in Spark? My workaround so far consists on distributing and reducing the index file as a normal processing, which takes around 10 seconds in my application. After that, I persist the file indicating it as a broadcast variable, as follows:

问题 I have a problem in DistributedCache in Hadoop 2.x the new API, I found some people working around this issue, but it does not solve my problem example this solution does not work with me Because i got a NullPointerException when trying to retrieve the data in DistributedCache My Configuration is as follows: Driver public int run(String[] arg) throws Exception { Configuration conf = this.getConf(); Job job= new Job(conf,"job Name"); ... job.addCacheFile(new URI(arg[1]); Setup protected void

问题 I am wondering if someone can explain how the distributed cache works in Hadoop. I am running a job many times, and after each run I notice that the local distributed cache folder on each node is growing in size. Is there a way for multiple jobs to re-use the same file in the distributed cache? Or is the distributed cache only valid for the lifetime of any individual job? The reason I am confused is that the Hadoop documentation mentions that "DistributedCache tracks modification timestamps

问题 I’ve 2 nodes cluster (v1.04), master and slave. On the master, in Tool.run() we add two files to the DistributedCache using addCacheFile() . Files do exist in HDFS. In the Mapper.setup() we want to retrieve those files from the cache using FSDataInputStream fs = FileSystem.get( context.getConfiguration() ).open( path ). The problem is that for one file a FileNotFoundException is thrown, although the file exists on the slave node: attempt_201211211227_0020_m_000000_2: java.io

问题 Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 4 years ago . I compiled trafficserver-4.1.2 on two openvz containers running on Debian Squeeze, located on two different physical root nodes. Everything including caching is working fine, except for the clustering. I added the same name to the two nodes, as traffic_line -s proxy.config.proxy_name -v fetest Configured to run

问题 Okay I am able to add external jars to my code using ilibjars path. Now how to use those external jars in my code. say I have a function defined in that jar which operates on String. How to use it. using context.getArchiveClassPaths(), i can get a path to it but i don't know how to instantiate that object. here is the sample jar class that i am importing package replace; public class ReplacingAcronyms { public static String Replace(String abc){ String n; n="This is trial"; return n; } }

问题 Lets say i have a MapReduce Job which is creating an output file part-00000 and there is one more job running after the completion of this job. How can i use the output file of the first job in the Distributed cache for the second job. 回答1: The below steps might help you, Pass the first job's output directory path to the Second job's Driver class. Use Path Filter to list files starts with part-* . Refer the below code snippet for your second job driver class, FileSystem fs = FileSystem.get

问题 I am using Hazelcast version 3.3.1. I have a 9 node cluster running on aws using c3.2xlarge servers. I am using a distributed executor service and a distributed map. Distributed executor service uses a single thread. Distributed map is configured with no replication and no near-cache and stores about 1 million objects of size 1-2kb using Kryo serializer. My use case goes as follow: All 9 nodes constantly execute a synchronous remote operation on the distributed executor service and generate

问题 I have run below Test-1 and Test-2 for longer run for performance test with redis configuration values specified, still we see the highlighted error-1 & 2 message and cluster is failing for some time, few of our processing failing. How to solve this problem. please anyone have suggestion to avoid cluster fail which is goes longer than 10seconds, cluster is not coming up within 3 retry attempts (spring retry template we are using for retry mechanism try count is set to 3, and retry after 5sec,

问题 I am trying to implement reduce side join , and using mapfile reader to look up distributed cache but it is not looking up the values when checked in stderr it showed the following error, lookupfile file is already present in hdfs , and seems to be loaded correctly into cache as seen in the stdout. java.lang.IllegalArgumentException: Wrong FS: file:/app/hadoop/tmp/mapred/local/taskTracker/distcache/-8118663285704962921_-1196516983_170706299/localhost/input/delivery_status/DeliveryStatusCodes