sequencefile

I would like to be able to create a custom InputFormat that reads sequence files, but additionally exposes the file path and offset within that file where the record is located. To take a step back, here's the use case: I have a sequence file containing variably-sized data. The keys are mostly irrelevant, and the values are up to a couple megabytes containing a variety of different fields. I would like to index some of these fields in elasticsearch along with the file name and offset. This way, I can query those fields from elasticsearch, and then use the file name and offset to go back to the

I am using Hadoop 1.0.3. I write logs to an Hadoop sequence file into HDFS, I call syncFS() after each bunch of logs but I never close the file (except when I am performing daily rolling). What I want to guarantee is that the file is available to readers while the file is still being written. I can read the bytes of the sequence file via FSDataInputStream, but if I try to use SequenceFile.Reader.next(key,val), it returns false at the first call. I know the data is in the file since I can read it with FSDataInputStream or with the cat command and I am 100% sure that syncFS() is called. I

There exists in Mahout a command for create sequence file as bin/mahout seqdirectory -c UTF-8 -i <input address> -o <output address> . I want use this command as code API. Julian Ortega You can do something like this: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.SequenceFile; import org.apache.hadoop.io.Text; Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); Path outputPath = new Path("c:\\temp"); Text key = new Text(); // Example, this can be another type of

I have a sequence file whose values look like (string_value, json_value) I don't care about the string value. In Scala I can read the file by val reader = sc.sequenceFile[String, String]("/path...") val data = reader.map{case (x, y) => (y.toString)} val jsondata = spark.read.json(data) I am having a hard time converting this to PySpark. I have tried using reader= sc.sequenceFile("/path","org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text") data = reader.map(lambda x,y: str(y)) jsondata = spark.read.json(data) The errors are cryptic but I can provide them if that helps. My question is, is

I am trying to write a snappy block compressed sequence file from a map-reduce job. I am using hadoop 2.0.0-cdh4.5.0, and snappy-java 1.0.4.1 Here is my code: package jinvestor.jhouse.mr; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.util.Arrays; import java.util.List; import jinvestor.jhouse.core.House; import jinvestor.jhouse.core.util.HouseAvroUtil; import jinvestor.jhouse.download.HBaseHouseDAO; import org.apache.commons.io.IOUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org