scalding

I am trying to figure out how to create an build.sbt file for my own Scalding -based project. Scalding source structure has no build.sbt file. Instead it has project/Build.scala build definition. What would be the right way to integrate my own sbt project with Scalding, so I could also import it later in Eclipse with sbt-eclipse plugin? Update: For the following code: import cascading.tuple.Fields import com.twitter.scalding._ class Scan(args: Args) extends Job(args) { val output = TextLine("tmp/out.txt") val wordsList = List( ("john"), ("liza"), ("nina"), ("x")) val orderedPipe =

We are using Scalding to do ETL and generate the output as a Hive table with partitions. Consequently, we want the directory names for partitions to be something like "state=CA" for example. We are using TemplatedTsv as follows: pipe // some other ETL .map('STATE -> 'hdfs_state) { state: Int => "State=" + state } .groupBy('hdfs_state) { _.pass } .write(TemplatedTsv(baseOutputPath, "%s", 'hdfs_state, writeHeader = false, sinkMode = SinkMode.UPDATE, fields = ('all except 'hdfs_state))) We adopt the code sample from How to bucket outputs in Scalding . Here are two issues we have: except can't be

We have many small files that need combining. In Scalding you can use TextLine to read files as text lines. The problem is we get 1 mapper per file , but we want to combine multiple files so that they are processed by 1 mapper. I understand we need to change the input format to an implementation of CombineFileInputFormat , and this may involve using cascadings CombinedHfs . We cannot work out how to do this, but it should be just a handful of lines of code to define our own Scalding source called, say, CombineTextLine . Many thanks to anyone who can provide the code to do this. As a side