Exporting nested fields with invalid characters from Spark 2 to Parquet [duplicate]

Answer 1

I solved the problem this way:

df.toDF(df
    .schema
    .fieldNames
    .map(name => "[ ,;{}()\\n\\t=]+".r.replaceAllIn(name, "_")): _*)

where I replaced all incorrect symbols by "_".

Answer 2

The only solution I have found to work,so far, is to reload the data with a modified schema. The new schema will load the attributes into a map.

Dataset<Row> newData = sql.read().json(path);
StructType newSchema = (StructType) toMapType(newData.schema(), null, "attributes");
newData = sql.read().schema(newSchema).json(path);

---

private DataType toMapType(DataType dataType, String fullColName, String col) {
    if (dataType instanceof StructType) {
        StructType structType = (StructType) dataType;

        List<StructField> renamed = Arrays.stream(structType.fields()).map(
            f -> toMapType(f, fullColName == null ? f.name() : fullColName + "." + f.name(), col)).collect(Collectors.toList());
        return new StructType(renamed.toArray(new StructField[renamed.size()]));
    }
    return dataType;
}

private StructField toMapType(StructField structField, String fullColName, String col) {
    if (fullColName.equals(col)) {
        return new StructField(col, new MapType(DataTypes.StringType, DataTypes.LongType, true), true, Metadata.empty());
    } else if (col.startsWith(fullColName)) {
        return new StructField(structField.name(), toMapType(structField.dataType(), fullColName, col), structField.nullable(), structField.metadata());
    }
    return structField;

}

Answer 3

I have the same problem with @:.

In our case, we solved flattering the DataFrame.

  val ALIAS_RE: Regex = "[_.:@]+".r
  val FIRST_AT_RE: Regex = "^_".r

  def getFieldAlias(field_name: String): String = {
    FIRST_AT_RE.replaceAllIn(ALIAS_RE.replaceAllIn(field_name, "_"), "")
  }

  def selectFields(df: DataFrame, fields: List[String]): DataFrame = {
    var fields_to_select = List[Column]()
    for (field <- fields) {
      val alias = getFieldAlias(field)
      fields_to_select +:= col(field).alias(alias)
    }

    df.select(fields_to_select: _*)
  }

So the following json:

{ 
  object: 'blabla',
  schema: {
    @type: 'blabla',
    name@id: 'blabla'
  }
}

That will be transformed [object, schema.@type, schema.name@id]. @ and dots (in your case =) will create problems for SparkSQL.

So after our SelectFields you can end with [object, schema_type, schema_name_id]. Flattered DataFrame.