问题
My source is 'Access Database'
Dynamically generating Source query as 'Select * from <tableName>
'
But I got field names with spaces in source table, and destination is of type .parquet, Data Factory pipeline is failing with below error
Example if Table Employee
got a column 'First Name'
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java,
message: java.lang.IllegalArgumentException:field ended by ';': expected ';' but got 'Area' at line 0: message adms_schema { optional binary RAM Area\ntotal entry:10\r
\norg.apache.parquet.schema.MessageTypeParser.check(MessageTypeParser.java:215)\r\norg.apache.parquet.schema.MessageTypeParser.addPrimitiveType(MessageTypeParser.java:188)\r
\norg.apache.parquet.schema.MessageTypeParser.addType(MessageTypeParser.java:112)\r\norg.apache.parquet.schema.MessageTypeParser.addGroupTypeFields(MessageTypeParser.java:100)\r
\norg.apache.parquet.schema.MessageTypeParser.parse(MessageTypeParser.java:93)\r\norg.apache.parquet.schema.MessageTypeParser.parseMessageType(MessageTypeParser.java:83)\r
\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.getSchema(ParquetWriterBuilderBridge.java:187)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.build
(ParquetWriterBuilderBridge.java:159)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBridge.open(ParquetWriterBridge.java:13)\r
\ncom.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createWriter(ParquetFileBridge.java:27)\r
\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Copy Table Data to Sink",
"details": []
}
if i change query to SELECT [First Name] as FirstName from Employee
, it works fine.
As am generating query dynamically, i was using '*
'
Is there some setting on Sink
(.parquet
) to ignore spaces
in column names?
EDIT some info here https://issues.apache.org/jira/browse/SPARK-4521, not sure how to deal in ADF.
And this link: https://github.com/MicrosoftDocs/azure-docs/issues/28320
来源:https://stackoverflow.com/questions/59044405/azure-data-factory-ms-access-as-source-database-error