Execute python scripts in Azure DataFactory

问题

I have my data stored in blobs and I have written a python script to do some computations and create another csv. How can I execute this in Azure Data Factory ?

回答1:

Mighty. You could use Azure Data Factory V2 custom activity for your requirements. You can directly execute a command to invoke python script using Custom Activity.

Please refer to this sample on the github.

Hope it helps you.

回答2:

Another option is using a DatabricksSparkPython Activity. This makes sense if you want to scale out, but could require some code modifications for PySpark support. Prerequisite of cause is an Azure Databricks workspace. You have to upload your script to DBFS and can trigger it via Azure Data Factory. The following example triggers the script pi.py:

{
    "activity": {
        "name": "MyActivity",
        "description": "MyActivity description",
        "type": "DatabricksSparkPython",
        "linkedServiceName": {
            "referenceName": "MyDatabricksLinkedservice",
             "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "pythonFile": "dbfs:/docs/pi.py",
            "parameters": [
                "10"
            ],
            "libraries": [
                {
                    "pypi": {
                        "package": "tensorflow"
                    }
                }
            ]
        }
    }
}

See the Documentation for more details.

来源：https://stackoverflow.com/questions/52271088/execute-python-scripts-in-azure-datafactory