How to run a python script with dependencies in a virtual environment in Nifi?

人走茶凉 提交于 2020-04-30 06:27:00

问题


Is there a way in Nifi to run a python script which has modules imported from a different folder, requirements specified in a pipfile and has arguments to pass?

In short, how to execute a python script which usually runs in my virtual environment using Nifi?

The end goal for me is to pick up a file using Get File and post it to API. I tried execute process, execute streamcommand processors.


回答1:


To perform follow-on processing on the flowfile using Python, you can use the ExecuteStreamCommand or ExecuteScript/InvokeScriptedProcessor processors.

The ExecuteStreamCommand processor will run an external shell command, like python3 my_python_script.py -arg1 string -arg2 213, which can wrap custom Python code and uses STDIN to pass the existing flowfile content and STDOUT to capture the new flowfile content. Populate the Command Arguments and Command Path properties of the processor to locate your python executable and provide CLI arguments, including flowfile attributes via NiFi Expression Language. See this answer for an example.

The ExecuteScript processor runs Jython code (Python but without access to native libraries, only Python 2.7 compatibility, and some other restrictions due to JSR-223) in the same JVM as NiFi. You can process the flowfile attributes and content directly with Python code. See this answer or this answer for more details.



来源:https://stackoverflow.com/questions/61356091/how-to-run-a-python-script-with-dependencies-in-a-virtual-environment-in-nifi

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!