passing pandas dataframe into a python subprocess.Popen as an argument

Question

I am attempting to call a python script from a master script. I need the dataframe to be generated only one from within the master script and then passed on to the subproces

Answer 1

Here is a complete example for Python 3.6 of two-way communication between the master script and a subprocess.

master.py

import pandas as pd
import pickle
import subprocess

df = pd.read_excel(r'C:\test_location\file.xlsx',sheetname='Table')

result = subprocess.run(['python', 'call_model.py'], input=pickle.dumps(df), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
returned_df = pickle.loads(result.stdout)
assert df == returned_df

If there is a problem, you can check result.stderr.

subroutine.py

import pickle
import sys

data = pickle.loads(sys.stdin.buffer.read())
sys.stdout.buffer.write(pickle.dumps(data))