Any way to execute a piped command in Python using subprocess module, without using shell=True?

Question

I want to run a piped command line linux/bash command from Python, which first tars files, and then splits the tar file. The command would look like something this in bash:<

Answer 1

Shameless plug, I wrote a subprocess wrapper for easier command piping in python: https://github.com/houqp/shell.py

Example:

shell.ex("tar -cvf - path_to_archive") | "split -b 20m -d -a 5 - 'archive.tar.split'"

Answer 2

If you want to avoid using shell=True, you can manually use subprocess pipes.

from subprocess import Popen, PIPE
p1 = Popen(["tar", "-cvf", "-", "path_to_archive"], stdout=PIPE)
p2 = Popen(["split", "-b", "20m", "-d", "-a", "5", "-", "'archive.tar.split'"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Note that if you do not use the shell, you will not have access to expansion of globbing characters like *. Instead you can use the glob module.

Answer 3

tar can split itself:

tar -L 1000000 -F name-script.sh cf split.tar largefile1 largefile2 ...

name-script.sh

#!/bin/bash
echo "${TAR_ARCHIVE/_part*.tar/}"_part"${TAR_VOLUME}".tar >&"${TAR_FD}"

To re-assemble

tar -M -F name-script.sh cf split.tar

Add this to your python program.

Answer 4

Is there any reason you can't use tarfile? | http://docs.python.org/library/tarfile.html

import tarfile
tar = tarfile.open("sample.tar.gz")
tar.extractall()
tar.close()

Just write like a file like object using tarfile rather than invoking subprocess.