OSError: [Errno 22] Invalid argument in subprocess

Question

Python 3.3.3 Windows 7

Here is the full stack:
Traceback (most recent call last):
  File \"Blah\\MyScript.py\", line 578, in Call
    output = process.commun         


        

Answer 1

@eryksun very well analyzed the core of the issue, but I think the bigger picture (the obvious things, probably) is missing in his answer.

So, anyone has an idea as to what is happening?

Your code suffers from a race condition. Sometimes your child process does not behave the way you think it behaves.

OSError: [Errno 22] Invalid argument is raised in your case when the child exits before communicate attempts to write to the named pipe, which subprocess has established between your parent and stdin of your child process.

Popen() does a lot under the hood. In your case, it first creates three named pipes (in _get_handles()), via _winapi.CreatePipe(), one for each of stdin/out/err. Then, it spawns the child process (in _execute_child()), using _winapi.CreateProcess().

_execute_child() finishes with cleanup procedures. Remember: all of this happens within Popen().

Only after Popen() returns, your Python VM in the parent process is about to proceed with the invocation of output = process.communicate(input=SPACE_KEY, timeout=600)

Given that you are on a multi core system, your system has time slices available to let the child process do some work while your Python interpreter is still within execution of Popen().

That is, there is a narrow time window between _winapi.CreateProcess() (after which the child process does some work) and Python's attempt to write to the child's stdin via communicate() (which makes a call to Windows' WriteFile, as eryksun nicely explained).

When your child exits in that time window you retrieve named error.

Why your child process exits earlier than expected? Only you can tell. Obviously, it does not always wait for data coming from stdin.

Answer 2

The command (args) might be incorrect for OS that you're using. Try to sanitize them (check / pass through only allowed characters).

Answer 3

Previously communicate only ignored an EPIPE error when writing to the process stdin. Starting with 3.3.5, per issue 19612, it also ignores EINVAL (22) if the child has already exited (see Lib/subprocess.py line 1199).

Background:

process.communiciate calls process.stdin.write, which calls io.FileIO.write, which on Windows calls the C runtime _write, which calls Win32 WriteFile (which in this case calls NtWriteFile, which dispatches to the NamedPipe filesystem, as either an IRP_MJ_WRITE or FastIoWrite).

If the latter fails, it sets a Windows system error code in the thread. In this case the underlying Windows error is probably ERROR_NO_DATA (232) because the child process has already exited. The C runtime maps this to an errno value of EINVAL (22). Then since _write failed, FileIO.write raises OSError based on the current value of errno.

---

Addendum:

There wouldn't have been a problem at all if the CRT instead mapped ERROR_NO_DATA to EPIPE. Python's own Windows error translation generally follows the CRT's, but per issue 13063, it makes an exception to map ERROR_NO_DATA to EPIPE (32). Thus if the child has already exited, _winapi.WriteFile raises BrokenPipeError.

The following example replicates the EINVAL error given the child process has already exited. It also shows how _winapi.WriteFile (3.3.3 source link) would instead map this error to EPIPE. IMO, this should be considered a bug in Microsoft's CRT.

>>> cmd = 'reg query hkcu'                                                
>>> process = Popen(cmd, stdin=PIPE, stdout=PIPE, universal_newlines=True)
>>> process.stdin.write(' ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

>>> hstdin = msvcrt.get_osfhandle(process.stdin.fileno())
>>> _winapi.WriteFile(hstdin, b' ')                                       
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BrokenPipeError: [WinError 232] The pipe is being closed