Executing shell commands in parallel but limiting jobs (Windows without Cygwin)

问题

Here is what I am trying to do. Suppose I have a program called myprogram.exe, which I have to execute 1000 times.

Under Windows, I could usually do something as simple as:

for /L %n in (1,1,1000) do start /myfolder/myprogram.exe

However, suppose I only have 5 CPU threads I can devote to running the 1000 instances of myprogram.exe, such that I launch only 5, then when one of these finishes another one is launched, etc until the whole 1000 end.

Under Linux and using GNU Parallel, I could simply do:

seq 1000 | parallel -N0 -j5 "nohup myprogram.exe"

How could I achieve something like that in Windows command line? Notice that in my case using Cygwin is not an option, so resorting to xargs and GNU Parallel under Windows are not options either.

回答1:

Here is a way, by using powershell to do the process count. and using a simply set /a as counter.

@echo off
setlocal enabledelayedexpansion
set /a cnt=0
:counter
if !cnt! lss 1000 (
for /F "tokens=*" %%i in ('powershell ^(Get-Process -Name 'myprogram'^).count') do set proc=%%i
             if !proc! lss 5 (
                          start "C:\myfolder\myprogram.exe"
                          set /a cnt+=1
              )
    goto :counter
   )

You could add echo !cnt! in the line before goto :counter if you want to see it count.

It can be done without using delayedexpansion but I prefer to use it here.

回答2:

This will run five processes in parallel. Each time one of them finishes, the next process will be started (so there are always 5 of them until they all are done)

@ECHO off
setlocal enabledelayedexpansion
set bunch=5

for /l %%a in (1,1,1000) do (
  call :loop 
  echo processing: %%a
  start "MyCommand" cmd /c timeout !random:~-1!
)
call :loop
goto :eof

:loop  REM waits for available slot
for /f %%x in ('tasklist /fi "windowtitle eq MyCommand"  ^| find /c "cmd.exe"') do set x=%%x
if %x% geq %bunch% goto :loop
goto :eof

Add the /min switch to start to minimize the started processes.
Give them a unique windowtitle (MyCommand here) to be able to count them.
Replace cmd /c timeout !random:~-1! with your actual command.

EDIT a slightly modified script, which may work better, if myprogram is a GUI (script above will work better with CLI applications):

@ECHO off
setlocal enabledelayedexpansion
set bunch=5

for /l %%a in (1,1,100) do (
  call :loop 
  echo processing: %%a
  start notepad.exe
)
call :loop
goto :eof

:loop  REM waits for available slot
for /f %%x in ('tasklist /fi "imagename eq Notepad.exe"^|find /c "."') do set x=%%x
if %x% geq %bunch% goto :loop
goto :eof

来源：https://stackoverflow.com/questions/51834425/executing-shell-commands-in-parallel-but-limiting-jobs-windows-without-cygwin

标签

windows

batch-file

cmd

parallel-processing

gnu-parallel