PowerShell: the mysterious -RemainingScripts parameter of ForEach-Object

Question

Short question: anyone has detailed information on -RemainingScripts parameter of ForEach-Object?

Long question:

I just started

Answer 1

The source code seems to agree. From powershell on github I was able to find the following relevant pieces of source for the ForEach-Object cmdlet.

The /* ... */ comments are mine, and I've otherwise somewhat altered the code, so refer the the actual source for proper reference.

The first part is just how the parameters are initially read. The important thing to note is that all <ScripBlock>s are put into the same array, save for -End. The -End parameter is handled specially, probably because it's easier to do it the way they did (i.e. it's easy to make sure that something is the first item in the list, just only append to it, but if you do that ensuring something is the last item in the list requires you know you won't append anything else to it).

private List<ScriptBlock> _scripts = new List<ScriptBlock>();

public ScriptBlock Begin {
  /* insert -Begin arg to beginning of _scripts */
  set { _scripts.Insert(0, value); }
}

ScriptBlock[] Process {
  /* append -Process args to _scripts */ }
  set {
    if (value == null) { _scripts.Add(null); }
    else { _scripts.AddRange(value); }
  }
}

private ScriptBlock _endScript;
private bool _setEndScript;

ScriptBlock End {
  set {
    _endScript = value;
    _setEndScript = true;
  }
}

public ScriptBlock[] RemainingScripts {
  set {
    if (value == null) { _scripts.Add(null); }
    else { _scripts.AddRange(value); }
  }
}

Here are the functions which invoke the <ScriptBlock>s in the _script data member. Basically, if there is more than one <ScripBlock> in _scripts, _start gets set to 1. If there is at least three, and -End wasn't explicitly set, _end gets set to _scripts[_scripts.Count - 1]. Then _script[0] gets invoked, the stream is opened up for _script[1.._end-1], and then _endScript is invoked (either _script[_end] or -End).

private int _start, _end;

private void InitScriptBlockParameterSet() {
  // Calculate the start and end indexes for the processRecord script blocks
  _end = _scripts.Count;
  _start = _scripts.Count > 1 ? 1 : 0;

  // and set the end script if it wasnt explicitly set with a named parameter.
  if (!_setEndScript) {
    if (_scripts.Count > 2) {
      _end = _scripts.Count - 1;
      _endScript = _scripts[_end];
    }
  }

  // only process the start script if there is more than one script...
  if (_end < 2) return;

  if (_scripts[0] == null) return;
  /* invoke scripts[0] */
}

private void ProcessScriptBlockParameterSet() {
  /* for (i in _start.._end) invoke _scripts[i] */
}

private void EndBlockParameterSet() {
  /* invoke _endScript */
}

It's not entirely satisfactory, as it could be considered an implementation detail if it's not explicitly stated in the documentation, however it gives me enough confidence to assume that this is the intention. This pattern is also discussed in Powershell in Action which seems to have some sort of blessing from the powershell team (Snover at least).

I guess part of them going "open source" and being more "*nix-y" is defining the source code to be part of the documentation :)

Answer 2

I believe -remainingscripts (with attribute 'ValueFromRemainingArguments') is to enable an idiom like this from Windows Powershell in Action, an idiom which almost nobody knows (20% of Powershell is only documented in that book):

Get-ChildItem | ForEach {$sum=0} {$sum++} {$sum}

The blocks end up acting like begin process end. The parameters being used are actually -process and -remainingscripts.

trace-command -name parameterbinding { Get-ChildItem | ForEach-Object {$sum=0} {$sum++} {$sum} } -PSHost

That trace-command seems to confirm this.

Here's a simple demo of ValueFromRemainingArguments with scriptblocks.

function remaindemo {
  param ($arg1,  [Parameter(ValueFromRemainingArguments)]$remain)
  & $arg1
  foreach ($i in $remain) {
    & $i
  }
}

remaindemo { 'hi' } { 'how are you' } { 'I am fine' }

Other commands with ValueFromRemainingArguments parameters:

gcm -pv cmd | select -exp parametersets | select -exp parameters |
  where ValueFromRemainingArguments | 
  select @{n='Cmdname';e={$cmd.name}},name

Cmdname        Name
-------        ----
ForEach-Object RemainingScripts
ForEach-Object ArgumentList
Get-Command    ArgumentList
Get-Command    ArgumentList
Join-Path      AdditionalChildPath
New-Module     ArgumentList
New-Module     ArgumentList
Read-Host      Prompt
Trace-Command  ArgumentList
Write-Host     Object
Write-Output   InputObject

Answer 3

Here are it's details. ValueFromRemainingArguments is set to true so your guess is correct.

help ForEach-Object

-RemainingScripts <ScriptBlock[]>
    Takes all script blocks that are not taken by the Process parameter.

    This parameter is introduced in Windows PowerShell 3.0.

---

gcm ForEach-Object | select -exp parametersets 

Parameter Name: RemainingScripts
  ParameterType = System.Management.Automation.ScriptBlock[]
  Position = -2147483648
  IsMandatory = False
  IsDynamic = False
  HelpMessage =
  ValueFromPipeline = False
  ValueFromPipelineByPropertyName = False
  ValueFromRemainingArguments = True
  Aliases = {}
  Attributes =
    System.Management.Automation.ParameterAttribute
    System.Management.Automation.AllowEmptyCollectionAttribute
    System.Management.Automation.AllowNullAttribute

Answer 4

I did more research and now feel confident to answer the behavior of -RemainingScripts parameter when multiple ScriptBlocks are passed in.

If you run the following commands and inspect the result carefully, you will find the pattern. It's not quite straightforward, but still not hard to figure out.

1..5 | foreach { "process block" } { "remain block" }
1..5 | foreach { "remain block" }  -Process { "process block" }
1..5 | foreach { "remain block" } -End { "end block" } -Process { "process block" } -Begin { "begin block" }
1..5 | foreach { "remain block 1" } -End { "end block" } -Process { "process block" } { "remain block 2" }
1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } -Begin { "begin block" }
1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } { "remain block 3" }
1..5 | foreach { "process block" } { "remain block 1" } { "remain block 2" } -Begin { "begin block" }
1..5 | foreach { "process block" } { "remain block 1" } { "remain block 2" } { "remain block 3" }

So what's the pattern here?

• When there's single ScriptBlock passed in: easy, it just goes to -Process (the most common usage)

• When exactly 2 ScriptBlocks are passed in, there are 3 possible combinations

  1. -Process & -Begin -> execute as specified
  2. -Process & -End -> execute as specified
  3. -Process & -RemainingScripts -> Process becomes Begin, while RemainingScripts becomes Process

If we run these 2 statements:

1..5 | foreach { "process block" } { "remain block" }
1..5 | foreach { "remain block" }  -Process { "process block" }

# Both of them will return:
process block
remain block
remain block
remain block
remain block
remain block

As you will find out, this is just a special case of the following test case:

• When more than 2 ScriptBlocks are passed in, follow this workflow:

  1. Bind all scriptblocks as specified (Begin,Process,End); remaining ScriptBlocks go to RemainingScripts
  2. Order all scripts as: Begin > Process > Remaining > End

  3. Result of ordering is a collection of ScriptBlocks. Let's call this collection OrderedScriptBlocks

     • If Begin/End are not bound, just ignore

  4. (Internally) Re-bind parameters based on OrderedScriptBlocks

     • OrderedScriptBlocks[0] becomes Begin
     • OrderedScriptBlocks[1..-2] become Process
     • OrderedScriptBlocks[-1] (the last one) becomes End

Let's take this example

1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } { "remain block 3" }

Order result is:

{ "process block" }    # new Begin
{ "remain block 1" }   # new Process
{ "remain block 2" }   # new Process
{ "remain block 3" }   # new End

Now the execution result is completely predictable:

process block
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 3

That's the secret behind -RemainingScripts and now we understand more internal behavior of ForEach-Object!

Still I have to admit there's no documentation to support my guess (not my fault!), but these test cases should be enough to explain the behavior I described.

Answer 5

Here are some more examples that support @FangZhou's ordering hypothesis.

If you specify the other blocks, it seems to make sense how it works:

PS C:\> 1..5 | ForEach-Object -Begin { "Begin: $_" } -End { "End: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Begin: 
Process: 1
R1: 1
R2: 1
R3: 1
Process: 2
R1: 2
R2: 2
R3: 2
Process: 3
R1: 3
R2: 3
R3: 3
Process: 4
R1: 4
R2: 4
R3: 4
Process: 5
R1: 5
R2: 5
R3: 5
End: 

Even if you pass empty blocks:

PS C:\> 1..5 | ForEach-Object -Begin {} -End {} -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 1
R1: 1
R2: 1
R3: 1
Process: 2
R1: 2
R2: 2
R3: 2
Process: 3
R1: 3
R2: 3
R3: 3
Process: 4
R1: 4
R2: 4
R3: 4
Process: 5
R1: 5
R2: 5
R3: 5

However, if you don't specify -End, it does something completely different. The last scriptblock passed to the command is used for -End.

PS C:\> 1..5 | ForEach-Object -Begin { "Begin: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Begin: 
Process: 1
R1: 1
R2: 1
Process: 2
R1: 2
R2: 2
Process: 3
R1: 3
R2: 3
Process: 4
R1: 4
R2: 4
Process: 5
R1: 5
R2: 5
R3: 

And you can change what happens by changing the order of properties:

PS C:\> 1..5 | ForEach-Object  -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" } -Begin { "Begin: $_" } -Process { "Process: $_" }
Begin: 
R1: 1
R2: 1
R3: 1
R1: 2
R2: 2
R3: 2
R1: 3
R2: 3
R3: 3
R1: 4
R2: 4
R3: 4
R1: 5
R2: 5
R3: 5
Process: 

And if you don't specify -Begin, it's different again. Now the first scriptblock passed is used for -Begin:

PS C:\> 1..5 | ForEach-Object -End { "End: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 
R1: 1
R2: 1
R3: 1
R1: 2
R2: 2
R3: 2
R1: 3
R2: 3
R3: 3
R1: 4
R2: 4
R3: 4
R1: 5
R2: 5
R3: 5
End: 

If you specify neither -Begin nor -End, it combines the two. Now the first scriptblock replaces -Begin and the last scriptblock of replaces -End:

PS C:\> 1..5 | ForEach-Object -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 
R1: 1
R2: 1
R1: 2
R2: 2
R1: 3
R2: 3
R1: 4
R2: 4
R1: 5
R2: 5
R3: 

As far as I can tell, it's intended to support positional scriptblocks, where you're going to write:

1..5 | ForEach-Object { "Begin: $_" } { "Process1: $_" } { "Process2: $_" } { "Process3: $_" } { "End: $_" }

Or like so:

1..5 | ForEach-Object { "Begin: $_" },{ "Process1: $_" },{ "Process2: $_" },{ "Process3: $_" },{ "End: $_" }

Both of which output:

Begin: 
Process1: 1
Process2: 1
Process3: 1
Process1: 2
Process2: 2
Process3: 2
Process1: 3
Process2: 3
Process3: 3
Process1: 4
Process2: 4
Process3: 4
Process1: 5
Process2: 5
Process3: 5
End: