Why does delayed expansion fail when inside a piped block of code?

Question

Here is a simple batch file that demonstrates how delayed expansion fails if it is within a block that is being piped. (The failure is toward the end of the script) Can anyo

Answer 1

I wasn't sure if I should edit my question, or post this as an answer.

I already vaguely knew that a pipe executes both the left and the right side each in its own CMD.EXE "session". But Aacini's and jeb's responses forced me to really think about and investigate what is happening with pipes. (Thank you jeb for demonstrating what is happening when piping into SET /P!)

I developed this investigative script - it helps explain a lot, but also demonstrates some bizarre and unexpected behavior. I'll post the script, followed by the output. Finally I will provide some analysis.

@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion

echo on
@echo NO PIPE - delayed expansion is ON
echo 1: %var1%, %var2%, !var1!, !var2!
(echo 2: %var1%, %var2%, !var1!, !var2!)

@echo(
@echo PIPE LEFT SIDE - Delayed expansion is ON
echo 1L: %%var1%%, %%var2%%, !var1!, !var2! | more
(echo 2L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(setlocal enableDelayedExpansion & echo 3L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(cmd /v:on /c echo 4L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 5L: %%var1%%, %%var2%%, !var1!, !var2! | more
@endlocal
@echo(
@echo Delayed expansion is now OFF
(cmd /v:on /c echo 6L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 7L: %%var1%%, %%var2%%, !var1!, !var2! | more

@setlocal enableDelayedExpansion
@echo(
@echo PIPE RIGHT SIDE - delayed expansion is ON
echo junk | echo 1R: %%var1%%, %%var2%%, !var1!, !var2!
echo junk | (echo 2R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (setlocal enableDelayedExpansion & echo 3R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (cmd /v:on /c echo 4R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 5R: %%var1%%, %%var2%%, !var1!, !var2!
@endlocal
@echo(
@echo Delayed expansion is now OFF
echo junk | (cmd /v:on /c echo 6R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 7R: %%var1%%, %%var2%%, !var1!, !var2!

Here is the output

NO PIPE - delayed expansion is ON

C:\test>echo 1: value1, , !var1!, !var2!
1: value1, , value1,

C:\test>(echo 2: value1, , !var1!, !var2! )
2: value1, , value1,

PIPE LEFT SIDE - Delayed expansion is ON

C:\test>echo 1L: %var1%, %var2%, !var1!, !var2!   | more
1L: value1, %var2%, value1,


C:\test>(echo 2L: %var1%, %var2%, !var1!, !var2! )  | more
2L: value1, %var2%, !var1!, !var2!


C:\test>(setlocal enableDelayedExpansion   & echo 3L: %var1%, %var2%, !var1!, !var2! )  | more
3L: value1, %var2%, !var1!, !var2!


C:\test>(cmd /v:on /c echo 4L: %var1%, %var2%, !var1!, !var2! )  | more
4L: value1, %var2%, value1, !var2!


C:\test>cmd /v:on /c echo 5L: %var1%, %var2%, !var1!, !var2!   | more
5L: value1, %var2%, value1,


Delayed expansion is now OFF

C:\test>(cmd /v:on /c echo 6L: %var1%, %var2%, !var1!, !var2! )  | more
6L: value1, %var2%, value1, !var2!


C:\test>cmd /v:on /c echo 7L: %var1%, %var2%, !var1!, !var2!   | more
7L: value1, %var2%, value1, !var2!


PIPE RIGHT SIDE - delayed expansion is ON

C:\test>echo junk   | echo 1R: %var1%, %var2%, !var1!, !var2!
1R: value1, %var2%, value1,

C:\test>echo junk   | (echo 2R: %var1%, %var2%, !var1!, !var2! )
2R: value1, %var2%, !var1!, !var2!

C:\test>echo junk   | (setlocal enableDelayedExpansion   & echo 3R: %var1%, %var2%, !var1!, !var2! )
3R: value1, %var2%, !var1!, !var2!

C:\test>echo junk   | (cmd /v:on /c echo 4R: %var1%, %var2%, !var1!, !var2! )
4R: value1, %var2%, value1, !var2!

C:\test>echo junk   | cmd /v:on /c echo 5R: %var1%, %var2%, !var1!, !var2!
5R: value1, %var2%, value1,

Delayed expansion is now OFF

C:\test>echo junk   | (cmd /v:on /c echo 6R: %var1%, %var2%, !var1!, !var2! )
6R: value1, %var2%, value1, !var2!

C:\test>echo junk   | cmd /v:on /c echo 7R: %var1%, %var2%, !var1!, !var2!
7R: value1, %var2%, value1, !var2!

I tested both the left and right side of the pipe to demonstrate that processing is symmetric on both sides.

Tests 1 and 2 demonstrate that parentheses don't have any impact on delayed expansion under normal batch circumstances.

Tests 1L,1R: Delayed expansion works as expected. Var2 is undefined, so %var2% and !var2! output demonstrates that the commands are executed in a command line context, and not a batch context. In other words, command line parsing rules are used instead of batch parsing. (see How does the Windows Command Interpreter (CMD.EXE) parse scripts?) EDIT - !VAR2! is expanded in the parent batch context

Tests 2L,2R: The parentheses disable the delayed expansion! Very bizarre and unexpected in my mind. Edit - jeb considers this an MS bug or design flaw. I agree, there doesn't seem to be any rational reason for the inconsistent behavior

Tests 3L,3R: setlocal EnableDelayedExpansion does not work. But this is expected because we are in a command line context. setlocal only works in a batch context.

Tests 4L,4R: Delayed expansion is initially enabled, but parentheses disable it. CMD /V:ON re-enables delayed expansion and everything works as expected. We still have command line context and output is as expected.

Tests 5L,5R: Almost the same as 4L,4R except delayed expansion is already enabled when CMD /V:on is executed. %var2% gives expected command line context output. But !var2! output is blank which is expected in a batch context. This is another very bizarre and unexpected behavior. Edit - actually this makes sense now that I know !var2! is expanded in the parent batch context

Tests 6L,6R,7L,7R: These are analogous to tests 4L/R,5L/R except now delayed expansion starts out disabled. This time all 4 scenarios give the expected !var2! batch context output.

If someone can provide a logical explanation for results of 2L,2R and 5L,5R then I will select that as the answer to my original question. Otherwise I will probably accept this post as the answer (really more of an observation of what happens than an answer) Edit - jab nailed it!

---

Addendum: In response to jeb's comment - here is more evidence that piped commands within a batch execute in a command line context, not a batch context.

This batch script:

@echo on
call echo batch context %%%%
call echo cmd line context %%%% | more

gives this output:

C:\test>call echo batch context %%
batch context %

C:\test>call echo cmd line context %%   | more
cmd line context %%

---

Final Addendum

I've added some additional tests and results that demonstrate all the findings so far. I also demonstrate that FOR variable expansion takes place before the pipe processing. Finally I show some interesting side effects of the pipe processing when a multi-line block is collapsed into a single line.

@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion

echo on
@echo(
@echo Delayed expansion is ON
echo 1: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^!, !var2!, ^^^!var2^^^!, %%cmdcmdline%% | more
(echo 2: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
for %%a in (Z) do (echo 3: %%a %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
(
  echo 4: part1
  set "var2=var2Value
  set var2
  echo "
  set var2
)
(
  echo 5: part1
  set "var2=var2Value
  set var2
  echo "
  set var2
  echo --- begin cmdcmdline ---
  echo %%cmdcmdline%%
  echo --- end cmdcmdline ---
) | more
(
  echo 6: part1
  rem Only this line remarked
  echo part2
)
(
  echo 7: part1
  rem This kills the entire block because the closing ) is remarked!
  echo part2
) | more

Here is the output

Delayed expansion is ON

C:\test>echo 1: %, %var1%, %var2%, !var1!, ^!var1^!, !var2!, ^!var2^!, %cmdcmdline%   | more
1: %, value1, %var2%, value1, !var1!, , !var2!, C:\Windows\system32\cmd.exe  /S /D /c" echo 1: %, %var1%, %var2%, value1, !var1!, , !var2!, %cmdcmdline% "


C:\test>(echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
2: %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"


C:\test>for %a in (Z) do (echo 3: %a %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more

C:\test>(echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
3: Z %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"

C:\test>(
echo 4: part1
 set "var2=var2Value
 set var2
 echo "
 set var2
)
4: part1
var2=var2Value
"
var2=var2Value

C:\test>(
echo 5: part1
 set "var2=var2Value
 set var2
 echo "
 set var2
 echo --- begin cmdcmdline ---
 echo %cmdcmdline%
 echo --- end cmdcmdline ---
)  | more
5: part1
var2=var2Value & set var2 & echo
--- begin cmdcmdline ---
C:\Windows\system32\cmd.exe  /S /D /c" ( echo 5: part1 & set "var2=var2Value
var2=var2Value & set var2 & echo
" & set var2 & echo --- begin cmdcmdline --- & echo %cmdcmdline% & echo --- end cmdcmdline --- )"
--- end cmdcmdline ---


C:\test>(
echo 6: part1
 rem Only this line remarked
 echo part2
)
6: part1
part2

C:\test>(echo %cmdcmdline%   & (
echo 7: part1
 rem This kills the entire block because the closing ) is remarked!
 echo part2
) )  | more

Tests 1: and 2: summarize all the behaviors, and the %%cmdcmdline%% trick really helps to demonstrate what is taking place.

Test 3: demonstrates that FOR variable expansion still works with a piped block.

Tests 4:/5: and 6:/7: show interesting side effects of the way pipes work with multi-line blocks. Beware!

I've got to believe figuring out escape sequences within complex pipe scenarios will be a nightmare.