The curious case of the missing exclamation mark in CMD files

Question

I have whittled down a more complex CMD script to the essentials. It reads an input file line by line, unquotes it (if quoted) and writes it out to another CMD file.

Answer 1

The problem at all is delayed expansion here.
With delayed expansion, exclamation marks are used to expand variables, but when there is only one exclamation mark in a line it will be removed.

Specially in FOR /F loops delayed expansion is tricky to handle, as the expansion of the FOR parameter is directly affected by the delayed expansion. The only solution is to disable it temporarily.
The next problem is the CALL, you can't transfer content with CALL (without destroying it).
It's better to transfer the variable by reference (only the variable name) and then get the content in the called function.
The last problem in your code are the percent expansions, do not use them

when delayed expansion is enabled, as the delayed expansion is evaluated after the percent expansion an expanded line will be expanded a second time by the delayed expansion.
Sample. Assume the content of var is Bang!

echo %var% expands to Bang! but then the delayed expansion will evaluate Bang! to Bang.

With echo !var! you simply get Bang!

@echo off
setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
if exist bang2.cmd del bang2.cmd
for /f "tokens=*" %%a in (bang1.txt) do (  
  setlocal DisableDelayedExpansion
  set "line=%%a"
  setlocal EnableDelayedExpansion
  call :doit1 line
  endlocal
  endlocal
)
exit /b

:doit1
set "P1=!%1!"
if "!P1!" EQU "" exit /b
call :unquotex P1
echo>>bang2.cmd echo P1:[!P1!]
exit /b

:unquotex
set "param=!%~1!"
if "!param:~0,1!" == ^""" (
    set "param=!param:~1,-1!"
)
set "%1=!param!"
exit /b