Number of tokens limit in a FOR command in a Windows batch script

Question

I was trying to process a text file in a Windows batch script and I ran into something that looks like a limitation to 31 tokens in a FOR loop. I isolated the issue in the c

Answer 1

The FOR loop limitation is documented under header "Tokens" here:
https://ss64.com/nt/for_f.html

"A single FOR /F command can never parse more than 31 tokens, to use more requires a workaround with multiple FOR commands."

Answer 2

There is more than one way to loop through the DATA array:

@ECHO OFF
setlocal ENABLEDELAYEDEXPANSION
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101

for /l %%a in (0,3,150) do @echo %%a data: !data:~%%a,2!

This is possible only because of the two character variables in the %DATA% array. As you will see, the limit is now 99 characters, rather than 31. When you get to the 100th and beyond, the number is truncated.

Rob

Answer 3

A token is the smallest unit of syntax that counts as one chunk. And a Batch command line in Windows 95/98/ME has a maximum limit of 64 tokens. Any more, and the command line will generate a Bad command or file name error.

which is why you're probably limited to 31 within DATA.

Answer 4

I came up with a solution. It's not elegant, but it solves my problem. When the commmand line interpreter cannot go further with the tokens, I pass the remaning of the data to a CALL :label command. Here is an example:

@ECHO OFF

SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO  1st token: %%i
    ECHO 31th token: %%j
    CALL :processdatatokens32-62 %%k
)

:processdatatokens32-62
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 32nd token: %%i
    ECHO 62th token: %%j
    CALL :processdatatokens63-83 %%k
)
GOTO :EOF

:processdatatokens63-83
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 63th token: %%i
    ECHO 93th token: %%j
)
GOTO :EOF

The output is:

 1st token: 01
31th token: 31
32nd token: 32
62th token: 62
63th token: 63
93th token: 93

Answer 5

I've created a typewriter function for batch (just for fun) and also encountered this limitation. Below a stripped-down version to show you how I fixed the problem. You can use up to 8,191 chars including space-separators when executed from the command line. When using a variable the maximum length is 32,767.

@echo off
::#############################################################################
 :_typeWriter <stringVariable> <skipLinefeed>
::#############################################################################
::
:: Copyleft Denny Lenselink 2018
::

%= Set local environment =%
( call ) & 2>nul setlocal EnableDelayedExpansion || exit /b 99

:: Set vars
set "_CNT=0" && set "_LEN=0" && set "_STR= %~1" && set "_TOT=0"

:_typeWriter_Loop
set /a "_CNT+=1"

:: 31 tokens limit fix. Cut the used part of the string and set counter to 1
if !_CNT! equ 32 ( set "_CNT=1" && set "_STR=!_STR:~%_TOT%!" && set "_TOT=0" )

:: Go through string (seeking words)
for /f "tokens=%_CNT% delims= " %%* in ( "%_STR%" ) do (

    :: Set word var
    set "_WRD=#%%*"

    :: Calculate word length
    for /l %%I in ( 12, -1, 0 ) do (

        set /a "_LEN|=1<<%%I"

        for %%J in ( !_LEN! ) do ( if "!_WRD:~%%J,1!"=="" set /a "_LEN&=~1<<%%I" )
    )

    :: Strip first char (used before to calculate correct word length)
    set "_WRD=!_WRD:~1!"

    :: Count chars including spaces
    set /a "_TOT=!_TOT!+!_LEN!+1"

    :: Type word or use echo
    <nul set /p "=!_WRD! "

    :: Do a loop
    goto _typeWriter_Loop
)

:: No linefeed when specified
if "%~2"=="" echo.

endlocal
exit /b

Execute: C:\Temp> typewriter "this is a test that works"

Result:

this is a test that works

Changing the line <nul set /p "=!_WRD! " to echo !_WRD! will result in:

this
is
a
test
that
works

Answer 6

From for /?:

%i is explicitly declared in the for statement and the %j and %k are implicitly declared via the tokens= option. You can specify up to 26 tokens via the tokens= line, provided it does not cause an attempt to declare a variable higher than the letter 'z' or 'Z'. Remember, FOR variables are single-letter, case sensitive, global, and you can't have more than 52 total active at any one time.