Bat file to delete files only when younger files are present

Question

Our backup system creates .bak files everyday which we can use to restore files if we ever run into issues. If left alone these would fill up our storage so I found a batch

Answer 1

Here's an untested example script which should work as long as you don't have file names where [DESC] contains _, = or other problematic characters.

@Echo Off
SetLocal DisableDelayedExpansion
For /F "Delims==" %%A In ('Set _[ 2^>Nul') Do Set "%%A="
If /I Not "%CD%"=="C:\xxx\yyy" (Set "_[:]=T"
    PushD "C:\xxx\yyy" 2>Nul||Exit /B)
For /F "Tokens=1* Delims=_" %%A In ('Dir /B /O-N *_backup_*_*_*_*_*.bak'
) Do If Defined _[%%A] (Del /A /F "%%A_%%B") Else Set "_[%%A]=T"
If Defined _[:] PopD
EndLocal
Exit /B

Answer 2

I think, an unknown list of [DESC] strings in all the backup file names is most difficult to handle in batch file. The code could be very simple on knowing this list as it can be seen below, or at least on knowing if those strings do not contain characters being critical on batch file processing like !%=.

But the coding challenge for unknown list of [DESCR] strings with special characters in file names was interesting for me and so I developed first following commented batch file:

@echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"

rem Search for files matching the wildcard pattern *_backup_*.bak in backup
rem folder, assign each file name without file extension to environment
rem variable FileName and call the subroutine GetUniqueDescs to get the
rem file description at beginning of each file name into a list in memory.

for /F "delims=" %%I in ('dir "%BackupFolder%\*_backup_*.bak" /A-D /B /ON 2^>nul') do (
    set "FileName=%%~nI"
    call :GetUniqueDescs
)

rem Run command SET with FileDesc: to output all environment variables
rem starting with that string in name and sorted by name and process
rem this list whereby each line ends with =1 as value 1 is assigned
rem to each of these environment variables.

rem For each unique file description in output list assign the file
rem description with =1 appended to environment variable FileDesc
rem and run subroutine DeleteFiles.

for /F "tokens=2 delims=:" %%I in ('set FileDesc: 2^>nul') do (
    set "FileDesc=%%I"
    call :DeleteFiles
)

rem Restore initial environment on starting this batch file and exit it.
endlocal
goto :EOF


rem The subroutine GetUniqueDescs first runs a string substitution which
rem gets the backup pattern part from file name, i.e. everything in file
rem name from _backup_ to end of file name.

rem Then another string substitution is used to remove this string from
rem current file name to get just the description and define an environment
rem variable of which name starts with FileDesc: and ends with the file
rem description. The value assigned to this environment variable is 1.

:GetUniqueDescs
set "BackupPart=%FileName:*_backup_=_backup_%"
call set "FileDesc:%%FileName:%BackupPart%=%%=1"
goto :EOF


rem The subroutine DeleteFiles removes first from passed file description
rem the last two characters being always =1 from list of environment
rem variables starting with FileDesc: and appends the backup wildcard
rem pattern.

rem Command DIR is used to find all files in backup folder starting
rem with current file description and _backup_ and output the found
rem files sorted by last modification date with newest modified file
rem first and oldest modified file last.

rem The command FOR processing this list skips the first file name
rem output by DIR which means the newest file. All other, older
rem files perhaps also found by DIR are deleted one after the other.

:DeleteFiles
set "FilePattern=%FileDesc:~0,-2%_backup_*.bak"
for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%FilePattern%" /A-D /B /O-D /TW') do ECHO del "%BackupFolder%\%%J"
goto :EOF

The command ECHO in last but one line before command del results in just getting displayed which files would be deleted instead of really deleting them.

The option skip=1 in last but one line determines how many backup files are always kept.

For example using skip=5 results in keeping the newest 5 files according to last modification date being usually on backup files also the creation date and deleting all others.

The advantage of such a backup deletion strategy is that it does not matter:

1. how often a specific backup is created – daily, weekly or monthly;
2. if last backup creation was successful at all;
3. if some or even all backup files were deleted manually;
4. how old each individual backup file is;
5. how often the batch file for backup files deletion is executed.

What really matters on deletion of backups is the storage size needed for each backup and how much free storage space remains after deletion process. The file date of a backup file is not limiting the free storage size. The file sizes of all remaining backup files and the total storage size on backup media are the factors which really matter. That's why I do not understand all those "delete older than" questions. Who has to care about age of a file as long as there is enough free space for new files?

The file creation date could be also used by using /TC instead of /TW in last but one line. But the file creation date is the date on which the file was created in that directory and not on which the file itself was created. For that reason the file creation date is only useful when the file was never copied or moved to another directory since first time creation.

I tested this batch file on following files:

C:\xxx\yyy\2004 !Apr_backup_2017_12_18_210001_2986007.bak
C:\xxx\yyy\2004 !Apr_backup_2017_12_19_210001_3168635.bak
C:\xxx\yyy\model%_backup_2017_12_19_210003_2544131.bak
C:\xxx\yyy\model%_backup_2017_12_20_210003_2544131.bak
C:\xxx\yyy\Subscribers=_backup_2017_12_19_210003_3012893.bak
C:\xxx\yyy\Subscribers=_backup_2017_12_20_210003_3012893.bak

The last modification date of each file matched the date in file name.

The output of the batch file was:

del "C:\xxx\yyy\2004 !Apr_backup_2017_12_18_210001_2986007.bak"
del "C:\xxx\yyy\model%_backup_2017_12_19_210003_2544131.bak"
del "C:\xxx\yyy\Subscribers=_backup_2017_12_19_210003_3012893.bak"

That is the expected result. The older file of each file pair would be deleted.

---

Then I thought getting [DESC] part of file name could be done easier as the remaining part of file name without file extension has a fixed length of 33 characters.

@echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"

rem Search for files matching the long wildcard pattern
rem *_backup_????_??_??_??????_???????.bak
rem in backup folder and assign each file name without
rem file extension to environment variable.

rem The last 33 characters are removed from each file name to get the
rem file description part at beginning of each file name. Then define
rem an environment variable of which name starts with FileDesc: and
rem ends with the file description. The value assigned to this
rem environment variable is 1.

for /F "delims=" %%I in ('dir "%BackupFolder%\*_backup_????_??_??_??????_???????.bak" /A-D /B /ON 2^>nul') do (
    set "FileName=%%~nI"
    call set "FileDesc:%%FileName:~0,-33%%=1"
)

rem Run command SET with FileDesc: to output all environment variables
rem starting with that string in name and sorted by name and process
rem this list whereby each line ends with =1 as value 1 is assigned
rem to each of these environment variables.

rem For each unique file description in output list assign the file
rem description with =1 appended to environment variable FileDesc
rem and run subroutine DeleteFiles.

for /F "tokens=2 delims=:" %%I in ('set FileDesc: 2^>nul') do (
    set "FileDesc=%%I"
    call :DeleteFiles
)

rem Restore initial environment on starting this batch file and exit it.
endlocal
goto :EOF


rem The subroutine DeleteFiles removes first from passed file description
rem the last two characters being always =1 from list of environment
rem variables starting with FileDesc: and appends the backup wildcard
rem pattern.

rem Command DIR is used to find all files in backup folder starting
rem with current file description and _backup_ and output the found
rem files sorted by last modification date with newest modified file
rem first and oldest modified file last.

rem The command FOR processing this list skips the first file name
rem output by DIR which means the newest file. All other, older
rem files perhaps also found by DIR are deleted one after the other.

:DeleteFiles
set "FilePattern=%FileDesc:~0,-2%_backup_*.bak"
for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%FilePattern%" /A-D /B /O-D /TW') do ECHO del "%BackupFolder%\%%J"
goto :EOF

That batch file containing also ECHO left to command del in last but one line produces the same result on the 6 files in the backup folder.

I don't know if the batch file could be even more optimized without knowing which characters could exist in [DESC] part of the file names. I did not think about a possible further optimization.

---

Let us assume the list of unique [DESC] strings is well known and can be hard coded in the batch file, for example 2004 !Apr, model% and Subscribers= for the 6 files in my test case:

@echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"
for %%I in ("2004 !Apr" "model%%" "Subscribers=") do for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%%~I*_backup_*.bak" /A-D /B /O-D /TW 2^>nul') do del "%BackupFolder%\%%J"
endlocal

This batch file really deletes files because there is no ECHO in last but one line.

Oh yes, knowing the individual backup file names makes everything much easier.

The batch file can be even optimized to a single command line:

@for %%I in ("2004 !Apr" "model%%" "Subscribers=") do @for /F "skip=1 delims=" %%J in ('dir "C:\xxx\yyy\%%~I*_backup_*.bak" /A-D /B /O-D /TW 2^>nul') do @del "C:\xxx\yyy\%%J"

---

Last let us assume on backup storage media is created:

1. a backup of an entire machine with file name ComputerName_backup_YYYY_MM.tib every 3 months which is huge as taking 200 GiB and where it is enough to have only last backup on the backup storage media;
2. a backup of a folder with files not often updated with file name Folder_backup_YYYY_MM_DD.zip every Saturday which takes about 400 MiB on storage media where it is enough to be able to restore the last 4 weeks;
3. a backup of a database file with file name Database_backup_YYYY_MM_DD.bak every day which takes at the moment 20 MiB per backup, but is growing more or less constant as typical for database files and on where it should be possible to restore data entries of the last 7 days.

The required minimum storage media size is:

(1+1) × 200 GiB + (4+1) × 400 MiB + (7+1) × (20×3) MiB

A storage media size of 1 TiB is really enough for approximately the next 3 years depending on growing rate of database backup on which an increase by a factor of 3 is included already in calculation.

It would be best to delete all backup files no longer needed on creating the daily database backup to keep the backup files management simple by using a single and simple batch file.

@echo off
set "BackupFolder=C:\xxx\yyy"
call :DeleteBackups 1 "ComputerName"
call :DeleteBackups 4 "Folder"
call :DeleteBackups 7 "Database"
goto :EOF

:DeleteBackups
for /F "skip=%1 delims=" %%I in ('dir "%BackupFolder%\%~2*_backup_*" /A-D /B /O-D /TW 2^>nul') do del "%BackupFolder%\%%I"
goto :EOF

Deletion of no longer needed backups can be really so easy on thinking about right strategy.

---

For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.

• call /?
• del /?
• dir /?
• echo /?
• endlocal /?
• for /?
• goto /?
• rem /?
• set /?
• setlocal /?

Read also the Microsoft article about Using Command Redirection Operators for an explanation of 2>nul. The redirection operator > must be escaped with caret character ^ on FOR command lines to be interpreted as literal character when Windows command interpreter processes this command line before executing command FOR which executes the embedded dir command line in a separate command process started in background.