TFS Database Backup Failed : There is an error in XML document - but which Doc?

Question

Our Nightly TFS 2012 backup has just started to fail. It also fails when run directly through TFS Express Administration Console.

Which file is the following error a

Answer 1

FIX (user workaround):

1. Rename the file BackupSets.xml in the backup destination folder

2. Re-run Full Backup*

   • In TFS Admin Console select Scheduled Backups, then click Take Full Backup Now. Or use command line, PowerShell script, API call as desired.

CAUSE: backupsets.xml in backup destination does not contain valid XML.

• Why does this cause failure? Backup wizard opens backupsettings.xml then calls XML deserializer function System.Xml.Serialization.XmlSerializer.Deserialize preparing to add new entry. Invalid XML content including empty/zero byte or text-only content will cause deserialize exception.

PRODUCTS IMPACTED: Repro confirmed in TFS2010 and on 2017-11-25 I had repro with TFS2015 SP3 :-O

Fix is fairly straightforward... once you understand what is going on. -Zephan

---

MICROSOFT CODE BUGFIX/feature improvement request:

BACKUP Wizard exception handling for backupsets.xml deserialize or parsing exceptions.

• If XML deserialization error then close backupsets.xml, rename it to backupsets-YYMMDD-hhmm-corrupt-backup.xml, then jump to backupsets.xml file not found functionality.

SEVERITY: HIGH (data loss)

• This is a long-standing problem that can lead to major data loss. I've personally seen over 1 month of data loss due to this issue silently blocking backups and making all earlier restore sets unusable (since parsing BackupSets.xml is VERY finicky I couldn't even hack to restore last successful backup.)