I\'m training the Keras object detection model linked at the bottom of this question, although I believe my problem has to do neither with Keras nor with the specific model I\'m
For anyone working in PyTorch, an easy solution which solves this specific problem is to specify in the DataLoader
to drop the last batch:
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=False,
pin_memory=(torch.cuda.is_available()),
num_workers=num_workers, drop_last=True)