I have changed the min_negative_per_image variable in the config to a nonzero constant according to this.
hard_example_miner { num_hard_examples: 3000