What is the correct way to implement gradient accumulation in pytorch?

Question

Broadly there are two ways:

1. Call loss.backward() on every batch, but only call optimizer.step() and optimizer.zero_grad()