How to update parameter by hand during training without disturbing autograd in Pytorch?

Question

Here is my code,

class model(torch.nn.Module):
    def __init__(self, config):
        super(model, self).__init__()
        self.memory_matrix = torch.zeros(