Pytorch custom activation functions?

Question

I\'m having issues with implementing custom activation functions in Pytorch, such as Swish. How should I go about implementing and using custom activation functions in Pytorch?<

Answer 1

You can write a customized activation function like below (e.g. weighted Tanh).

class weightedTanh(nn.Module):
    def __init__(self, weights = 1):
        super().__init__()
        self.weights = weights

    def forward(self, input):
        ex = torch.exp(2*self.weights*input)
        return (ex-1)/(ex+1)

Don’t bother about backpropagation if you use autograd compatible operations.