activation-function

import numpy as np def relu(z): return np.maximum(0,z) def d_relu(z): z[z>0]=1 z[z<=0]=0 return z x=np.array([5,1,-4,0]) y=relu(x) z=d_relu(y) print("y = {}".format(y)) print("z = {}".format(z)) The code above prints out: y = [1 1 0 0] z = [1 1 0 0] instead of y = [5 1 0 0] z = [1 1 0 0] From what I understand the function calls I've used should only be doing passing by value,passing a copy of the variable. Why is my d_relu function affecting the y variable? Your first mistake is in assuming python passes objects by value... it doesn't - it's pass by assignment (similar to passing by reference

Relu function as defined in keras/activation.py is: def relu(x, alpha=0., max_value=None): return K.relu(x, alpha=alpha, max_value=max_value) It has a max_value which can be used to clip the value. Now how can this be used/called in the code? I have tried the following: (a) model.add(Dense(512,input_dim=1)) model.add(Activation('relu',max_value=250)) assert kwarg in allowed_kwargs, 'Keyword argument not understood: ' + kwarg AssertionError: Keyword argument not understood: max_value (b) Rel = Activation('relu',max_value=250) same error (c) from keras.layers import activations uu = activations

I'm learning tensorflow, deep learning and experimenting various kinds of activation functions. I created a multi-layer FFNN for the MNIST problem. Mostly based on the tutorial from the official tensorflow website, except that 3 hidden layers were added. The activation functions I have experimented are: tf.sigmoid , tf.nn.tanh , tf.nn.softsign , tf.nn.softmax , tf.nn.relu . Only tf.nn.relu doesn't converge, the network output random noise (testing accuracy is about 10%). The following are my source code: import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist =

I have been experimenting with neural networks these days. I have come across a general question regarding the activation function to use. This might be a well known fact to but I couldn't understand properly. A lot of the examples and papers I have seen are working on classification problems and they either use sigmoid (in binary case) or softmax (in multi-class case) as the activation function in the out put layer and it makes sense. But I haven't seen any activation function used in the output layer of a regression model. So my question is that is it by choice we don't use any activation