How to implement maclaurin series in keras?

Question

I am trying to implement expandable CNN by using maclaurin series. The basic idea is the first input node can be decomposed into multiple nodes with different orders and coeffic

Answer 1

Interesting question. I have implemented a Keras model that computes the Taylor expansion as you described:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Input, Lambda


def taylor_expansion_network(input_dim, max_pow):
    x = Input((input_dim,))

    # 1. Raise input x_i to power p_i for each i in [0, max_pow].
    def raise_power(x, max_pow):
        x_ = x[..., None]  # Shape=(batch_size, input_dim, 1)
        x_ = tf.tile(x_, multiples=[1, 1, max_pow + 1])  # Shape=(batch_size, input_dim, max_pow+1)
        pows = tf.range(0, max_pow + 1, dtype=tf.float32)  # Shape=(max_pow+1,)
        x_p = tf.pow(x_, pows)  # Shape=(batch_size, input_dim, max_pow+1)
        x_p_ = x_p[..., None]  # Shape=(batch_size, input_dim, max_pow+1, 1)
        return x_p_

    x_p_ = Lambda(lambda x: raise_power(x, max_pow))(x)

    # 2. Multiply by alpha coefficients
    h = LocallyConnected2D(filters=1,
                           kernel_size=1,  # This layer is computing a_i * x^{p_i} for each i in [0, max_pow]
                           use_bias=False)(x_p_)  # Shape=(batch_size, input_dim, max_pow+1, 1)

    # 3. Compute s_i for each i in [0, max_pow]
    def cumulative_sum(h):
        h = tf.squeeze(h, axis=-1)  # Shape=(batch_size, input_dim, max_pow+1)
        s = tf.cumsum(h, axis=-1)  # s_i = sum_{j=0}^i h_j. Shape=(batch_size, input_dim, max_pow+1)
        s_ = s[..., None]  # Shape=(batch_size, input_dim, max_pow+1, 1)
        return s_

    s_ = Lambda(cumulative_sum)(h)

    # 4. Compute sum w_i * s_i each i in [0, max_pow]
    s_ = LocallyConnected2D(filters=1,  # This layer is computing w_i * s_i for each i in [0, max_pow]
                            kernel_size=1,
                            use_bias=False)(s_)  # Shape=(batch_size, input_dim, max_pow+1)
    y = Lambda(lambda s_: tf.reduce_sum(tf.squeeze(s_, axis=-1), axis=-1))(s_)  # Shape=(batch_size, input_dim)

    # Return Taylor expansion model
    model = Model(inputs=x, outputs=y)
    model.summary()
    return model

The implementation applies the same Taylor expansion to each element of the flattened tensor with shape (batch_size, input_dim=512) coming from the convolutional network.

---

UPDATE: As we discussed in the comments section, here is some code to show how your function expandable_cnn could be modified to integrate the model defined above:

def expandable_cnn(input_shape, nclass, approx_order):
    inputs = Input(shape=(input_shape))
    h = inputs
    h = Conv2D(filters=32, kernel_size=(3, 3), padding='same', activation='relu', input_shape=input_shape)(h)
    h = Conv2D(filters=32, kernel_size=(3, 3), activation='relu')(h)
    h = MaxPooling2D(pool_size=(2, 2))(h)
    h = Dropout(0.25)(h)
    h = Flatten()(h)
    h = Dense(512, activation='relu')(h)
    h = Dropout(0.5)(h)
    taylor_model = taylor_expansion_network(input_dim=512, max_pow=approx_order)
    h = taylor_model(h)
    h = Activation('relu')(h)
    print(h.shape)
    h = Dense(nclass, activation='softmax')(h)
    model = Model(inputs=inputs, outputs=h)
    return model

Please note that I do not guarantee that your model will work (e.g. that you will get good performance). I just provided a solution based on my interpretation of what you want.