Did you tried wrapping the backend functions into a Lambda layer?
I think there are some necessary operations within a Keras layer's __call__() method for a Keras Model to be properly built, which will not be executed if you call the backend functions directly.