GPFlow-2.0 - issue with default_float and likelihood variance

问题

I am trying to use gpflow (2.0rc) with float64 and had been struggling to get even simple examples to work. I configure gpflow using:

gpflow.config.set_default_float(np.float64)

I am using GPR:

# Model construction:
k = gpflow.kernels.Matern52(variance=1.0, lengthscale=0.3)
m = gpflow.models.GPR((X, Y), kernel=k)
m.likelihood.variance = 0.01

And indeed if I print a summary, both parameters have dtype float64. However if I try to predict with this model, I get an error.

tensorflow.python.framework.errors_impl.InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a double tensor but is a float tensor [Op:AddV2] name: add/

A debugging session lead me to the following line in gpr.py (line 88)

s = tf.linalg.diag(tf.fill([num_data], self.likelihood.variance))

This creates a matrix with dtype float32 which causes the blow-up as described above. So the issue seems to be with the way I'm setting the likelihood variance?

Here's a full Python script that demonstrates the issue:

import numpy as np
import gpflow

gpflow.config.set_default_float(np.float64)

# data:
X = np.random.rand(10, 1)
Y = np.sin(X)

assert X.dtype == np.float64
assert Y.dtype == np.float64

# Model construction:
k = gpflow.kernels.Matern52(variance=1.0, lengthscale=0.3)
m = gpflow.models.GPR((X, Y), kernel=k)
m.likelihood.variance = 0.01

gpflow.utilities.print_summary(m)

# Predict
xx = np.array([[1.0]])
assert xx.dtype == np.float64

mean, var = m.predict_y(xx)
print(f'mean: {mean}')
print(f'var: {var}')

回答1:

I've checked and apparently it's ok to answer your own question here.

After further investigation and raising an issue on github, I want to add some follow-up.

The quick answer to my question is that yes the issue is with the way I'm setting the variance. Had I set it this way:

m.likelihood.variance.assign(0.01)

then my code would have worked. (Although this is not exactly equivalent since the likelihood variance is now a trainable parameter.)

But this is only a manifestation of a wider issue with Tensorflow's API - as raised here: https://github.com/tensorflow/tensorflow/issues/26033

Trying to use float64s instead of the default float32 in any library built on top of tensorflow is likely to be painful until this issue is addressed.

It leads to pretty user hostile API features if you are trying to use float64s as in some places Python float arguments are fine while in others they need to be cast to the correct float size.

Here is an example from tensorflow-probability which demonstrates the issue:

adaptive_hmc = tfp.mcmc.SimpleStepSizeAdaptation(
    hmc, 
    num_adaptation_steps=10, 
    target_accept_prob=0.75,
    adaptation_rate=0.1
)

Compare the target_accept_prob and adaption_rate arguments. To actually get this to work, requires casting the target_accept_prob argument with something like "tf.cast(0.75, dtype=tf.float64)" but the adaption_rate argument is fine whether using float32 or float64. You need to dig into the source or wait for an exception and try to figure out which argument is causing it to know where such casts are required.

来源：https://stackoverflow.com/questions/60055919/gpflow-2-0-issue-with-default-float-and-likelihood-variance

标签

gpflow