Structuring a Keras project to achieve reproducible results in GPU

Question

I am writing a tensorflow.Keras wrapper to perform ML experiments.

I need my framework to be able to perform an experiment as specified in a configuration yaml file

Answer 1

Try adding seed parameters to weights/biases initializers. Just to add more specifics to Alexander Ejbekov's comment.

Tensorflow has two random seeds graph level and op level. If you're using more than one graph, you need to specify seed in every one. You can override graph level seed with op level, by setting seed parameter within function. And you can make two functions even from different graphs output same value if same seed is set. Consider this example:

g1 = tf.Graph()
with g1.as_default():
    tf.set_random_seed(1)
    a = tf.get_variable('a', shape=(1,), initializer=tf.keras.initializers.glorot_normal())
    b = tf.get_variable('b', shape=(1,), initializer=tf.keras.initializers.glorot_normal(seed=2))
with tf.Session(graph=g1) as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(a)) 
    print(sess.run(b))
g2 = tf.Graph()
with g2.as_default():
    a1 = tf.get_variable('a1', shape=(1,), initializer=tf.keras.initializers.glorot_normal(seed=1))

with tf.Session(graph=g2) as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(a1))

In this example, output of a is the same as a1, but b is different.

Answer 2

Keras + Tensorflow.

Step 1, disable GPU.

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = ""

Step 2, seed those libraries which are included in your code, say "tensorflow, numpy, random".

import tensorflow as tf
import numpy as np
import random as rn

sd = 1 # Here sd means seed.
np.random.seed(sd)
rn.seed(sd)
os.environ['PYTHONHASHSEED']=str(sd)

from keras import backend as K
config = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)
tf.set_random_seed(sd)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
K.set_session(sess)

Make sure these two pieces of code are included at the start of your code, then the result will be reproducible.