Interpolated sampling of points in an image with TensorFlow

Question

Given is a grayscale image I as 2D Tensor (Dimension W,H) and a Tensor of coordinates C (Dim. None,2). I want to interpret the rows of

Answer 1

There is no built-in op that performs this kind of interpolation, but you should be able to do it using a composition of existing TensorFlow ops. I'd suggest the following strategy for the bilinear case:

1. From your tensor C of indices, compute integer tensors corresponding to the four corner points. For example (with names assuming that the origin is at the top left):

   top_left = tf.cast(tf.floor(C), tf.int32)
   
   top_right = tf.cast(
       tf.concat(1, [tf.floor(C[:, 0:1]), tf.ceil(C[:, 1:2])]), tf.int32)
   
   bottom_left = tf.cast(
       tf.concat(1, [tf.ceil(C[:, 0:1]), tf.floor(C[:, 1:2])]), tf.int32)
   
   bottom_right = tf.cast(tf.ceil(C), tf.int32)
   

2. From each tensor representing a particular corner point, extract a vector of values from I at those points. For example, for the following function does this for the 2-D case:

   def get_values_at_coordinates(input, coordinates):
     input_as_vector = tf.reshape(input, [-1])
     coordinates_as_indices = (coordinates[:, 0] * tf.shape(input)[1]) + coordinates[:, 1]
     return tf.gather(input_as_vector, coordinates_as_indices)
   
   values_at_top_left = get_values_at_coordinates(I, top_left)
   values_at_top_right = get_values_at_coordinates(I, top_right)
   values_at_bottom_left = get_values_at_coordinates(I, bottom_left)
   values_at_bottom_right = get_values_at_coordinates(I, bottom_right)
   

3. Compute the interpolation in the horizontal direction first:

   # Varies between 0.0 and 1.0.
   horizontal_offset = C[:, 0] - tf.cast(top_left[:, 0], tf.float32)
   
   horizontal_interpolated_top = (
       ((1.0 - horizontal_offset) * values_at_top_left)
       + (horizontal_offset * values_at_top_right))
   
   horizontal_interpolated_bottom = (
       ((1.0 - horizontal_offset) * values_at_bottom_left)
       + (horizontal_offset * values_at_bottom_right))
   

4. Now compute the interpolation in the vertical direction:

   vertical_offset = C[:, 1] - tf.cast(top_left[:, 1], tf.float32)
   
   interpolated_result = (
       ((1.0 - vertical_offset) * horizontal_interpolated_top)
       + (vertical_offset * horizontal_interpolated_bottom))
   

Answer 2

This turned out to be tricky for nearest neighbor given that TF doesn't have Numpy slicing generality yet (github issue #206), and the fact that gather only works on first dimension. But here's a way to work around it by using gather->transpose->gather->extract diagonal

def identity_matrix(n):
  """Returns nxn identity matrix."""
  # note, if n is a constant node, this assert node won't be executed,
  # this error will be caught during shape analysis 
  assert_op = tf.Assert(tf.greater(n, 0), ["Matrix size must be positive"])
  with tf.control_dependencies([assert_op]):
    ones = tf.fill(n, 1)
    diag = tf.diag(ones)
  return diag

def extract_diagonal(tensor):
  """Extract diagonal of a square matrix."""

  shape = tf.shape(tensor)
  n = shape[0]
  assert_op = tf.Assert(tf.equal(shape[0], shape[1]), ["Can't get diagonal of "
                                                       "a non-square matrix"])

  with tf.control_dependencies([assert_op]):
    return tf.reduce_sum(tf.mul(tensor, identity_matrix(n)), [0])


# create sample matrix
size=4
I0=np.zeros((size,size), dtype=np.int32)
for i in range(size):
  for j in range(size):
    I0[i, j] = 10*i+j

I = tf.placeholder(dtype=np.int32, shape=(size,size))
C = tf.placeholder(np.int32, shape=[None, 2])
C0 = np.array([[0, 1], [1, 2], [2, 3]])
row_indices = C[:, 0]
col_indices = C[:, 1]

# since gather only supports dim0, have to transpose
I1 = tf.gather(I, row_indices)
I2 = tf.gather(tf.transpose(I1), col_indices)
I3 = extract_diagonal(tf.transpose(I2))

sess = create_session()
print sess.run([I3], feed_dict={I:I0, C:C0})

So starting with a matrix like this:

array([[ 0,  1,  2,  3],
       [10, 11, 12, 13],
       [20, 21, 22, 23],
       [30, 31, 32, 33]], dtype=int32)

This code extracts diagonal one above the main

[array([ 1, 12, 23], dtype=int32)]

There's some magic happening with [] operators getting turned into Squeeze and Slice