问题
I'm looking to apply an affine transformation, defined in homogeneous coordinates on images of different resolutions, but I encounter an issue when one ax is of different resolution of the others.
Normally, as only the translation part of the affine is dependent of the resolution, I normalize the translation part by the resolution and apply the corresponding affine on the image, using scipy.ndimage.affine_transform.
If the resolution of the image is the same for all axes, it works perfectly, you can see below images of the same transformation (being a scale+translation, or rotation+translation, see code below) being applied to an image, its downsampled version (meaning at a lower resolution). Images match (almost) perfectly, differences in voxel values are mainly caused as far as I know by interpolation errors.
But you can see that the shapes overlay between the downsampled transformed image and the transformed (downsampled for comparison) image
Scale affine transformation applied on the same image, at two different (uniform) resolutions
Rotation affine transformation applied on the same image, at two different (uniform) resolutions
Unfortunately, if one of the image axis has a different resolution than the other (see code below), it works well with affine transform with null non-diagonal terms (like translation, or scaling) but the result of the transformation gives a completely wrong result.
Rotation affine transformation applied on the same image, at two different (non-uniform) resolutions
Here you can see a minimal working example of the code:
import numpy as np
import nibabel as nib
from scipy.ndimage import zoom
from scipy.ndimage import affine_transform
import matplotlib.pyplot as plt
################################
#### LOAD ANY 3D IMAGE HERE ####
################################
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% TO BE DEFINED BY USER
orig_img = np.zeros((100, 100, 100))
orig_img[25:75, 25:75, 25:75] = 1
ndim = orig_img.ndim
################################
##### DEFINE RESOLUTIONS #######
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% TO BE DEFINED BY USER
# Comment/uncomment to choose the resolutions (in mm) of the images
# ORIG_RESOLUTION = [1., 1., 1.]
# TARGET_RESOLUTION = [2., 2., 2.]
ORIG_RESOLUTION = [1., 0.5, 1.]
TARGET_RESOLUTION = [2., 2., 2.]
#####################################
##### DEFINE AFFINE TRANSFORM #######
affine_scale_translation = np.array([[1.5, 0.0, 0.0, -25.],
[0.0, 0.8, 0.0, 0. ],
[0.0, 0.0, 1.0, 0. ],
[0.0, 0.0, 0.0, 1.0]])
a = np.sqrt(2)/2.
affine_rotation_translation = np.array([[a , a , 0.0, -25.],
[-a , a , 0.0, 50. ],
[0.0, 0.0, 1.0, 0.0 ],
[0.0, 0.0, 0.0, 1.0]])
# #%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% TO BE DEFINED BY USER
# Comment/uncomment to choose the transformation to be applied
# affine_tf, name_affine = affine_scale_translation, "Tf scale"
affine_tf, name_affine = affine_rotation_translation, "Tf rotation"
######################################################
######## DOWNSAMPLE IMAGE TO LOWER RESOLUTION ########
######################################################
downsample_img = zoom(orig_img,
zoom=np.array(ORIG_RESOLUTION)/np.array(TARGET_RESOLUTION),
prefilter=False, order=1)
##############################################################################
######## APPLY AFFINE TRANSFORMATION TO ORIGINAL AND DOWNSAMPLE IMAGE ########
##############################################################################
affine_st_full_res, affine_st_low_res = affine_tf.copy(), affine_tf.copy()
# Inverse transform as affine_transform apply the tf from the target space to the original space
affine_st_full_res, affine_st_low_res = np.linalg.inv(affine_st_full_res), np.linalg.inv(affine_st_low_res)
# Normalize translation part (normally expressed in millimeters) for the resolution
affine_st_full_res[:ndim, ndim] = affine_st_full_res[:ndim, ndim] / ORIG_RESOLUTION
affine_st_low_res[:ndim, ndim] = affine_st_low_res[:ndim, ndim] / TARGET_RESOLUTION
# Apply transforms on images of different resolutions
orig_tf_img = affine_transform(orig_img, affine_st_full_res, prefilter=False, order=1)
downsample_tf_img = affine_transform(downsample_img, affine_st_low_res, prefilter=False, order=1)
# Downsample result at full resolution to be compared to result on downsample image
downsample_orig_tf_img = zoom(orig_tf_img, zoom=np.array(
ORIG_RESOLUTION)/np.array(TARGET_RESOLUTION),
prefilter=False, order=1)
# print(orig_img.shape)
# print(downsample_img.shape)
# print(orig_tf_img.shape)
# print(downsample_orig_tf_img.shape)
###############################
######## VISUALISATION ########
###############################
# We'll visualize in 2D the slice at the middle of the z (third) axis of the image, in both resolution
mid_z_slice_full, mid_z_slice_low = orig_img.shape[2]//2, downsample_img.shape[2]//2
fig, ((ax1, ax2, ax3), (ax4, ax5, ax6)) = plt.subplots(nrows=2, ncols=3)
ax1.imshow(orig_img[:, :, mid_z_slice_full], cmap='gray')
ax1.axis('off')
ax1.set_title('1/ Origin image, at full res: {}'.format(ORIG_RESOLUTION))
ax2.imshow(downsample_img[:, :, mid_z_slice_low], cmap='gray')
ax2.axis('off')
ax2.set_title('2/ Downsampled image, at low res: {}'.format(TARGET_RESOLUTION))
ax3.imshow(downsample_tf_img[:, :, mid_z_slice_low], cmap='gray')
ax3.axis('off')
ax3.set_title('3/ Transformed downsampled image')
ax4.imshow(orig_tf_img[:, :, mid_z_slice_full], cmap='gray')
ax4.axis('off')
ax4.set_title('4/ Transformed original image')
ax5.imshow(downsample_orig_tf_img[:, :, mid_z_slice_low], cmap='gray')
ax5.axis('off')
ax5.set_title('5/ Downsampled transformed image')
error = ax6.imshow(np.abs(downsample_tf_img[:, :, mid_z_slice_low] -\
downsample_orig_tf_img[:, :, mid_z_slice_low]), cmap='hot')
ax6.axis('off')
ax6.set_title('Error map between 3/ and 5/')
fig.colorbar(error)
plt.suptitle('Result for {} applied on {} and {} resolution'.format(name_affine, ORIG_RESOLUTION, TARGET_RESOLUTION))
plt.tight_layout()
plt.show()
回答1:
The downsampling basically changes the basis vectors of the image. This makes the downsampling act like a translation.
But rotation and translation are not commutative (not interchangeable).
https://gamedev.stackexchange.com/questions/16719/what-is-the-correct-order-to-multiply-scale-rotation-and-translation-matrices-f
So after your downsampling you have to translate the content of the image back to the original position. Which can be done by composing a affine transformation matrix using ORIG_RESOLUTION and applying it before the actual transformation.
Or use a matrix multipication (e.g. with np.dot) to modify the actual transformation.
来源：https://stackoverflow.com/questions/59846059/behaviour-of-affine-transform-on-3d-image-with-non-uniform-resolution-with-scipy