Automated Multimodal Image Registration for Prostate Cancer Using Squeeze-and-Excitation ResNet with Thin Plate Spline Transformation: A Deep Learning Approach

Introduction

In 2024, an estimated 2 001 140 new cancer cases are expected to be diagnosed in the United States. Among men, prostate cancer is projected to have the highest incidence, with approximately 299 010 cases (29%), and the second-highest mortality rate, with about 35 250 deaths (11%) [1]. Currently, research in prostate cancer image registration focuses on correlating histopathological images with radiological images to characterize disease features and improve interpretation of radiological findings. Although prostate magnetic resonance imaging (MRI) offers detailed anatomical information, pathological interpretation based solely on MRI remains challenging. In contrast, histopathological images allow for more accurate identification of cancerous regions [2]. Therefore, integrating histopathological and radiological images facilitates both interpretation and computer-aided classification and segmentation of prostate cancer [3].

Traditional image registration based on intensity has some limitations, such as high requirements for initial estimation, inability to deal with non-uniform deformation, and sensitivity to noise, so it is difficult to achieve good results in multi-mode registration. To develop similarity measures for multimodal image registration, manual descriptors such as mutual information are proposed. To improve its performance, various variants of mutual information (MI) have been proposed, such as MI based on contextual conditioning [4], which incorporates spatial contextual information to enhance the robustness of similarity measurement, MI based on correlation ratios [5], and mode-independent neighborhood descriptors (MIND) [6]. However, it is very difficult to obtain well-aligned multimodal images for network training, and the quality of the images indirectly affects the accuracy of depth similarity measurement. Sedghi et al used special data enhancement techniques called jitter and symmetry to meet the need for well-aligned images in deep metric learning [7], and Wu et al proposed to use stacked auto-encoders (SAE) to learn intrinsic feature representations through unsupervised learning [8]. However, these methods are limited by high computational cost and time-consuming and significant physical gaps between consecutive histopathological sections, which complicates the establishment of continuous spatial correspondence with volumetric MRI data. Cao et al used thin-plate spline (TPS) for deformation registration of MRI scans [9], and Eppenhof et al used TPS for deformation registration of CT scans [10]. Neural networks have been used to predict parameters of the TPS model. Researchers have widely applied popular deep learning networks in image classification problems to solve the registration problem of prostate cancer MRI and tissue cases, mainly using VGG and ResNet [11,12]. Although these methods have achieved some improvement in time and accuracy through deep learning, the overall network architecture is not perfect, the feature recalibration at the channel level and the utilization of global information are lacking, and there are certain limitations in generalization and accuracy.

Neural networks have been used to predict parameters of the TPS model. To address the challenges of multimodal registration between prostate MRI and histopathological images, we developed an unsupervised Squeeze-and-Excitation ResNet with Thin-Plate Spline Transformation (SE-ResNet-TPS) framework. This method integrates 2 pivotal components: 1) Squeeze-and-Excitation (SE) modules [13] that enhance feature representation through adaptive channel-wise feature recalibration, enabling multi-scale convolutional processing of input data to prioritize anatomically significant regions; and 2) Thin-Plate Spline (TPS) transformation [14] that synergizes global deformation modeling with localized fine-grained adjustments, effectively balancing structural coherence and tissue-level registration precision. The framework eliminates dependence on labeled datasets while addressing the inherent modality discrepancy in MRI-histopathological alignment by using unsupervised learning on single-modal composite images with a customized anisotropy metric. The SE-ResNet backbone optimizes feature extraction through attention-driven spatial-channel interactions, enabling the subsequent TPS module to achieve superior approximation of ground-truth deformations.

Experimental validation demonstrates the framework’s clinical efficacy, attaining a Dice similarity coefficient (DSC) of 0.964 and Hausdorff distance (HD) of 2.91 mm for prostate contour alignment, with cancer-specific registration reaching 0.578 DSC and 4.97 mm HD. Comparative analysis against CNNGeometric [15] and VoxelMorph [16] reveals significant advances: For prostate registration, DSC improved by 1.9% and 4.0%, respectively, with corresponding HD reductions of 1.34 mm and 2.74 mm. In cancer regions, DSC increased by 2.1% and 4.1%, accompanied by HD improvements of 1.04 mm and 1.57 mm. These quantitative gains, coupled with visual assessments of pathological feature congruence, confirm the framework’s dual capability in preserving macroscopic anatomical structures while capturing microscale tumor morphology, establishing a robust solution for prostate cancer image-guided interventions.

Material and Methods

PRE-PROCESSING MODULE:

It is very difficult to register the histopathology of prostate and MRI images. Due to the large difference in storage between the histopathology and MRI, various preparation problems such as tissue rotation, contraction, deformation, and tearing can occur in preparation of resected tissue. Therefore, the overall shape of the prostate in the MRI and histopathology images is very different. To solve this series of problems, in the pre-processing module, we perform a 2-step process. First, the angle of the prostate pathology picture is manually processed to align it with MRI; second, the matched pathology picture and the area of interest are resampled to 224×224 to facilitate subsequent feature extraction [17].

SE-RESNET-TPS FEATURE EXTRACTION MODULE:

The feature extraction network of SE-ResNet-TPS is built based on convolution operation, which extracts information features by fusing spatial and channel information in the local receptive domain. Its core module is made up of ResNet152 scraps and SE modules [18].

According to CNN theory, convolution operators can fit any transformation, COP: K→O, K∈RC’×H’×W’, O∈RC×H×W, We take COP as a convolution operator that will input data, It converts input data K (size C’×H’×W’) into output data O (size C×H×W) [19]. Let K be the input of the SE-ResNet-TPS feature extraction module, K=[k1,k2,…kC’], Use E=[e1,e2,…eC] to describe the set of filter cores learned. Where eC is the parameter of the C filter, we can write the output O=[o1,o2,…oC], where

The symbol * represents convolution, ECS=[eC1,eC2,…eCC′], K=[k1,k2,…kC’]. OC∈RH×W. ECS is a two-dimensional space kernel, represents a single channel of eC, but it is entangled with the local spatial correlation captured by the filter [13]. The channel relationships in convolutional modeling are implicit and local in nature. The SE module captures the correlation between channels in the convolutional neural network to enhance the feature learning ability and improve its sensitivity to information features. The module achieves efficient extraction of global information and recalibrates the filter response through 2 steps of squeezing and excitation before passing to the next layer. This mechanism allows the network to better understand the interactions between different channels, thus improving the model’s perception of complex features, helping to improve its performance and generalization ability. We do this by introducing statistics: S=[s1,s2,…sC]∈RC is generated by shrinking O through the spatial dimension H×W, where

To take advantage of the information aggregated during the squeeze operation, we perform a second step to fully capture the channel dependencies. We use a fully connected neural network with 2 hidden layers to automatically learn nonlinear interactions and non-exclusive relationships between channels. The output of this fully connected neural network can be defined as

The delta as the ReLU function n[20], σ as the sigmoid function and W1∈RC%×C, W2∈RC%×C, C%=Cr for shrink ratio. We can write the final convolutional layer as, O%=[o1%,o2%,…,oC%] and a shortcut connection (SR) is a connection that skips 1 or more layers, allowing the gradient to propagate further and allowing efficient training of very deep networks [13]. Assuming that the input and output dimensions are the same, we can write the final output of the SE-ResNet-TPS feature extraction module,

oi% is obtained by channel-to-channel multiplication between the scalar si% and the feature map o_i where K_O is the initial feature map of the input and K^% is the final output feature map. During the training of each epoch, the loss function is calculated to measure the gap between the predicted result and the target result, and the back-propagation algorithm is used to obtain the loss value for updating the convolutional neural network weights. The gradient calculation formula of the SE-ResNet-TPS feature extraction module is defined as

This fast connection design can effectively solve the gradient disappearance problem common in CNN. The feature extraction module of SE-ResNet-TPS is unique in that it uses global average pooling operation in the squeeze phase, and 2 small fully connected layers in the excitation phase, followed by a channel scaling operation. Such a design not only improves the model’s perception of global information, but also helps capture more complex features. The deformation field is generated to realize the non-rigid deformation process, as shown in Figure 2.

PRO-TPS MODEL:

In the process of image registration, it is often necessary to identify a set of corresponding feature points to estimate their transformations. Traditional algorithms usually adopt specific detection methods to identify these feature points [21]. However, when there are fewer feature points, the effect of these algorithms may not be ideal. In addition, the accuracy of the detection method can also lead to unnecessary errors, which can affect the accuracy of the subsequent alignment model.

TPS registration is widely used in the field of medical image registration. Unlike rigid transformations, TPS registration can match alignment points more precisely [22]. In an environment with large parallax, if rigid transformation is used for image mosaic, artifacts may appear in the overlapping area. In this report, the parameters of TPS are provided by feature extraction network, and the TPS algorithm is used for registration. This method has 2 advantages: first, the features extracted by the feature extraction network enable more accurate TPS registration results; secondly, by using non-rigid transformation, the smoothness and global properties of TPS make it an ideal choice to deal with image deformation, and effectively overcome the limitations of linear interpolation in traditional registration methods.

The Pro-TPS registration module first accepts the relevant parameters derived from the SE-ResNet feature extraction network in the initialization stage – specifically, the learned spatial features and deformation priors encoded in the feature maps. These features are processed through 2 consecutive fully connected layers: the first layer projects the features into a latent space of dimension 2N (where N is the number of control points), encoding the (x, y) coordinates of the N control points; the second layer outputs N scalar weights corresponding to these control points. Thus, the SE-ResNet output is explicitly translated into TPS parameters – control point coordinates and their associated weights – via learnable linear transformations. To ensure meaningful and smooth deformations while preventing image folding or crumpling, a regularization term is incorporated into the loss function. Specifically, we use a bending energy penalty, defined as the L2 norm of the second derivatives of the deformation field, which penalizes abrupt changes in the transformation. This term is weighted by a coefficient λ (set to 0.01 in our experiments) and added to the intensity-matching loss (Equation 6), balancing alignment accuracy and deformation smoothness.

POST-PROCESSING MODULE:

After the TPS image registration, the estimated composite transformation is used to map the histopathological images onto the corresponding MRI sections. The original high-resolution histopathologic images are directly estimated by composite transformation, so that the morphed histopathologic images still maintain the same size as the original histopathologic images. For visualization, we remove sampling artifacts in the deformed image by applying a binary threshold and set the intensity of the pixels outside the prostate to zero.

LOSS FUNCTION:

To effectively register prostate MRI and histopathological images, we use CNNGeometric’s efficient loss function [15]. This loss function has a remarkable effect on TPS transformation, which is realized by learning the transformation parameters.

For image registration tasks, the image intensity difference is often used as a measure of similarity between 2 images. In unsupervised learning, the image intensity difference is used as part of the loss function, and the d function represents the difference measure between the 2 transform fields, using the mean square error. The parameter θ of the model corresponds to the generator of the transform field, which is used to transform the input image. Through the optimization process, the model learns to adjust the image so that it is more similar to the target image in image intensity. By integrating an image intensity difference measure into the loss function, the model can learn the registration relationship between images without supervision, without the need for explicit labels. The following formula is a specific loss function:

Results

DATASET:

We used an internal dataset from Beijing Hospital, which was collected from January 2020 to December 2022 and contains multi-parameter M0RI from 67 patients (age range: 45–75 years; sex distribution: 42 males and 25 females; primary diagnoses: prostate cancer stages T1–T3). These MRI scans are T2-weighted (T2w) because T2w MRI provides the highest soft tissue contrast required for comparison with histopathological images. In terms of histopathological images, this dataset is obtained through routine specimen preparation, pathological processing, and section staining [23,24], and finally obtained the histological sections stained with hematoxylin and eosin (H&E) for 67 patients. Anatomical area labels, specifically including prostate contours and cancerous regions, are manually annotated on T2-wMRI of 67 patients by 2 body imaging radiologists with more than 3 years of experience. These labels serve exclusively for post-training performance evaluation: they are used to compute the Dice similarity coefficient (DSC) and Hausdorff distance (HD) to quantify the alignment accuracy of prostate contours and cancerous regions, respectively. Importantly, these annotated labels are not involved in any stage of the model training process, consistent with the unsupervised nature of the SE-ResNet-TPS framework, which eliminates dependency on labeled datasets for learning. Cases where there was no overlap between the labels from the radiologists and the 2 pathologists in the same area were excluded due to inconsistencies.

EVALUATION METRICS:

DSC coefficient and HD are used to evaluate the alignment accuracy of the deformable histopathology and corresponding MRI. The degree of overlap between the prostate contour M and the corresponding histopathological images on T2w MRI is assessed by DSC coefficients. The DSC coefficient measures the relative overlap between M_A and M_B and is given by the following formula:

M_A is the deformed histopathological prostate mask, M_B is the corresponding MRI prostate mask, and is the base of the set (number of elements). HD is a distance measure that measures the similarity between MRI images and corresponding histopathological images.

The distance is expressed by this formula:

Where d(a,b) represents the distance between point a and point b while sup represents the upper certainty and inf represents the lower certainty.

Our method has a registration run time of 3–9 seconds on this dataset, depending on the resolution of the histopathological images and the number of sections in each case. Compared with other high-precision registration networks, our network consumes more time in registering the same data, but our accuracy is significantly improved, and this time cost is acceptable.

REGISTRATION RESULTS AND COMPARATIVE EXPERIMENTS:

To reflect the universality and versatility of the registration network, cancer regions of different sizes and results with obvious rotation angles that did not match during the process of MRI or pathological acquisition are selected and presented in the experimental results diagram [25].

The results showed that SE-ResNet-TPS achieved better prostate cancer boundary alignment than CNNGeometric and VoxelMorph. The accuracy of our analysis may be affected by inconsistencies between radiologists’ cancer labels and pathologists’ cancer labels, as well as landmark-based registration errors [2].

Figure 3 shows the registration results of different patients. The original histopathological image and the cancer area, the morphed histological image, and the final registration results of the histopathological image mapped to the MRI image, including the prostate area and the cancer area, are shown respectively. The subject’s prostate boundaries are well aligned on both MRI and histopathological slides, indicating that the SE-ResNet-TPS registration network achieved accurate global alignment of the prostate. Cancerous areas of the prostate are also well aligned on MRI and histopathological images. The precise alignment of the anatomic regions indicates that the SE-ResNet-TPS pipeline has achieved the alignment of local prostatic features. Our precise comparison of histopathological and MRI images shows that we can use estimated transformations to carefully map labels in histopathological images to the corresponding MRI sections.

Two cases are included: Column A shows the original histopathological images with the prostate cancer area marked in green; Column B shows the registered histopathological images; Column C shows the MRI with superimposed registered histopathological images; Column D shows the MRI with superimposed images of the cancer areas. The first row shows the SE-ResNet-TPS registration results for each case (top); the second row shows the CNNGeometric registration results (middle); the third row shows the VoxelMorph registration results (bottom).

Detailed comparative experimental results for the prostate region are presented in Figure 4 and Table 1. The DSC coefficients obtained by our SE-ResNet-TPS method are substantially higher than those obtained by CNNGeometric and VoxelMorph [15,16]. This is because we have introduced the SE module and TPS, which provide parameters for feature extraction in SE-ResNet-TPS.

Compared with CNNGeometric method, which uses affine transform and TPS, the SE module performs better in the deep learning network by introducing a channel attention mechanism. The network can be more flexible to focus on important feature channels, thus improving the model’s ability to perceive key information. Compared with the VoxelMorph registration network using affine registration, the registration method and network are significantly improved. In summary, the SE-ResNet-TPS registration network achieves better alignment near prostate boundaries than the CNNGeometric method and VoxelMorph. SE-ResNet-TPS achieves higher DSC coefficients than the CNNGeometric and VoxelMorph methods. The high DSC coefficient indicates that our SE-ResNet-TPS registration network can accurately align the overall shape and edges of the prostate on the dataset.

The results also show that the SE-ResNet-TPS registration network achieves lower HD than the CNNGeometric and VoxelMorph methods. The low HD means that our SE-ResNet-TPS registration network can have a small registration error of no more than 5 mm near the prostate boundary. Because we have deepened the depth of the feature extraction network, our registration network takes longer than with the CNNGeometric registration network, but compared with VoxelMorph method, our time advantage is even more significant.

Table 2 presents the DSC and Hausdorff distance comparing cancer labels from radiologists with those obtained from each registration method. The comparison through the chart shows that SE-ResNet-TPS has a higher DSC and better Hausdorff distance than CNNGeometric and VoxelMorph, indicating that our registration method achieves better alignment of cancer regions. The accuracy of our analysis may be affected by inconsistencies between the cancer labels from radiologists and those from pathologists.

ABLATION EXPERIMENT:

Table 3 is a comparison table of the experimental results of DSC coefficients, HD, and calculation time for the prostate region obtained from ablation experiments on this dataset. The baseline is ResNet152. The results show that the ProSE network designed by us can achieve higher registration accuracy and lower HD. The ProSE (probabilistic semantic embedding) network is a deep learning model that uses a probabilistic approach to represent semantic relationships in data. It generates embeddings that account for uncertainties, making it suitable for tasks with ambiguous data like natural language understanding.

SE-ResNet-TPS performed well in DSC coefficient, with the DSC in the prostate region reaching 0.964, compared to 0.902 to 0.957 for other models. This indicates that SE-ResNet-TPS achieved significant advantages in the image segmentation task, with the generated segmentation of the prostate region being closer to the real annotation. Through the ablation experiment, we found that by continually improving the base network and registration method of ResNet152, the SE-ResNet-TPS registration network we finally obtained shows superior registration results.

In terms of the experimental results for HD, it can be clearly seen that with the increase in module improvements, HD has significantly decreased. This is especially evident after the introduction of the TPS model, where HD has significantly decreased.

Regarding time performance, although our SE-ResNet-TPS registration network shows some increase in time compared to the base network, it does not exceed 1 order of magnitude and only increases by about 1 second. The extra time consumption is acceptable and is closely related to the lightweight design of the SE module and the simplicity of the TPS.

Discussion

The proposed SE-ResNet-TPS framework enhances the discriminative capacity and representational efficacy of feature extraction in prostate imaging applications. By integrating the Squeeze-and-Excitation (SE) module, the network dynamically recalibrates channel-wise feature responses, enabling focused attention on diagnostically relevant anatomical structures while suppressing noise and confounding artifacts inherent in prostate images. This adaptive feature selection mechanism improves the network’s ability to capture multi-scale pathological patterns, particularly in scenarios involving heterogeneous tissue textures or low-contrast regions. Furthermore, the incorporation of adaptive multi-scale convolution kernels allows the architecture to effectively handle spatial variations across different anatomical scales, facilitating precise localization of critical landmarks during registration. The subsequent TPS-based registration stage employs neural network-derived deformation parameters to perform nonlinear geometric alignment, where the smoothness-preserving nature of thin-plate spline interpolation maintains structural continuity while enabling fine-grained shape adjustments.

The streamlined parameterization framework enhances clinical translation efficiency by balancing computational complexity with diagnostic accuracy. This study elucidates the computational pipeline, network architecture, and validation methodology, demonstrating through extensive ablation studies that the proposed architecture achieves superior robustness compared to conventional methods. Quantitative evaluation reveals a statistically significant improvement in Dice similarity coefficient and reduced Hausdorff distance, validating its superiority in prostate image registration. Limitations of the current framework include computational overhead arising from architectural enhancements; specifically, increased network depth, bidirectional attention modules, and multi-scale feature fusion, while the semi-automated workflow necessitates manual correction steps for angular misalignment and axial inversion. Future work will focus on automated spatial normalization algorithms incorporating affine transformation parameter optimization to eliminate manual corrections and enable end-to-end registration of multi-parametric MRI datasets. Future studies should prioritize addressing the current limitations to further enhance clinical translatability. Specifically, efforts are needed to develop fully automated spatial normalization algorithms that eliminate manual correction steps for angular misalignment and axial inversion, using optimized affine transformation parameters to enable end-to-end registration of multi-parametric MRI datasets. Additionally, validating the framework on larger, multi-institutional cohorts with diverse pathological subtypes will be critical to assessing generalizability. Exploring hybrid loss functions that integrate both intensity matching and anatomical prior constraints could further improve deformation smoothness in cancer-specific regions. Finally, reducing computational overhead through model compression techniques while preserving registration accuracy remains a key direction for practical clinical use.

The proposed SE-ResNet-TPS registration framework achieves sub-voxel registration accuracy in prostate imaging, attaining a Dice similarity coefficient (DSC) of 0.964 and Hausdorff distance (HD) of 2.91. Compared with VoxelMorph and CNNGeometric methods, our network registration results are better, and ablation experiments have proved that our addition of various modules significantly improves registration. The SE-ResNet-TPS registration network provides a comprehensive and effective solution for multimodal prostate image registration tasks, capable of dealing with complex image features and shape changes, and our network significantly advances the field of prostate image registration imaging.

Conclusions

We propose a SE-ResNet-TPS registration framework to register preoperative MRI volume with prostate histopathological images. This method can be used to correlate disease features with radiological image features to support computer-assisted prostate cancer diagnosis and lesion annotation, and provide highly accurate prostate MRI and histopathological registration results for radiologists, thus improving prostate cancer detection on MRI.