Prepare multimodal scans by preserving their physical geometry, aligning them to a shared reference space, resampling images and labels with appropriate interpolation, and applying normalization that matches the model. The correct target grid and intensity pipeline depend on the modalities, anatomy, annotations, and model; there is no universal recipe.
What “aligned” means for multimodal segmentation
Two image arrays can have identical dimensions and still represent different anatomy in physical space. Conversely, scans with different array shapes may cover the same anatomy at different resolutions. Multimodal preprocessing must preserve and reconcile spatial metadata—orientation, voxel spacing, origin, and the relationship between voxel coordinates and patient coordinates—not just make the arrays the same size.
As an Amazon Associate I earn from qualifying purchases.
Choose one image or grid as the reference. Every other input, and every annotation, must be represented in that same physical frame before the model receives them as channels. If modalities were acquired at different times or with patient motion between scans, matching their metadata alone does not establish anatomical alignment: registration may be needed.
Build a reproducible preprocessing pipeline
1. Inventory each study and its annotations
Before conversion or registration, record what each series represents and how it will be used. Keep a stable case identifier so that a scan, its other modalities, and its labels cannot be accidentally mixed with another case or time point.
#1 Best Overall
- Record modality, sequence or contrast, acquisition, dimensions, voxel spacing, coordinate information, and physical coverage.
- Record which modality an annotation was drawn on and which anatomy or classes it marks.
- Decide whether scans from different visits belong together; do not combine time points unintentionally.
- Set and document a stable channel order for the model input.
2. Convert while preserving geometry
Convert the source data into a format accepted by the training or inference pipeline without discarding spatial metadata. The NIfTI FAQ identifies DICOM Pixel Spacing, Image Orientation (Patient), and Image Position (Patient) as relevant to reconstructing volume geometry, and notes that qform can store rigid alignment information. Because those fields and their interpretation matter to the conversion, inspect the resulting image rather than assuming that a plausible-looking volume is correct.
After conversion, check orientation and direction, spacing, origin or affine, slice ordering, and physical coverage. Array shape alone cannot reveal a left-right flip, incorrect slice ordering, or a mismatch in physical location.
3. Select a reference space and register when needed
Choose a reference modality and grid appropriate to the task and the model’s requirements. A MONAI Physio registration API describes the fixed image as the target coordinate system and maps the moving image into that frame. If the scans do not already share physical space, estimate a registration suited to the anatomy and acquisitions, review the result, and retain the transform.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsRegistration can be rigid, affine, or deformable. The choice is a task decision: consider cross-modality contrast, anatomy, motion, and whether local deformation is appropriate for the segmentation target. Do not treat any one registration type or reference modality as the universal default.
Keep annotations tied to their source image during this process. MONAI Physio’s documentation describes keeping masks and labelmaps in frame with the image through pre-warping. A label drawn on one modality must not be moved or resampled as though it were already in another modality’s space.
4. Define the target grid before resampling
Specify the target spacing, field of view, and grid based on the anatomy, acquired resolution, and model constraints. Ensure that the chosen grid covers the anatomy and labels needed for the task. Apply the same spatial transform chain to related images and labels so corresponding structures remain together.
Use interpolation appropriate to the data type:
- Intensity images: use a continuous interpolation method appropriate to the modality and task. Document the method and avoid unnecessary repeated resampling.
- Labelmaps and masks: use nearest-neighbor interpolation to preserve discrete class IDs. Other interpolation can introduce values that are not valid labels.
Keep the complete transform chain so that a model output can be mapped from its input grid back to the image’s native space when needed.
Recommended Free Tools
5. Normalize according to modality and model
Use explicit channel names and a stable channel order. Do not apply one normalization rule to every modality by habit: CT has a physical intensity scale, MRI sequences can vary, and PET values have modality-specific meaning. The preprocessing must also match the model’s documented expectations.
For nnU-Net v2, the channel_names entries determine normalization behavior: CT uses dataset-level foreground-based normalization, while other channel names default to per-case z-score normalization. Its documentation says normalization is channel-wise and that it has no built-in joint multichannel normalization scheme. Those are nnU-Net-specific behaviors, not rules for every architecture; confirm the configuration used for the actual model.
6. Meet the model’s input requirements
Check the exact model or bundle instructions for supported modalities, orientation, spacing, preprocessing, label schema, and output grid. For example, MONAI Physio’s NV-Segment-CTMR documentation lists CT_BODY, MRI_BODY, and MRI_BRAIN input modes. MRI_BRAIN expects a skull-stripped T1 volume affinely aligned to the LUMIR template; that preparation is not performed by the model itself.
The same documentation describes NV-Segment-CTMR as fine-tuned with “30K+ CT and MRI scans,” without identifying a publication year for that count. Treat it as the documentation’s description, not a dated independent benchmark. The weights are stated to be under NVIDIA’s OneWay Non-Commercial License. The page identifies NV-Segment-CT as a commercially licensed, CT-only alternative. Verify the current release, task fit, and license terms before use; the modality options and license distinction do not by themselves establish that either model suits a particular clinical or research task.
Quality-check the final model inputs
Inspect the data in the final model grid, not just in the original files. Compare modalities side by side and as overlays in multiple planes, and check annotation boundaries against the corresponding anatomy. A practical quality-control pass should include:
Best Value
- Quad-Screen Diagnostic Power - 2 pcs 36-inch crossbar supports four 21" displays simultaneously, enabling side-by-side PACS image comparison, EHR documentation, and real-time vital sign monitoring on a single mobile platform. Certified industrial-grade strength, tested to meet stringent ANSI/BIFMA X5.5-2021 standards
- Adjustable Monitor Angle - Fully motion mounts for holding 2 monitors that tilt 45° up and down & side to side rotate in 360°. Supports dual 21" horizontal monitors (VESA 75x75mm & 100x100mm compatible), easy to adjust the angle to fit your sight well
- Heavy Duty Workstation - This is more than just a home desk; it's a professional-grade workstation designed for durability and long-term security.Heavy duty aluminum that is wear and corrosion resistant. Each shelf has a maximum load capacity of 44lbs, providing you with a sturdy and stable working platform
- Complete Mobile Workstation - Includes adjustable keyboard tray, dedicated CPU holder, printer shelf, utility basket, and integrated power strip mount. Everything you need for a fully functional diagnostic station at the point of care
- Purpose-Built for Medical Environments - Designed for ORs, ICU/CCU, emergency departments, and radiology suites. 4 smooth-rolling Wheels for flexible mobility, 2 of which are lockable provide silent maneuverability and rock-solid stability when positioned for patient evaluation. Item may be shipped in multiple packages.
- left-right orientation and slice ordering;
- matching anatomical landmarks across modalities;
- complete coverage of the target anatomy and labels;
- registration failures or visibly implausible deformations;
- valid label IDs and boundaries after resampling;
- unexpected intensity ranges, missing data, or empty channels; and
- correct channel order at the model input.
Trace a small set of cases through native space, the common reference grid, the model tensor, and output reconstruction. Keep a transform ledger recording conversion, orientation handling, registration, resampling, cropping, normalization, channel order, and label handling. These are workflow controls inferred from the documented geometry and alignment requirements; the cited documentation does not define a universal QC checklist.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose fusion and preprocessing for the task
Preprocessing choices should serve the model’s fusion strategy and target, rather than forcing all inputs into a convenient but unjustified recipe. Relevant trade-offs include:
- Registration: balance robustness to cross-modality contrast and motion against the amount of geometric flexibility the task warrants.
- Reference grid: weigh the model’s assumptions against voxel spacing, physical coverage, and the modality most suitable as the fixed image.
- Interpolation: distinguish continuous image values from categorical labels, and minimize repeated resampling.
- Intensity handling: account for modality-specific scales, MRI sequence variability, PET quantification, and the model’s channel-specific normalization.
- Fusion: early or feature-level fusion combines modalities within the representation; intermediate or classifier-level fusion combines learned features or predictions later; late or decision-level fusion combines decisions. Each approach has different tolerance for missing or misaligned inputs.
- Deployment: confirm supported modalities, runtime needs, output label taxonomy, license, and any preparation required outside the model bundle.
One 2017 soft-tissue sarcoma study used PET, CT, T1-MR, and T2-MR. It rigidly registered modalities to transfer tumor annotations, cropped wider PET/CT coverage to the MR field of view, and linearly interpolated PET to match resolution. Feature-level fusion performed best overall in that experiment but was less robust to large errors in any modality. These are findings and preprocessing choices from that dataset, not general recommendations or a guarantee of performance elsewhere.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Tools and implementation scope
MONAI is described by NVIDIA as an open-source, freely available, community-supported PyTorch framework for healthcare imaging. Its Physio documentation provides examples of registration and model-specific input requirements. nnU-Net v2 documentation is useful for understanding that framework’s channel-dependent preprocessing. These tools do not remove the need to inspect converted geometry, verify alignment, and follow the chosen model’s current instructions.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
