Skip to content

Pipeline and integration

MicroStructFormer is a standalone framework for fine-grained semantic segmentation of electron microscopy cross-sections, designed to model subcellular and periaxonal ultrastructural components with high spatial specificity. While the model can be applied independently to generate semantic label maps for its target structures, it also supports integration with complementary segmentation tasks when a broader ultrastructural context is required.

This page focuses on a common use case in which MicroStructFormer outputs are combined with axon–myelin segmentation provided by external tools, allowing joint analysis within a unified semantic label space.

Overview of the processing pipeline

At a high level, the analysis workflow consists of three complementary components:

  1. Subcellular and periaxonal segmentation (MicroStructFormer)
    MicroStructFormer processes electron microscopy images to segment mitochondria-related structures, periaxonal space, and morphologically abnormal or non-compact compartments.

  2. Axon–myelin segmentation (external tools)
    Axon and myelin masks are generated using a dedicated axon–myelin segmentation method, with AxonDeepSeg used as the default implementation in the current pipeline (see Citation).

  3. Post-processing and subsequent quantitative analysis
    Outputs from these segmentation tasks are geometrically aligned and combined to allow image-level and instance-level morphometric analysis.

This modular design allows individual components to be independently improved, replaced, or extended without redefining the overall analysis pipeline.

MicroStructFormer target structures

MicroStructFormer focuses on the semantic segmentation and modeling of ultrastructural compartments that are not explicitly addressed by axon–myelin segmentation tools, including:

  • Mitochondria inside axons
  • Mitochondria outside axons
  • Mitochondria-like organelles
  • Periaxonal space
  • Abnormal or non-compact structures

These structures are modeled as semantic classes using transformer-based segmentation architectures and are designed to be analyzed both at the image level and in association with individual axonal instances defined by external segmentation tools.

Example segmentation output

Axon–myelin segmentation

MicroStructFormer does not perform axon or myelin segmentation in its current implementation.
Instead, axon and myelin masks are provided by external segmentation tools, with AxonDeepSeg serving as the reference method used in this project.

These masks are used exclusively to define:

  • Axonal areas
  • Myelin areas

Users interested in axon or myelin segmentation methodology and performance should refer directly to the AxonDeepSeg documentation and associated publications.

Mask alignment and integration

In order to perform quantitative analysis jointly, predictions from AxonDeepSeg (ADS) and MicroStructFormer (MSF) are integrated into a unified semantic label mask through a set of operations. In the current workflow for reference, MSF provides fine-grained subcellular and periaxonal classes, while ADS provides axon and myelin regions. This section assumes that the predicted masks from both models are already available. Instructions for obtaining ADS and MSF outputs are provided in the Getting started section and the Citation page. The two outputs are then merged into a shared pixel space using deterministic priority rules.

Step 1. Remap ADS outputs into the unified label space

The native output of ADS uses grayscale intensity values to encode axon and myelin regions and therefore does not directly correspond to the unified semantic label space used in the combined mask. As a result, ADS masks are first remapped from their grayscale encodings into the unified label IDs used for downstream integration:

  • 255 → 6 (axon)
  • 127 → 7 (myelin)
  • all other values (usually there is only an existence of 0) → 0 (background)
import numpy as np
from skimage.io import imread, imsave
from skimage.transform import resize


def ads_to_labels(ads_path: str, target_size=None) -> np.ndarray:

    m = imread(ads_path)
    if m.ndim == 3:
        m = m[..., 0]

    out = np.zeros_like(m, dtype=np.uint8)
    out[m == 255] = 6
    out[m == 127] = 7

    if target_size is not None and out.shape[:2] != tuple(target_size):
        out = resize(out, target_size, order=0, preserve_range=True, anti_aliasing=False).astype(np.uint8)

    return out

Step 2. Load MSF predictions as label masks

MSF predictions are stored as integer label masks (typically containing labels 0~5 for the target structures modeled by MSF). Masks are loaded as single-channel uint8 arrays.

def load_label_mask(path: str) -> np.ndarray:

    m = imread(path)
    if m.ndim == 3:
        m = m[..., 0]
    return m.astype(np.uint8)

Step 3. Fuse masks with deterministic priority rules

The combined semantic mask is constructed using ADS as the base, and overlaying MSF predictions wherever MSF > 0. This priority rule is chosen to preserve fine-grained subcellular and periaxonal compartments when combining outputs from different segmentation tasks. If the two masks have different spatial resolutions, both are resized to a common target size using nearest-neighbor interpolation before merging.

def upsample_nearest(src: np.ndarray, target_hw) -> np.ndarray:
    if src.shape[:2] == tuple(target_hw):
        return src
    return resize(src, target_hw, order=0, preserve_range=True, anti_aliasing=False).astype(src.dtype)

def combine_semantic(ads_mask: np.ndarray, msf_mask: np.ndarray) -> np.ndarray:

    hA, wA = ads_mask.shape[:2]
    hM, wM = msf_mask.shape[:2]
    H, W = max(hA, hM), max(wA, wM)

    A = upsample_nearest(ads_mask, (H, W)).astype(np.uint8)
    M = upsample_nearest(msf_mask, (H, W)).astype(np.uint8)

    merged = np.where(M > 0, M, A).astype(np.uint8)
    return merged

Step 4.Minimal end-to-end example (single image)

Using the helper functions defined above and the saved prediction masks from ADS and MSF, a unified semantic mask can be generated for a single image as follows:

ads_path = "folder_for_your_masks/ads_seg-axonmyelin.png"
msf_path = "folder_for_your_masks/msf_pred_mask.png"
out_path = "folder_for_your_masks/merged_mask.png"

ads_mask = ads_to_labels(ads_path)
msf_mask = load_label_mask(msf_path)
merged = combine_semantic(ads_mask, msf_mask)

imsave(out_path, merged)

Instance-level association (semantic → fiber-centered instances)

Semantic-to-instance conversion schematic

Semantic masks identify classes, but they do not directly produce fiber-level instances or fiber-centric measurements. As illustrated above, to enable fiber-centered morphometrics, we apply a lightweight post-processing procedure that (i) separates fibers into distinct instances using a myelin-guided watershed, (ii) assigns surrounding compartments (myelin, PAS, abnormality) to their most plausible axon based on geodesic proximity, and (iii) links mitochondria to host axons using contact or nearest-distance rules.

This workflow converts semantic labels (0-7) into axon-centered instances:

import numpy as np
from skimage.io import imread
from skimage.morphology import binary_opening, remove_small_objects, dilation, disk
from skimage.segmentation import watershed, relabel_sequential
from skimage.measure import label
from scipy.ndimage import distance_transform_edt

def semantic_to_instance(mask_path: str):
    """Convert semantic mask to axon-centered instances."""
    # Load mask
    mask = imread(mask_path)
    if mask.ndim == 3:
        mask = mask[..., 0]

    # Extract semantic classes
    axon = binary_opening(mask == 6, disk(1))
    axon = remove_small_objects(axon, 80)
    myelin = binary_opening(mask == 7, disk(1))
    myelin = remove_small_objects(myelin, 60)
    pas = remove_small_objects(mask == 5, 10)
    abn = remove_small_objects(mask == 1, 10)

    # Step 1: Myelin-guided axon watershed
    barrier = dilation(myelin, disk(2))
    ax_core = axon & (~barrier)
    dist = distance_transform_edt(ax_core)

    cc = label(ax_core, connectivity=1)
    markers = np.zeros_like(ax_core, dtype=np.int32)
    mid = 0
    for cid in range(1, cc.max() + 1):
        if (cc == cid).sum() < 80:
            continue
        yx = np.argmax(dist * (cc == cid))
        y, x = np.unravel_index(yx, dist.shape)
        mid += 1
        markers[y, x] = mid

    axon_inst = watershed(-dist, markers, mask=ax_core)

    # Step 2: Merge axon pieces via PAS/abn connectivity
    domain = ((axon_inst > 0) | pas | abn) & (~myelin)
    comp = label(domain, connectivity=1)

    axon_merged = np.zeros_like(axon_inst)
    core_labels = np.zeros_like(axon_inst)
    new_id = 0
    for cid in range(1, comp.max() + 1):
        ax_mask = (comp == cid) & (axon_inst > 0)
        if ax_mask.any():
            new_id += 1
            axon_merged[ax_mask] = new_id
            core_labels[comp == cid] = new_id

    # Step 3: Assign myelin to nearest axon
    markers_my = np.zeros_like(axon_merged)
    for a in np.unique(axon_merged[axon_merged > 0]):
        rim = dilation(core_labels == a, disk(3))
        markers_my[rim & myelin] = int(a)

    myelin_part = watershed(np.zeros_like(axon_merged), 
                           markers_my, mask=myelin) if markers_my.any() else np.zeros_like(axon_merged)

    # Step 4: Direct assignment for PAS/abn
    pas_part = np.where(pas, core_labels, 0)
    abn_part = np.where(abn, core_labels, 0)

    return {
        'axon': axon_merged,
        'myelin': myelin_part,
        'pas': pas_part,
        'abn': abn_part
    }

Basic usage: 

from skimage.io import imsave
import matplotlib.pyplot as plt

result = semantic_to_instance("folder_for_your_masks/merged_mask.png")

axon_inst   = result["axon"]
myelin_part = result["myelin"]
pas_part    = result["pas"]
abn_part    = result["abn"]

Simplest visualization: 

import matplotlib.pyplot as plt
import numpy as np

def show_instances(label_map, title):
    plt.figure(figsize=(4,4))
    plt.imshow(label_map, cmap="tab20")
    plt.title(title)
    plt.axis("off")

show_instances(axon_inst, "Axon instances")
show_instances(myelin_part, "Myelin assignment")
show_instances(pas_part, "PAS assignment")
show_instances(abn_part, "Abnormality assignment")
plt.show()

The complete processing pipeline and visualization functions shown in the manuscript are available as a reference implementation in the Model Zoo section.

This fiber-centric representation enables direct computation of per-fiber metrics including compartment areas and organelle–myelin spatial relationships. Full algorithmic details can be found in the manuscript referenced on the Citation page.

Quantitative analysis and outputs

Following mask integration, MicroStructFormer produces semantic segmentation outputs that serve as inputs to downstream quantitative analysis.
Image-level and instance-level morphometrics are obtained through explicit post-processing procedures applied to the predicted masks.

Typical downstream outputs include:

  • Per-image area fractions for each compartment
  • Per-instance morphometrics (e.g. fiber-associated mitochondrial content)
  • Spatial relationships between organelles and myelin-associated geometry
  • Summary tables formatted for use in statistical analysis workflows

These post-processing steps are implemented as separate analysis utilities and are designed to integrate with common scientific Python toolchains.

Extensibility and alternative segmentation tools

Although AxonDeepSeg is used as the default axon–myelin segmentation method in the current pipeline, MicroStructFormer is fully compatible with any axon–myelin segmentation tool. Alternative methods may be substituted, provided they generate compatible axon and myelin masks.

This design ensures that MicroStructFormer remains extensible as segmentation methodologies and experimental contexts evolve.

Citation and attribution

When MicroStructFormer is used together with external axon–myelin segmentation tools, appropriate citation of each component is required.
MicroStructFormer should be cited for subcellular segmentation, mask integration, and quantitative analysis, while axon–myelin segmentation tools such as AxonDeepSeg should be cited for axonal and myelin delineation.

See the Citation page for recommended references and citation formats.