Refactor decoding with minimal changes and reuse of existing pipeline

[QiongWang1]

Overview of all 6 major changes:

1. Validation Set - Added validation support with epoch-based reseeding
2. decode_distance_watershed - New decoding method with improved accuracy
   2.1 SDT Recomputation - Recomputes EDT before watershed for geometric precision
3. Accuracy Metric - Detailed instance-level detection assessment
4. ConstantLR Scheduler – Setup, but not currently in use at the end
5. Inference Debug Utils - Print-centric debugging for inference pipeline
6. Training Debug Utils - Global debug flag and normalization checking

Major Changes:

1. Validation Set Implementation

What Changed:

• Added comprehensive validation set support to the training pipeline
• Implemented validation dataloader with epoch-based reseeding
• Added automatic calculation of validation iteration count

Key Files:

• connectomics/training/lit/data.py
• connectomics/training/lit/config.py
• connectomics/training/lit/validation_callbacks/validation_reseeding.py

2. New Decoding Method: `decode_distance_watershed

What Changed:

• Implemented new watershed-based decoding function
• Uses recomputed SDT instead of raw predicted SDT values
• Recomputes SDT from foreground mask before watershed, rather than using network predictions directly

Key Files:

• connectomics/decoding/segmentation.py
• connectomics/inference/io.py

3. Accuracy Metric Implementation

What Changed:

• Enhanced accuracy metric computation with proper formula implementation
• Integrated into evaluation pipeline
• Accuracy = TP / (TP + FP + FN)

Key Files:

• connectomics/metrics/segmentation_numpy.py
• connectomics/metrics/metrics_seg.py

4. ConstantLR Scheduler Setup

What Changed:

• Set up infrastructure for Constant Learning Rate scheduler
• Configuration prepared but not actively used at BetaSeg

5. Training Debug Utilities

Location: connectomics/utils/debug_utils.py

What Changed:

• Created global debugging utilities for training pipeline
• Focus on normalization and value range tracking

Key Features:

• DEBUG_NORM = True: Global toggle for debug printing
• print_tensor_stats(): Print comprehensive tensor statistics at each training stage
• print_normalization_check(): Verify values are within expected normalization ranges

Why It Matters:

• Helps identify normalization bugs early
• Prevents silent errors in training pipeline
• Monitors value distributions throughout data flow
• Ensures data preprocessing is working correctly

6. Inference Debug Utilities

Location: connectomics/inference/debug_utils.py

What Changed:

• Created comprehensive debugging utilities for inference pipeline
• Print-centric design for easy troubleshooting

Key Features:

• analyze_array(): Print detailed array statistics (shape, dtype, min/max, unique values, distributions)
• analyze_h5_file(): Analyze HDF5 files used in inference
• save_as_nifti(): Export inference results as NIfTI format

Why It Matters:

• Assists with debugging inference issues
• Helps analyze intermediate results
• Enables easy export to data for visualization