Instance Segmentation · Noisy Labels · Benchmark & Method
Sfumato Supervision

Learning Instance Segmentation with Noisy Labels

1 Technion – Israel Institute of Technology 2 Ben-Gurion University of the Negev

* Corresponding author: moshekimhi@cs.technion.ac.il

Natural objects rarely have razor-sharp binary outlines.
Keep the object core crisp. Model contested contours “in the manner of smoke.”

Inspired by Leonardo da Vinci’s sfumato technique—blending transitions “without lines or borders, in the manner of smoke”—we audit 24,809 overlapping COCO–LVIS images (showing that 47% of crowdsourced masks suffer severe contour imprecision), establish a unified benchmark suite spanning human, foundation-model (SAM2), and procedural noise, and introduce CB-LNL (Copula-Boundary-LNL) to couple directional inner-core (pin) and inverse-exterior (1 − pout) evidence along object boundaries.

24,809 overlapping COCO–LVIS images audited (59,840 object pairs)
47.0% of human COCO masks suffer contour noise (BIoU < 0.75)
0.956 contour error-detection AUROC via Fréchet copula
+2.23 AP downstream Mask AP gain from automated dataset cleaning
01 / Interactive Annotation Noise & Repair Explorer (8 Real Scenes)

See what coarse polygons miss.
Drag the curtain to repair them.

Choose any of the 8 real scenes below (7 MS-COCO scenes + 1 high-resolution 1080p VIPER driving scene) and switch the Left Curtain between crowdsourced human polygons (COCO-RealNoise), foundation-model weak prompts (SAM2 Box / Click), procedural stress tiers (COCO-N / VIPER-N), or isolated geometric operators (coral highlights indicate contour errors). Drag the split curtain to compare against our CB-LNL Cleaned repair or the Clean Reference on the right.

Scene (8):
Right Curtain:
Left Curtain — Benchmark Supervision Regimes Coral = Contour Discrepancy
Left Curtain — Isolated Spatial Operators Single-Factor Corruptions
Right comparison annotation Left noisy annotation
Left: Polygon Simplification · BIoU 56.6%
Right: CB-LNL Cleaned · BIoU 95.6%
52% / 48%
Live Contour Telemetry

Scene 1: Cat, Potted Plant & TV (COCO #37907)

Standard Mask IoU stays deceptively high because interior pixels dominate area, whereas Boundary IoU (BIoU) exposes severe contour degradation.

Polygon Simplification
56.6%
Boundary IoU (Mask IoU: 89.8%)
CB-LNL Cleaned
95.6% (+39.0%)
Boundary IoU (Mask IoU: 99.1%)
Isolated Polygon Simplification: Ramer–Douglas–Peucker corner cutting removes concavities and limbs. Coral highlights show inward shortcuts healed by our Foreground Continuation Copula Cλ(pin, pout).
02 / Quantitative Benchmark & Multi-Seed Verification

Rigorous evaluation across seeds,
frozen predictions, and dataset cleaning.

All models are evaluated against the strictly fixed LVIS-verified Clean Reference validation split (val_clean_lvis_ref.json). Switch between the four experimental suites below to inspect our 3-seed paired comparisons, frozen-prediction scoring controls, controlled cleaning vs. direct teacher-mask replacement, and multi-architecture vulnerability.

Table 3 — Comparison of LNL Baselines vs. Boundary-LNL and CB-LNL (Single-Seed Across All 6 Regimes & 3-Seed Paired Repeat on COCO-RealNoise and COCO-N Medium · ResNet-50-FPN)
Method RealNoise (Mask / Bnd) WAN-Box (Mask / Bnd) WAN-Point (Mask / Bnd) COCO-N Easy (Mask / Bnd) COCO-N Med (Mask / Bnd) COCO-N Hard (Mask / Bnd)
Standard (CE + BCE) 30.45 / 17.05 28.63 / 16.57 26.21 / 15.04 29.09 / 16.29 26.70 / 14.34 24.09 / 10.48
SCE / GCE 29.52 / 30.01 (Bnd: 16.40) 27.17 / 28.53 (Bnd: 16.29) 25.30 / 25.94 (Bnd: 15.26) 28.91 / 29.68 (Bnd: 16.86) 26.22 / 27.21 (Bnd: 14.34) 23.16 / 23.07 (Bnd: 10.13)
DivideMix-Cls / DivideMix-Mask 27.06 / 28.32 (Bnd: 15.33) 25.10 / 27.44 (Bnd: 15.53) 22.75 / 25.78 (Bnd: 14.99) 27.24 / 29.40 (Bnd: 16.57) 25.25 / 26.01 (Bnd: 13.92) 21.92 / 23.39 (Bnd: 10.39)
Spatial Self-Correction 29.87 / 16.53 27.69 / 15.68 25.32 / 14.28 28.91 / 15.90 26.82 / 14.47 23.51 / 10.71
Boundary-LNL (Ours, Seed 1) 30.54 / 17.19 28.50 / 16.71 25.97 / 15.16 29.68 / 16.86 28.30 / 15.48 24.97 / 11.45
CB-LNL (Ours, Seed 1) 30.87 / 17.22 28.14 / 16.48 26.07 / 15.78 30.01 / 17.19 28.45 / 15.33 25.62 / 11.25
Standard (3-Seed Mean ± Std) 30.19±0.26 / 16.87±0.17 — — — 27.48±0.83 / 14.60±0.28 —
Boundary-LNL (3-Seed Mean ± Std) 30.85±0.34 / 17.44±0.22 — — — 28.12±0.16 / 15.31±0.17 —
CB-LNL (3-Seed Mean ± Std) 30.54±0.35 / 17.25±0.08 — — — 28.57±0.27 / 15.37±0.04 —
Paired Δ (CB-LNL − Boundary-LNL) −0.31±0.56 / −0.19±0.23 — — — +0.44±0.38 (3/3+) / +0.06±0.20 (2/3+) —
03 / Hugging Face Datasets & Quickstart Guide

Download the annotations.
Plug into any COCO pipeline in 3 lines.

All benchmark annotations—including COCO-RealNoise, COCO-WAN (SAM2), COCO-N (Easy/Medium/Hard across 3 seeds), all 13 single-factor disentanglement splits, CB-LNL Cleaned annotations, and VIPER-N—are hosted on Hugging Face in standard MS-COCO JSON format.

Hugging Face Dataset · 37 JSON Splits

Kimhi/coco-n

Complete MS-COCO noisy-label suite: coco_realnoise/ (LVIS Clean Ref + Real Noisy), coco_wan/ (4 SAM2 prompt regimes), coco_n/ (Easy/Med/Hard × 3 seeds), coco_n_ablations/ (13 single-factor splits), and cb_lnl_cleaned/.

Open Kimhi/coco-n on Hugging Face ↗
Hugging Face Dataset · Driving Noise

Kimhi/viper-n

High-resolution (1920×1080) urban driving noisy instance segmentation benchmark: viper_n/easy_seed1.json, medium_seed1.json, hard_seed1.json, plus aligned train_clean.json and val_clean.json.

Open Kimhi/viper-n on Hugging Face ↗
Hugging Face Dataset · Driving Images + GT

Kimhi/viper

Full VIPER driving image archives (images/train, images/val) packaged together with clean COCO-format instance segmentation annotations. Pair directly with Kimhi/viper-n.

Open Kimhi/viper on Hugging Face ↗
1. Download Original MS-COCO 2017 Images & LVIS Reference
# 1. Download official MS-COCO 2017 images (cocodataset.org)
mkdir -p datasets/coco/images && cd datasets/coco/images
wget http://images.cocodataset.org/zips/train2017.zip
wget http://images.cocodataset.org/zips/val2017.zip
unzip -q train2017.zip && unzip -q val2017.zip

# 2. (Optional) Official LVIS v1.0 annotations (lvisdataset.org)
wget https://dl.fbaipublicfiles.com/LVIS/lvis_v1_train.json.zip
wget https://dl.fbaipublicfiles.com/LVIS/lvis_v1_val.json.zip
2. Load Any Noisy or CB-LNL Cleaned Split in Python
from huggingface_hub import snapshot_download
from pycocotools.coco import COCO

# Download all benchmark annotations from Hugging Face
root = snapshot_download("Kimhi/coco-n", repo_type="dataset")

# Drop-in replacements for instances_train2017.json:
coco_real  = COCO(f"{root}/coco_realnoise/train_real_noisy_coco.json")
coco_wan   = COCO(f"{root}/coco_wan/coco_wan_box_noisy.json")
coco_med   = COCO(f"{root}/coco_n/medium_seed1.json")
coco_clean = COCO(f"{root}/cb_lnl_cleaned/medium_seed1_copula_cleaned.json")

# Fixed LVIS-verified Clean Reference validation split:
coco_val   = COCO(f"{root}/coco_realnoise/val_clean_lvis_ref.json")
04 / Interactive Sfumato Boundary Simulator

How directional copulas separate soft true edges from contour errors.

On a natural object contour, even a clean pixel sits near p(u) ≈ 0.50, confounding any single-pixel boundary loss. CB-LNL instead couples interior foreground (pin) and inverse-exterior background (1 − pout) via the Fréchet copula Cλ(a, b) = (1 − λ)ab + λ min(a, b):

Contour State Presets

Select a Boundary Regime on Bd

Live Copula Output

1D Pointwise BCE vs. 2D Directional Copulas

1D Single-Pixel Boundary BCE −log p(u) 0.71 nats (floor ≈ 0.69)
State 1 — Contrastive Edge Cλ(pin, 1 − pout) 0.87
State 2 — Foreground Continuation Cλ(pin, pout) (Cut) 0.08
State 3 — Inverse-Background Cλ(1 − pin, 1 − pout) (Bleed) 0.08
05 / Reference This Work

Citation

If you find our empirical audit, benchmark datasets (COCO-RealNoise, COCO-WAN, COCO-N, VIPER-N), or CB-LNL (Sfumato Supervision) useful in your research, please cite our paper:

@article{kimhi2026sfumato,
  title   = {Sfumato Supervision: Learning Instance Segmentation with Noisy Labels},
  author  = {Kimhi, Moshe and Rivlin, Ehud and Baskin, Chaim},
  journal = {arXiv preprint arXiv:2406.10891},
  url     = {https://arxiv.org/abs/2406.10891},
  year    = {2026}
}