Computer Vision

CNN-Based Object Detection

Object detection localizes and classifies multiple objects in images via bounding boxes. Across roughly six years (2014-2020) the literature settled into five overlapping families; two-stage refinement, single-shot dense prediction, focal-loss balanced one-stage, anchor-free per-pixel and keypoint heads, and set-prediction transformers; each occupying a different point on the accuracy/speed/simplicity frontier. Each family also addressed a characteristic set of well-known problems (scale, imbalance, NMS heuristics, anchor design, slow convergence, etc.) walked through paragraph-by-paragraph after the SoTA leaderboard below.

The historical context matters because the same problems recur under new names. Pre-deep-learning detection plateaued near 33% mAP on PASCAL VOC under HOG-feature pipelines¹ (Dalal and Triggs 2005) and the deformable part model (Felzenszwalb, McAllester, and Ramanan 2008) (a sliding-window root + parts mixture trained with latent-SVM that defined the pre-CNN state of the art for almost a decade); R-CNN (Girshick et al. 2014) (2014) lifted it to 53.3% by reusing ImageNet (Deng et al. 2009) pretrained CNN features on selective-search proposals, but at ~49 seconds per image. Fast R-CNN (Girshick 2015) (2015) shared backbone computation across proposals for a 213$\times$ speedup; Faster R-CNN (Ren et al. 2015) (2015) replaced the external proposal stage with a learned Region Proposal Network. Single-shot detectors emerged in parallel: YOLO (Redmon et al. 2016) (2016) reframed detection as direct regression on a 7$\times$7 grid at 45 FPS, SSD (W. Liu et al. 2016) (2016) added multi-scale feature maps. RetinaNet (2017) closed the one-stage/two-stage accuracy gap with focal loss (Lin, Goyal, et al. 2017), while FPN (Lin, Dollár, et al. 2017) became the universal neck. Anchor-free methods (CornerNet (Law and Deng 2018), FCOS (Tian et al. 2019), CenterNet (Zhou, Wang, and Krähenbühl 2019)) eliminated the anchor-design hyperparameters in 2019; DETR (Carion et al. 2020) removed NMS via Hungarian matching in 2020 (covered in the transformer chapter).

The five families differ on which problems they accept and which they engineer away. Two-stage detectors handle the foreground/background imbalance via balanced sampling at the RoI head, but inherit a complex multi-stage pipeline and slow inference. One-stage detectors face the imbalance directly through hard mining (SSD), focal loss (RetinaNet), or implicit balanced anchors (YOLO grid). Anchor-free heads sidestep the entire scale-and-aspect anchor-tuning enterprise but still need a label-assignment heuristic (smallest-area box, centerness, etc.). Set-prediction transformers bypass NMS at the cost of 500-epoch training. No family dominates: the COCO leaderboard top is a moving boundary between Cascade-style two-stage refinement, EfficientDet-style compound-scaled one-stage, and DINO-DETR-style transformers, and the production-grade default depends on whether latency, throughput, or peak accuracy is the primary constraint²³.

The leaderboard separates six eras and a speed column whose meaning differs from the accuracy columns. Reading the table from left to right gives the family-by-family progression (R-CNN $\to$ one-stage $\to$ Focal $\to$ anchor-free $\to$ compound-scaled $\to$ transformer); reading top-to-bottom by Year reveals the COCO AP trajectory from 19.7 (Fast R-CNN, 2015) to 66.0 (Co-DETR, 2023), a 3.4$\times$ gain spread across backbone scaling⁴, pyramid sophistication⁵, and head redesign. Three diagnostics. First, AP_S almost always lags AP_M and AP_L by 15-25 points, the single most persistent failure mode of CNN detectors⁶. Second, FPS within a family is reliable but cross-family comparisons span four GPU generations⁷. Third, the gap between best two-stage and best one-stage inverted between 2018 and 2020, eliminating the historical "two-stage = accuracy, one-stage = speed" dichotomy⁸.

The two-stage R-CNN family defines the era. R-CNN ran 2000 selective-search proposals through an AlexNet/VGG forward pass each, so per-image inference at ~49s was untenable; Fast R-CNN shared a single backbone pass and replaced the per-class SVM cascade with a softmax + smooth-L1 multi-task head, while Faster R-CNN folded proposals into a learned RPN sliding 9 hand-tuned anchors per location⁹. Mask R-CNN extended the same scaffolding with a per-class binary mask branch and replaced quantising RoIPool with bilinear RoIAlign; the pixel-precise alignment matters far more at strict IoU thresholds¹⁰. Cascade R-CNN refines proposals through three heads at IoU thresholds 0.5/0.6/0.7, exploiting the quality-mismatch observation that a single head cannot simultaneously be accurate at all IoU regimes; this is also where label assignment becomes structurally explicit¹¹.

The single-shot SSD/YOLO line trades the proposal stage for direct dense prediction. YOLO partitioned the image into a 7$\times$7 grid where each cell predicts $B=2$ boxes plus class probabilities at 45 FPS but only 63.4 VOC mAP, paying for speed in localization (19% of YOLO's error is localization vs Fast R-CNN's 8%); SSD recovered accuracy by attaching prediction heads to multiple feature-map resolutions (38$\times$38 through 1$\times$1) so that small objects map to high-resolution layers and large objects to coarse layers, a structurally different solution to scale variance¹². SSD also introduced 3:1 hard negative mining as its blunt-instrument response to foreground/background imbalance¹³, and depends notoriously hard on data augmentation (74.3 $\to$ 65.5 mAP without augmentation). YOLOv3 added k-means anchor priors and FPN-style three-scale prediction; YOLOv4 split engineering into "Bag of Freebies" (Mosaic, CIoU, label smoothing) and "Bag of Specials" (Mish, DIoU-NMS, PANet neck), reaching 43.5 COCO AP at 65 FPS by combining many small recipes rather than one architectural insight.

RetinaNet + FPN closed the one-stage/two-stage accuracy gap with two orthogonal contributions. Focal loss $-(1-p_t)^\gamma \log p_t$ with $\gamma=2,\alpha=0.25$ reweights cross-entropy by prediction confidence so that the cumulative loss from ~100k easy negatives no longer drowns ~100 hard positives¹⁴, and the classification head's bias is initialized to $-\log((1-\pi)/\pi),\ \pi=0.01$ to make the initial logits match the empirical class prior. Focal loss beat OHEM because it uses every negative with smooth down-weighting rather than a hard cutoff. Crucially, RetinaNet sat on top of FPN (Lin, Dollár, et al. 2017), whose top-down + lateral pathway propagates semantic strength to high-resolution feature levels at single-pass cost; AP_S rose from 14.1 to 19.9 (+5.8) on COCO purely from FPN, and FPN became the universal neck across Faster R-CNN, Mask R-CNN, RetinaNet, FCOS, and EfficientDet¹⁵.

The anchor-free turn (CornerNet, FCOS, CenterNet) made a structural rather than empirical case. FCOS predicts $(l,t,r,b)$ distances at every foreground pixel of the FPN levels and resolves multi-GT overlap by assigning to the smallest-area box; the only heuristic the paper retains¹⁶. The centerness branch $\sqrt{\min(l,r)/\max(l,r) \cdot \min(t,b)/\max(t,b)}$ suppresses low-quality peripheral predictions for +3.6 AP and is a cheap analogue of objectness. CornerNet detects top-left and bottom-right corner heatmaps with associative-embedding grouping, and CenterNet reduces this further to a single Gaussian-heatmap center per object plus a $(w,h)$ regression. Anchor-free heads sidestepped hand-tuned anchor scales/ratios entirely¹⁷ and proved that proper loss weighting plus per-level FPN assignment closes the gap; ATSS later showed that adaptive label assignment (mean+std of per-object IoU as the dynamic threshold) brings anchor-based and anchor-free recipes to within 0.1 AP, confirming that the anchor itself was never the load-bearing piece¹⁸.

The transformer set-prediction line, covered fully in the next chapter, starts with DETR replacing dense prediction + NMS with 100 learned object queries and Hungarian bipartite matching against the ground-truth set. NMS disappears because each query commits to one GT, eliminating the threshold-sensitive greedy suppression heuristic that limits CNN detectors in dense crowds¹⁹²⁰. The cost was a 500-epoch convergence schedule (versus 12-24 for Faster R-CNN); Deformable DETR's sparse multi-scale attention reduced this to 50 epochs, DAB-DETR/DN-DETR added content-aware queries and denoising auxiliaries, and DINO-DETR with Swin-L plus contrastive denoising reaches 58.5 COCO AP²¹. Co-DETR closed the loop by re-introducing one-to-many auxiliary heads (ATSS, Faster R-CNN) alongside the one-to-one Hungarian head during training, lifting Swin-L COCO val from 58.5 to 59.5 and ViT-L COCO test-dev to 66.0 AP, the current chapter ceiling.

The closing reading is that detection has not so much "solved" any of the listed problems as redistributed them. Scale variance moved from image pyramids into FPN; foreground/background imbalance moved from RPN sampling into focal loss into per-query bipartite assignment; anchor design was first adaptive (ATSS), then dropped (FCOS), then absent (DETR); NMS was softened (Soft-NMS, DIoU-NMS) and then removed (DETR). What remains stubborn is small-object AP (still 15-25 points below large-object AP across the entire table, even at Swin-L scale), cross-paper FPS comparability (every paper reports its own GPU)²², and ImageNet/Object365 pretraining dependence²³; the latter starts to dissolve only with self-supervised backbones (MAE, DINO) trained at 10$^8$+ image scale²⁴.

Detection and Segmentation Transformers

Object detection has traditionally relied on two-stage detectors like Faster R-CNN or single-stage detectors like YOLO and RetinaNet. These methods require hand-designed components: anchor boxes for proposal generation, non-maximum suppression (NMS) for duplicate removal, and complex label assignment strategies. The introduction of DETR (Carion et al. 2020) fundamentally changed this paradigm by framing detection as a set prediction problem.

DETR uses a transformer encoder-decoder architecture with learned object queries and bipartite matching for training. While conceptually elegant, the original DETR suffered from slow convergence and difficulty with small objects. Subsequent work addressed these limitations through deformable attention (Zhu et al. 2021), improved query formulations (Shilong Liu et al. 2022), and denoising training strategies (F. Li et al. 2022).

Image segmentation has similarly been transformed by the mask classification paradigm. MaskFormer (Cheng, Schwing, and Kirillov 2021) showed that semantic, instance, and panoptic segmentation can be unified under a single framework. Mask2Former (Cheng et al. 2022) improved this with masked attention, achieving state-of-the-art across all segmentation tasks.

The Swin Transformer (Ze Liu et al. 2021) introduced hierarchical vision transformers with shifted window attention, providing efficient backbones for dense prediction tasks. Swin Transformer V2 (Ze Liu et al. 2022) scaled these models further with improved training stability.

Vision Transformers

Vision Transformers (ViT) represent a paradigm shift from CNNs to pure attention-based architectures. The foundational ViT paper (Dosovitskiy et al. 2021) demonstrated that a standard Transformer encoder, applied directly to sequences of image patches, achieves excellent results on image classification when pretrained on sufficient data (ImageNet-21K (Deng et al. 2009) or JFT-300M). ViT-Huge achieved 88.55% top-1 accuracy on ImageNet, matching state-of-the-art CNNs while using substantially fewer computational resources during training. The key insight: inductive biases of convolutions are not strictly necessary; transformers learn these from data.

Data efficiency became the focus of subsequent work. DeiT (Touvron et al. 2021) showed ViT can be trained on ImageNet-1K alone using strong regularization, achieving 81.8% top-1 accuracy without external data (83.1% with distillation from a RegNetY-16GF teacher). DeiT introduced knowledge distillation via a distillation token, demonstrating that ViT benefits from CNN-style inductive biases transferred through distillation. DeiT III (Touvron, Cord, and Jégou 2022) pushed this further with improved training recipes, achieving 85.2% with ViT-H/14.

Hierarchical designs addressed the computational limitations of global attention. Swin Transformer (Ze Liu et al. 2021) introduced shifted window attention, computing attention within local windows while enabling cross-window connections through shifting. This achieves linear complexity and provides multi-scale feature maps compatible with dense prediction. Swin achieved 87.3% on ImageNet and set new state-of-the-art on COCO detection (58.7 box AP) and ADE20K segmentation (53.5 mIoU). Swin Transformer V2 (Ze Liu et al. 2022) scaled to 3B parameters with techniques for training stability (residual post-normalization, scaled cosine attention).

Alternative hierarchical approaches emerged. Pyramid Vision Transformer (PVT) (Wang et al. 2021) uses spatial reduction attention to handle high-resolution feature maps. Twins (Chu et al. 2021) combines locally-grouped attention with global sub-sampled attention. Focal Transformer (Yang et al. 2021) introduces focal attention attending both locally and globally with varying granularity. Nested Hierarchical Transformer (NesT) (Z. Zhang et al. 2021) aggregates tokens in a nested manner to create hierarchical representations.

Hybrid architectures combine CNN and Transformer strengths. Convolutional vision Transformer (CvT) (Wu et al. 2021) introduces convolutions into attention projection. CoAtNet (Dai et al. 2021) systematically combines depthwise convolution and attention, achieving 90.88% ImageNet accuracy with JFT-3B pretraining. MaxViT (Tu et al. 2022) interleaves block and grid attention patterns with convolutions for efficient multi-scale attention. Mobile-Former (Y. Chen et al. 2022) uses parallel mobile and former branches with bidirectional connections. MobileViT (Mehta and Rastegari 2022) unfolds features for transformer processing while maintaining mobile efficiency.

Self-supervised learning proved particularly effective for ViT. DINO (Caron et al. 2021) showed self-supervised ViT features contain semantic segmentation information not present in supervised models, achieving 78.3% k-NN accuracy on ImageNet. A follow-up diagnostic, Vision Transformers Need Registers (Darcet et al. 2024), identified a structural artifact in large pre-trained ViTs: a small fraction of patch tokens develop anomalously high feature norms and attract disproportionate attention weight, producing irregular blobs in attention maps. These high-norm tokens appear preferentially at low-information background patches, not at semantically salient locations. The interpretation is that the model repurposes uninformative patch tokens as global scratch space to store and transmit information that does not fit naturally into the attention pattern of any single semantic patch.

The fix is minimal: append $r = 4$ additional learnable [REG] tokens to the input sequence (alongside the [CLS] token) and include them throughout all layers. These tokens have no spatial correspondence and no loss target; they serve purely as internal workspace. With registers present, the high-norm artifact disappears from patch tokens, attention maps become spatially smooth, and dense prediction improves. The intervention requires no change to the training recipe, the loss function, or the backbone architecture. The paper reports new state of the art for self-supervised models on dense visual prediction tasks and demonstrates improvements in object discovery quality with larger models. The caveat is that register count $r = 4$ is a hyperparameter tuned on DINOv2-scale models; smaller ViTs trained on less data show less severe artifact behaviour and smaller gains from registers. MAE (He et al. 2021) demonstrated that masking 75% of patches and reconstructing provides strong pretraining, with ViT-Huge achieving 87.8% accuracy using only ImageNet-1K. BEiT (Bao et al. 2022) predicted discrete visual tokens from masked patches. SimMIM (Xie et al. 2022) showed simple random masking with direct pixel prediction works well.

Efficiency improvements targeted inference cost. DynamicViT (Rao et al. 2021) dynamically prunes uninformative tokens based on learned importance scores, reducing FLOPs by 31-37% with <0.5% accuracy drop. EfficientFormer (Y. Li et al. 2022) designed latency-driven architectures achieving mobile speed. Token-to-Token ViT (T2T-ViT) (Yuan et al. 2021) progressively tokenizes images to capture local structure.

Understanding ViT behavior revealed important properties. Empirical studies (Xinlei Chen, Xie, and He 2021) showed ViT optimizes differently from CNNs, with sensitivity to optimizer, weight decay, and learning rate schedules. "How to train your ViT" (Steiner et al. 2022) provided comprehensive training recipes. Sharpness-aware minimization (SAM) (Foret et al. 2021) improved ViT generalization by optimizing for flatter minima. Adversarial robustness analysis (Naseer et al. 2021) (Shao et al. 2022) found ViT more robust than CNNs due to attention's global context. Studies on when transformers beat CNNs (Xiangning Chen, Hsieh, and Gong 2022) identified data scale and model capacity as key factors.

Cross-attention between scales improved multi-scale reasoning. CrossViT (C.-F. Chen, Fan, and Panda 2021) uses dual-branch architecture with different patch sizes, fusing via cross-attention. This achieves efficient multi-scale processing while maintaining reasonable computational cost.

The field converged on: (1) hierarchical designs for dense prediction (Swin), (2) strong regularization for data-efficient training (DeiT), (3) self-supervised pretraining for label efficiency (MAE, DINO), (4) hybrid CNN-Transformer architectures for mobile deployment, and (5) the critical importance of training recipes matching architectural choices.

Related-work leaderboard (representative numbers, ImageNet-1K top-1 unless noted, COCO box AP with Mask R-CNN or HTC++ as cited in source papers, ADE20K mIoU with UperNet):

References