1. Overview

• Contributions of this work:
  • Introduce a new robotic vision task: probabilistic object detection
  • Define a new performance measure for probabilistic object detection: probabilistic detection quality (PDQ)
  • Evaluate PDQ
  • Evaluate state-of-the-art detectors using PDQ

• Questions:
  • Is probabilistic object detection necessary?
  • Can/should this be applied in SLAM?

• The Probabilistic Object Detection Challenge

2. Motivation

• Current CV paradigm for object detection: deterministic bounding box and semantic class with confidence score/label distribution

• Evaluation metrics based on this idea of object detection, influence training

• Conventional object detectors can over confidently assign incorrect label

• Need spatial and label uncertainty estimates

3. Background & Related Work

3.1. Evaluation Metrics

• Detections classified with threshold on Intersection over Union (IoU)

• Average precision (AP): sort detections by confidence, compute area under precision-recall curve

• Mean average precision (mAP) - vary IoU threshold from 0.5:0.05:0.95, take mean of APs

3.2. Conventional Object Detection

• Typical detector output: bounding boxes and class label scores

• CNNs rapidly improving in accuracy and speed

• Spatial and semantic uncertainty not typically provided

3.3. Uncertainty Estimation in Object Detection

• Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks (Kampffmeyer, 2016)
  • Monte Carlo dropout approximates Bayesian inference
  • Pixel-wise classification uncertainty for semantic labels

• Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding (Kendall, 2017)
  • Bayesian model outputs pixel-wise semantic segmentation with model uncertainty per class

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? (Kendall and Gal, 2017)
  • Examine aleatoric (observation) and epistemic (model) uncertainty

• Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection (Miller, 2019)
  • Estimate spatial and classification uncertainties for object detection
  • Use uncertainty to accept/reject detections in near open-set conditions

4. Probabilistic Object Detection

• Object detection consists of:
  • Probability distribution over known labels
  • Bounding box: Gaussian distributions for corner positions

5. Probability-based Detection Quality (PDQ)

• Frame $f$, evaluate detections $\mathcal{D}_j^f$ with ground truth objects $\mathcal{G}_i^f$

• Foreground Loss $L_{FG}$

• Background Loss $L_{BG}$

• Spatial Quality $Q_{S}$

• Label Quality $Q_{L}$

• Pairwise PDQ $\mathrm{pPDQ}$

• PDQ Score $\mathrm{pPDQ}$

• Example PDQ scores

6. PDQ Evaluation

• Perform experiments with simulated object detectors
  • For all ground truth objects, simulate detector by adding true variance to detections
  • Independent of true variance, simulated detector gives observations with some random reported variance

• PDQ score maximized when reported variance best matches true variance

• PDQ directly affected by label probability, vs mAP which only affected by dominant class

7. Evaluation of Object Detectors

• Evaluate state-of-the-art detectors
  • Convert standard detections to probabilistic detections, assuming $P(\mathbf{x}\in\mathcal{S}_j^f)=1-\epsilon$ for pixels inside bounding box, $P(\mathbf{x}\notin\mathcal{S}_j^f)=\epsilon$ for pixels outside

• Probabilistic detectors
  • MC-Dropout SSD: use MC Dropout with SSD-400 detector, BSAS clustering to estimate Gaussians
  • probFRCNN: find detections suppressed by nonmax supression with IoU > 0.75, cluster and estimate Gaussians for corners

• Observations
  • Probabilistic detectors perform best in PDQ
  • Top performing standard detectors have poor spatial quality eg YOLOv3
  • mAP measure does not penalise high false positives eg FRCNN X+FPN(0.05)

8. Conclusions and Future Work

• Performance metrics strongly influence how object detectors are designed and trained to perform

• New performance measure for probabilistic object detection will steer work towards better spatial uncertainty estimation

• Extend to probabilistic instance segmentation - replace bounding box spatial probability density with segmentation mask density

9. Discussion

• Probabilistic object detection better suited to robotic deployment in real world

• How should probabilistic object detection be used in SLAM?
  • PDQ metric, probabilistic approach to object detection will steer object detection field towards characterising uncertainty
  • Add uncertain object labels to factor graph
  • Future SLAM maps should incorporate instance segmentations - these need accurate uncertainty estimates