TransMAD: Transformer-based Masked Adversarial Defenses to Patch Attacks

Patch attacks create threats that can generalize to the physical world and alter the predictions of deep neural networks in visual recognition tasks. Existing defenses require retraining the network with adversarial examples, which are costly to generate, or alter existing architectures to deal with perturbations. Masked Adversarial Defenses (MAD) are a promising alternative that aims to ignore patch attacks by masking them. In this paper, we leverage the transformer’s ability to (1) mask tokens at random without losing representation capacity and (2) ignore attacked regions by not attending to them. We propose a transformer-based denoiser that removes adversarial patches and recovers benign examples. For classification, we propose a defense that uses multiple random masks combined with majority voting for prediction. For object detection, since different random masks produce different detections, we propose a method to fuse such detections robustly. In addition, since masking can eliminate objects that have not been attacked, we propose to exploit the transformer’s attention mechanism to select suitable masks. Experiments on classification and object detection benchmarks demonstrate the efficiency and robustness of the proposed TransMAD approach.