Faculty, Staff and Student Publications
Language
English
Publication Date
1-1-2024
Journal
Proceedings of Machine Learning Research
PMID
39206101
PMCID
PMC11350389
PubMedCentral® Posted Date
8-28-2024
PubMedCentral® Full Text Version
Author MSS
Abstract
Training Deep Neural Networks (DNNs) with adversarial examples often results in poor generalization to test-time adversarial data. This paper investigates this issue, known as adversarially robust generalization, through the lens of Rademacher complexity. Building upon the studies by Khim and Loh (2018); Yin et al. (2019), numerous works have been dedicated to this problem, yet achieving a satisfactory bound remains an elusive goal. Existing works on DNNs either apply to a surrogate loss instead of the robust loss or yield bounds that are notably looser compared to their standard counterparts. In the latter case, the bounds have a higher dependency on the width𝑚 of the DNNs or the dimension𝑑 of the data, with an extra factor of at least𝒪(√𝑚) or𝒪(√𝑑).
This paper presents upper bounds for adversarial Rademacher complexity of DNNs that match the best-known upper bounds in standard settings, as established in the work of Bartlett et al. (2017), with the dependency on width and dimension being 𝒪(ln (𝑑𝑚)). The central challenge addressed is calculating the covering number of adversarial function classes. We aim to construct a new cover that possesses two properties: 1) compatibility with adversarial examples, and 2) precision comparable to covers used in standard settings. To this end, we introduce a new variant of covering number called the uniform covering number, specifically designed and proven to reconcile these two properties. Consequently, our method effectively bridges the gap between Rademacher complexity in robust and standard generalization.
Keywords
Adversarially Robust Generalization, Rademacher Complexity, Covering Number
Published Open-Access
yes
Recommended Citation
Xiao, Jiancong; Sun, Ruoyu; Long, Qi; et al., "Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization" (2024). Faculty, Staff and Student Publications. 933.
https://digitalcommons.library.tmc.edu/uthshis_docs/933