Robust Object Recognition
Outcome/Accomplishment
The Center for Smart Streetscapes (NSF CS3), a U.S. National Science Foundation (NSF)-funded Engineering Research Center (ERC) based at Columbia University, researches ways to forge livable and safe streetscapes. One area is situational awareness, which involves, among other things, multi-modal integration of inputs relating to attention. Researchers found that they could address spurious correlations in visual classification by filtering at the input level—before they enter the model. A key finding was that decoupling "where to look" from "what to extract" showed that small models (ViT-S in the figure) yield sharper, more focused attention, while large models (ViT-G) produce richer crop embeddings.
Impact/Benefits
This finding leads to the next frontier of research in this area: non-spatial spurious features. In the figure, A² assumes attention naturally concentrates on causal features — this breaks for tasks like CelebA blond-hair prediction where the spurious feature is gender. A small MLP adapter (3,000 parameters) redirects attention to the causally relevant region (hair vs. face/body). This improves the worst-group accuracy from 23% to 66% on blond men, outperforming the full-image baseline by 35%.
Explanation/Background
The researchers are now testing across additional datasets and are set to reshape how we think about input-level debiasing.
Location
New York City, NYwebsite
Start Year
Microelectronics and IT
Quantum, Microelectronics, Sensing, and IT
Lead Institution
Core Partners
Fact Sheet
Outcome/Accomplishment
The Center for Smart Streetscapes (NSF CS3), a U.S. National Science Foundation (NSF)-funded Engineering Research Center (ERC) based at Columbia University, researches ways to forge livable and safe streetscapes. One area is situational awareness, which involves, among other things, multi-modal integration of inputs relating to attention. Researchers found that they could address spurious correlations in visual classification by filtering at the input level—before they enter the model. A key finding was that decoupling "where to look" from "what to extract" showed that small models (ViT-S in the figure) yield sharper, more focused attention, while large models (ViT-G) produce richer crop embeddings.
Location
New York City, NYwebsite
Start Year
Microelectronics and IT
Quantum, Microelectronics, Sensing, and IT
Lead Institution
Core Partners
Fact Sheet
Impact/benefits
This finding leads to the next frontier of research in this area: non-spatial spurious features. In the figure, A² assumes attention naturally concentrates on causal features — this breaks for tasks like CelebA blond-hair prediction where the spurious feature is gender. A small MLP adapter (3,000 parameters) redirects attention to the causally relevant region (hair vs. face/body). This improves the worst-group accuracy from 23% to 66% on blond men, outperforming the full-image baseline by 35%.
Explanation/Background
The researchers are now testing across additional datasets and are set to reshape how we think about input-level debiasing.