Support for foolbox attacks via a simple interface by henba1 · Pull Request #166 · ADA-research/VERONA

henba1 · 2025-11-25T16:21:10Z

Purpose

Adds a generic interface to foolbox so any of its attacks (FGSM, PGD, DeepFool, Carlini-Wagner, boundary attacks, L2/L-inf/L1 variants, ...) can be used as a VERONA Attack without a dedicated wrapper per attack, alongside the existing FGSMAttack, PGDAttack, and AutoAttackWrapper.

What changed

FoolboxAttack (ada_verona/verification_module/attacks/foolbox_attack.py) — a thin Attack subclass that wraps any foolbox.attacks.* class:
- Takes the foolbox attack class, input bounds (default (0, 1)), and arbitrary constructor kwargs (e.g. steps=40), and runs it through foolbox.PyTorchModel inside execute().
- Normalizes input/target tensors to the batched shape foolbox expects (adds a batch dimension for unbatched single-sample inputs from the robustness-distribution pipeline).
- Supports both untargeted (foolbox.criteria.Misclassification, default) and targeted (targeted=True + target_class, foolbox.criteria.TargetedMisclassification) attacks; targeted mode is attack-dependent (single-step attacks like FGSM don't support it and raise ValueError from foolbox).
foolbox as an optional dependency — added to pyproject.toml/requirements.txt/uv.lock; not required for the rest of VERONA.
Examples — examples/scripts/create_robustness_dist_foolbox.py: an end-to-end tutorial comparing L-inf attacks (FGSM, PGD, DeepFool) on an ImageFileDataset, plus an L2 attack demo loading data via PytorchExperimentDataset to show the pipeline is norm- and dataset-loader-agnostic.
Docs & credits — docs/how-to-guides.md lists foolbox attacks alongside the existing methods; README.md credits the foolbox authors (Rauber et al., 2017 and 2020) in the acknowledgements section.
Two unrelated pre-existing bugs fixed in passing (surfaced while exercising the pipeline for this PR, independent of the foolbox feature itself):
- ExperimentRepository.get_result_df() crashed with AttributeError: 'DataFrame' object has no attribute 'network_path' on result CSVs that already contain a network column (the schema actually written by save_result()/save_results()) — it now only derives network from network_path for the legacy schema.
- CI: pinned ada_verona/foolbox as first-/third-party in the ruff isort config (a local foolbox/ checkout in the repo root was being misdetected as first-party, hiding real import-order errors) and guarded uv venv so a restored .venv cache doesn't fail environment creation.

How it was tested

New unit tests in tests/test_verification_module/attacks/test_foolbox_attack.py covering untargeted/targeted execution, the missing-target_class validation error, and tensor-shape normalization branches (added incrementally, including a pass specifically to close codecov patch-coverage gaps).
tests/test_database/test_experiment_repository.py covers the get_result_df fix.
Full existing test suite continues to pass; ruff check / ruff format --check pass.
Manually ran the new create_robustness_dist_foolbox.py example end-to-end (both the L-inf comparison and the L2/PytorchExperimentDataset part) to confirm the interface produces sensible robustness distributions for both the targeted as well as untargeted case via a dedicated repo shared with @Aaron99B (the reviewer): https://github.com/henba1/verona-foolbox-debug

…supported attacks in our documentation

codecov · 2025-11-25T16:24:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

henba1 · 2025-11-25T16:32:06Z

potentially leave out of mandatory dependencies

addressed, not a mandatory dependency anymore

…RONA into henba1-foolbox-interface

AWbosman · 2025-12-01T13:38:07Z

+        if data.dim() == 3:
+            data = data.unsqueeze(0)
+
+        if target.dim() == 0:


What happends when its a targeted attack?

targeted attacks are currently not supported - the target here is the correct label

AttackEstimationModule.verify() currently always passes the true label (verification_context.data_point.label) as the "target"

covered now in 62bf5da : targeted attacks are supported and handled through foolbox criterion obj criterion object. The untargeted wrapper passes the true label, which foolbox auto-wraps as Misclassification . Targeted mode just swaps that for TargetedMisclassification ("reach this class"). No changes to the Attack.execute signature or the downstream pipeline needed.

AWbosman · 2025-12-01T13:38:42Z

+    BinarySearchEpsilonValueEstimator,
+)
+from ada_verona.verification_module.attack_estimation_module import AttackEstimationModule
+from ada_verona.verification_module.attacks.foolbox_attack import FoolboxAttack


add the check whether foolbox is installed like we have for the other scripts

AWbosman · 2025-12-01T13:39:40Z

+from ada_verona.database.dataset.image_file_dataset import ImageFileDataset
+from ada_verona.database.experiment_repository import ExperimentRepository
+from ada_verona.dataset_sampler.predictions_based_sampler import PredictionsBasedSampler
+from ada_verona.epsilon_value_estimator.binary_search_epsilon_value_estimator import (


why do we need two example scripts for such a small different?

will remove one

AWbosman · 2025-12-01T13:40:12Z

its not mandatory though, treat like AutoAttack and AutoVerify\

add check for foolbox like we have for auto-verify and autoattack

henba1 · 2025-12-09T12:05:03Z

I will check the tests later

- Move foolbox from mandatory to optional dependency in pyproject.toml (add [project.optional-dependencies].foolbox and keep in dev for CI) - Merge two foolbox example scripts into one configurable script; add foolbox installation check matching the autoattack pattern - Remove duplicate create_robustness_dist_foolbox_cw.py - Document that only untargeted attacks are supported in FoolboxAttack docstring - Fix broken IdentitySampler import in __init__.py (file never existed) - Apply ruff formatting across the codebase (pre-commit hook)

- Pin foolbox as third-party and ada_verona as first-party in ruff isort config so import sorting is identical locally and in CI. A local foolbox/ source checkout in the repo root was being misdetected as first-party, hiding the I001 errors that CI (ruff 0.15.18) reports. - Reorder imports in the 3 affected files accordingly. - Guard 'uv venv' with '[ -d .venv ]' in both pytest workflows so a restored .venv cache no longer fails virtual-environment creation. The ruff config change adds no dependencies, so requirements.txt is unchanged (the sync pre-commit hook is skipped for this commit only).

FoolboxAttack now accepts keyword-only 'targeted' and 'target_class' options. By default it stays untargeted (Misclassification of the true label), so all existing usage is unchanged. With targeted=True it uses foolbox's TargetedMisclassification criterion to drive predictions toward target_class. Targeted mode is attack-dependent: iterative attacks (LinfPGD, BIM, DeepFool) support it; single-step FGSM does not and foolbox raises 'unsupported criterion'. No changes to the One2Any pipeline are needed: reaching target_class also implies the true label is no longer predicted, so it scores as a (harder) counterexample. Adds tests for untargeted defaults, the target_class validation error, and a targeted PGD run.

ExperimentRepository.get_result_df() unconditionally did: df["network"] = df.network_path.str.split("/")... but the result CSVs written by save_result()/save_results() come from VerificationContext.get_dict_for_epsilon_result(), which stores the network name in a 'network' column and never writes 'network_path'. Reading any such result file therefore raised: AttributeError: 'DataFrame' object has no attribute 'network_path' which also broke save_plots() (it calls get_result_df()). This bug is pre-existing on main and is independent of the foolbox feature. Fix: only derive 'network' from 'network_path' for the legacy schema; when a 'network' column is already present, use it as-is.

The example now showcases more of the package: - Part 1: three L-inf attacks (FGSM, PGD, DeepFool) on ImageFileDataset, with the cross-attack comparison plots (early stopping removed; the bundled dataset is small enough to use in full). - Part 2: an L2 attack on a torchvision dataset wrapped in PytorchExperimentDataset, using an L2-appropriate (larger) epsilon grid, to show the pipeline is norm-agnostic and how to use that dataset loader. DEPENDENCY: the comparison plots use the grouping-agnostic ReportCreator (group_by=...), added in branch henba1-enhance-report-creator commit 53f7c65. This script requires that change on the path until it is merged.

henba1 · 2026-06-22T17:12:16Z

The comparison plots in the new foolbox example script use the grouping-agnostic ReportCreator (group_by=...), which is added in a different branch henba1-enhance-report-creator (commit 53f7c65).

Aaron99B

There is a lot of noise in the PR due to the linting changes. Could you maybe do the small changes like additional lines, changing spaces in a separate PR first? Then this one will be way easier to review

henba1 · 2026-07-10T13:55:47Z

There is a lot of noise in the PR due to the linting changes. Could you maybe do the small changes like additional lines, changing spaces in a separate PR first? Then this one will be way easier to review

Agreed, raised the issue already in #179. Let's do this first. I will create an issue from this and open a PR.

Aaron99B

Looks good. Just some minor things

Aaron99B · 2026-08-04T20:09:12Z

-torchvision>=0.18.1
-typing_extensions
+# This file was autogenerated by uv via the following command:
+#    uv pip compile /gpfs/home2/jvrijn/code/VERONA/pyproject.toml -o /gpfs/home2/jvrijn/code/VERONA/requirements.txt


These are a lot of requirements. Are we sure we need them all for foolbox?

Aaron99B · 2026-08-04T20:13:28Z

        if result_df_path.exists():
            df = pd.read_csv(result_df_path, index_col=0)
-            df["network"] = df.network_path.str.split("/").apply(lambda x: x[-1]).apply(lambda x: x.split(".")[0])
+            if "network" not in df.columns and "network_path" in df.columns:


Why do we need this change for foolbox?

Aaron99B · 2026-08-04T20:14:32Z

+    so it is scored as a (typically harder) counterexample; `target_class` should
+    therefore differ from the true label, otherwise the search is degenerate.
+
+    Attributes:


Why are the attributes listed here and in the init?

Aaron99B · 2026-08-04T20:15:44Z

+        """
+        fmodel = foolbox.PyTorchModel(model, bounds=self.bounds)
+
+        attack = self.attack_cls(**self.kwargs)


Why do we create the attack here. Can't we already do it in the init?

henba1 added 4 commits November 25, 2025 16:56

Add foolbox interface

6457b90

Add example scripts for linf-pgd and carlini wagner attacks from foolbox

d0b330e

Add credits to authors of foolbox in our README

e064b85

Add foolbox to dependency and update uv.lock. Add foolbox attacks to …

ff01131

…supported attacks in our documentation

henba1 requested review from AWbosman and Aaron99B November 25, 2025 16:21

henba1 self-assigned this Nov 25, 2025

henba1 added the enhancement New feature or request label Nov 25, 2025

Tests for foolbox interface

547cc3e

henba1 commented Nov 25, 2025

View reviewed changes

henba1 added 5 commits November 25, 2025 17:34

Adjust test to adhere to default bounds (0,1) expected by foolbox attack

3ef0a4b

Merge branch 'main' into henba1-foolbox-interface

9627dc9

Extend foolbox test to cover lines 64-69 in foolbox_attack.py

9a13cc3

Merge branch 'henba1-foolbox-interface' of github.com:ADA-research/VE…

ef1d36e

…RONA into henba1-foolbox-interface

Adjust foolbox test

cd36bab

AWbosman reviewed Dec 1, 2025

View reviewed changes

henba1 added 2 commits December 9, 2025 12:32

Update __init__.py

793fbd2

add check for foolbox like we have for auto-verify and autoattack

Update foolbox_attack.py to add batch dim of one for pytorch models

ac36bce

henba1 added 5 commits June 17, 2026 15:54

Merge remote-tracking branch 'origin/main' into henba1-foolbox-interface

67a27f1

Add tests covering FoolboxAttack branches for codecov patch coverage

af63673

henba1 force-pushed the henba1-foolbox-interface branch from f98df8e to 0230f44 Compare June 22, 2026 16:12

henba1 force-pushed the henba1-foolbox-interface branch from 0230f44 to 1d64106 Compare June 22, 2026 16:16

henba1 requested a review from AWbosman June 22, 2026 17:13

Aaron99B reviewed Jul 5, 2026

View reviewed changes

This was referenced Jul 10, 2026

Apply ruff formatting consistently across the codebase (one-time) #180

Closed

Apply ruff formatting consistently across the codebase #181

Merged

henba1 added 2 commits July 24, 2026 14:35

Merge remote-tracking branch 'origin/main' into henba1-foolbox-interface

a201b9d

Merge branch 'main' into henba1-foolbox-interface

ef549c8

Aaron99B reviewed Aug 4, 2026

View reviewed changes

Conversation

henba1 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

What changed

How it was tested

Uh oh!

codecov Bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henba1 commented Dec 9, 2025

Uh oh!

henba1 commented Jun 22, 2026

Uh oh!

Aaron99B left a comment

Choose a reason for hiding this comment

Uh oh!

henba1 commented Jul 10, 2026

Uh oh!

Aaron99B left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

henba1 commented Nov 25, 2025 •

edited

Loading

codecov Bot commented Nov 25, 2025 •

edited

Loading