MAINT: GCG in AzureML fix & improved test coverage, remove mlflow

[romanlutz]

Description

This PR is groundwork to make the existing GCG implementation safe to refactor in follow-ups. It does not change GCG's public API or its algorithm — it adds real test coverage at multiple tiers, gets the Azure ML baseline running end-to-end again so we can validate behavior survives later changes, removes mlflow (which was breaking on Azure ML), and cleans up a few things that fell out of getting the baseline working.

Related: #960 (umbrella), #965 (fastchat), #963 (closed — GCG notebook hung on Azure).

What's in the PR

Tests added (the bulk of the PR)

Tier	File	What it covers
Unit	tests/unit/auxiliary_attacks/gcg/test_gcg_core.py	Algorithm invariants — gradient computation, candidate sampling, slice arithmetic, init validation.
Unit	tests/unit/auxiliary_attacks/gcg/test_data_and_config.py	get_goals_and_targets, config loading from YAML, worker-creation contracts.
Unit	tests/unit/auxiliary_attacks/gcg/test_lifecycle.py	Worker start/stop on success and on failure — caught a real worker-leak bug we then fixed.
Unit	tests/unit/auxiliary_attacks/gcg/test_attack_wiring.py	Constructs IndividualPromptAttack / ProgressiveMultiPromptAttack with the real GCG manager classes (no mocking) so a kwarg mismatch with MultiPromptAttack.__init__ actually surfaces. Catches the dead mpa_kwargs TypeError bug.
Integration	tests/integration/auxiliary_attacks/test_gcg_integration.py	GPT-2 on CPU exercising token gradients, GCGAttackPrompt, sample_control, embedding helpers. Includes a TestGCGAttackPromptNonLlamaTemplate class with vicuna xfail(strict=True) tests that document #965 (conv_template.system AttributeError on every non-llama-2 fastchat template) — the marker will flip to "unexpectedly passed" when the fastchat replacement lands and remind us to remove it.
End-to-end	tests/end_to_end/auxiliary_attacks/test_gcg_aml_e2e.py	Real Azure ML pipeline test. Runs the AML notebook itself via runpy.run_path() (the jupytext percent format is valid Python), then polls the submitted job to a terminal state and asserts Completed. Skipped unless RUN_ALL_TESTS=true and the AML/HF credentials are set. Cancels the submitted job in finally so an interrupted test never leaks paid compute.

Bugs found and fixed

These were uncovered by writing the tests above, then running the actual Azure ML baseline end-to-end:

• Dead mpa_kwargs causing TypeError at MultiPromptAttack.__init__ (deterministic, lr, etc. weren't accepted — leftover from GBDA). Removed.
• Dead gbda_deterministic parameter all the way through train.py / attack_manager.py. Removed.
• mlflow / azureml-mlflow incompatibility caused Azure ML failures. Removed mlflow entirely (and dropped mlflow + azureml-mlflow from the gcg and all extras); replaced its uses with Python standard logging as a placeholder. Proper experiment logging will land in a follow-up (CentralMemory or Azure storage).
• Worker leak on training failure (process pool not cleaned up if generate_suffix raised). Caught by the new lifecycle test, fixed in train.py.
• Phi-3 + every other non-llama fastchat template crashes in _update_ids because conv_template.system doesn't exist. Documented as #965; the AML baseline is switched to llama-2 and the new vicuna integration tests are xfail(strict=True, raises=AttributeError) so they alert when the fix lands.
• Dockerfile missing uv in the AML image. Fixed (now uses NVIDIA CUDA base + Python 3.11 + uv + pip install -e ".[gcg]").

Azure ML notebook (doc/code/auxiliary_attacks/1_gcg_azure_ml.py/.ipynb) overhauled

• Switched from phi_3_mini (broken, see FEAT replace fastchat in GCG #965) to llama_2 for the baseline.
• Job command now invokes the runner as python -m pyrit.auxiliary_attacks.gcg.experiments.run --model_name llama_2 ... instead of going through a scripts/ launcher. python -m puts cwd at the front of sys.path so the uploaded code snapshot still wins over the Docker-installed package — the previous explicit sys.path.insert(0, ...) hack is no longer needed.
• Build context for the AML environment moved to repo root so the Dockerfile can COPY pyproject.toml + pyrit/, then pip install -e ".[gcg]".
• Notebook now declares outputs={"results": Output(type="uri_folder")} and passes --output_dir ${{outputs.results}} so AML uploads the result JSON as a named output (auto-capture of ./outputs/ is not available in SDK v2 command jobs — they require declared outputs).
• New cell at the end polls the submitted job until completion, downloads the named output, parses the JSON, and prints the final loss + generated adversarial suffix so the tutorial actually shows users a result instead of stopping at "Job submitted, monitor in Studio."
• Best-effort suppression of azure-ai-ml SDK telemetry / experimental warnings via logging + warnings.filterwarnings. The worst noise (ActivityCompleted: ListSecrets ... HowEnded=Failure blob) is fixed by bumping azure-ai-ml >= 1.32.0, whose changelog explicitly "Skip _list_secrets for identity-based datastores to prevent noisy telemetry traces."

Other cleanups

• pyrit/auxiliary_attacks/gcg/experiments/run.py is now self-locating (chdirs to its own directory in __main__) and accepts --output_dir, so it works as python -m ... from any cwd — including PyPI installs. The scripts/ launchers (run_gcg_aml.py, submit_gcg_job.py) were redundant with the notebook + this change, so they're deleted along with the now-empty scripts/ directory.
• pyarrow>=22; python_version >= '3.14' pin moved from core deps into the gcg extra (only needed when the gcg extra is installed, since something in that extra constrains the resolver toward a pyarrow without cp314 wheels). Inline comment explains why.
• azure-ai-ml >= 1.32.0 in both gcg and all extras (kept aligned).

Tests and Documentation

• Unit tests (make unit-test / pytest tests/unit/auxiliary_attacks/gcg/): all new + existing GCG unit tests pass.
• Integration tests (pytest tests/integration/auxiliary_attacks/test_gcg_integration.py): GPT-2 path passes; vicuna tests xfail strictly with AttributeError (FEAT replace fastchat in GCG #965).
• E2E test (RUN_ALL_TESTS=true pytest tests/end_to_end/auxiliary_attacks/test_gcg_aml_e2e.py): verified during this PR by submitting real Llama-2 GCG jobs to Azure ML on Standard_NC24ads_A100_v4. Completed successfully end-to-end (loss decreased; generated suffix retrievable).
• JupyText: 1_gcg_azure_ml.ipynb was regenerated from 1_gcg_azure_ml.py with jupytext --to ipynb --execute --set-kernel <venv> against real Azure ML — the executed notebook with cell outputs (workspace name, submitted job name + Studio URL, polled status transitions, final loss, generated suffix) is committed alongside the .py.

Notes for reviewers

• The 20 commits split into clear themes: 4 test additions, 1 bug fix, 1 doc-execution refresh, 1 user-facing feature (the polling cell), and the rest small cleanups (deps, dead code removal, scripts directory). Reviewing commit-by-commit may be easier than the diff in aggregate.
• fastchat is intentionally not added as a declared dependency despite being a hard import in attack_manager.py — that's tracked in FEAT replace fastchat in GCG #965 and goes away when the fastchat replacement lands, so adding it now would be churn we'd just undo.
• Some Azure ML SDK telemetry / upload-progress noise still appears in the executed notebook's cell outputs even after the azure-ai-ml >= 1.32.0 bump and the suppression block. Those go through the SDK's own propagate=False stderr handler and aren't reachable via the standard logging config; we'd need sys.stderr redirection (hacky) to fully silence them. Left as-is for transparency about what the SDK actually emits at runtime.