Skip to content

[Feature]: ARCBackend — submit + status over ARC 7 REST, token auth #22

Description

@aldbr

User Story

As the WMS consumer,
I want an ARCBackend that submits and monitors jobs on an ARC 7 CE via REST with token auth,
So that the primary grid-CE path (DIRAC's AREX CE) exists on the new contract, validated against
the arc-slurm stack.

Feature Description

Essential half of the AREX port (IC-ADR-001 migration map):

  • ARCBackend: implements Submitter + Monitor against the ARC REST
    API (/arex/rest/1.0) using pyarcrest (evaluate fitness; fall back to httpx against the REST
    spec if it can't do async/bulk cleanly — record the choice).
  • Job description generation from SubmissionSpec (XRSL or ADL — pick one, document why;
    DIRAC builds XRSL), incl. input-sandbox declaration and queue targeting.
  • Bulk semantics: ARC REST supports multi-job operations — submit count copies and poll many
    ids per call; per-id failures ride the outcome maps.
  • State normalisation: ARC state map → JobStatus (port DIRAC's STATES_MAP, incl. the
    auto-kill-on-Hold question — surface as policy, don't hide it in the backend).
  • Token auth per IC-ADR-003: bearer header, audience exposure. Proxy/delegation
    is issue-14.
  • OutputRetriever (destructive=False): list the ARC session dir, stream every member into
    dest file-by-file, return the JobOutput manifest (.stdout/.stderr/.files/.log).
    Bounded materialisation enforced inline: byte ceiling mid-stream, file-count ceiling,
    per-transfer timeout, path containment (../absolute members rejected) — IC-ADR-001 §4.
  • Cancellable.kill / Purgeable.purge (DIRAC killJob/cleanJob): bulk, per-id outcomes.
  • LoadReporter.counts() from the CE info endpoint.
  • SupportsLiveDiagnostics.diagnostics(): the diagnose/errors endpoint — the one genuinely
    independent, non-destructive pre-completion fetch.
  • Proxy/delegation auth: create delegation (CSR → sign with the
    proxy chain → upload), renew, AlwaysIncludeProxy parity where tokens+proxy coexist. Add the
    config/proxy/ configuration axis to the arc-slurm stack (ADR-002 §7) to test it.
  • Fetch-twice regression test proving re-fetchability (destructive=False) — the counterpart
    of HTCondor's one-shot test.
  • Phase 2 wiring: contract suite (submit/status subset) against the arc-slurm stack
    (markers remote and arc).

Definition of Done

  • Full contract suite (all capabilities) green against arc-slurm, both token and proxy
    configs
  • count > 1 submission bundles one Submission with per-copy failure reporting
  • Unknown/expired ids → JobStatus.UNKNOWN (never dropped)
  • State map unit-tested; Hold handling decided and documented
  • Fetch-twice test green; containment/ceiling tests green (hostile filenames, oversized
    output)
  • Delegation renewal covered by a test (short-lived test-CA proxy)
  • arc-slurm stack gains config/proxy/; manifest updated
  • Registered via intercede.backends as arc

Alternatives Considered

  • Wrapping the arc CLI tools — spawns processes, brittle parsing; REST is first-class in ARC 7
    and DIRAC's AREX already proves it.

Related Issues

Additional Context

ARC floor is 7.x (IC-ADR-002 §8). DIRAC reference: `AREXComputingElement.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions