Skip to content

feat(tasks): add dummy job executor#995

Open
anpicci wants to merge 5 commits into
DIRACGrid:mainfrom
anpicci:issue-964-dummy-job-executor
Open

feat(tasks): add dummy job executor#995
anpicci wants to merge 5 commits into
DIRACGrid:mainfrom
anpicci:issue-964-dummy-job-executor

Conversation

@anpicci

@anpicci anpicci commented Jul 16, 2026

Copy link
Copy Markdown

Summary

  • add DummyJobExecutorMonitorTask, a settings-controlled periodic task that:
    • is disabled by default;
    • when enabled, searches for jobs in Received, moves them to Waiting, and schedules one executor task per job;
  • add DummyJobExecutorTask, a one-shot task that simulates job execution through the valid state sequence Waiting → Matched → Running → Done, while holding a per-job MutexLock;
  • register and publicly export both tasks as jobs:DummyJobExecutorMonitorTask and jobs:DummyJobExecutorTask;
  • add administrator documentation for the dummy-executor environment settings;
  • enable the monitor explicitly at a 10-second interval in run_local.sh;
  • provide the corresponding demo-deployment enablement in fix(demo): enable dummy job executor diracx-charts#286;
  • add focused tests for scheduling activation, interval validation, locking, status transitions, task fan-out, and the no-job case.

Motivation

This provides a standalone dummy job-execution pipeline for exercising the DiracX task framework end to end—job discovery, status transitions, task fan-out, and per-job locking—without requiring execution through legacy DIRAC. The monitor remains opt-in so production/default deployments do not schedule it automatically.

Closes #964.

Implementation notes

  • DummyJobExecutorMonitorTask uses Size.SMALL and does not define a retry policy, since an enabled monitor runs periodically.
  • DummyJobExecutorTask uses Size.LARGE and ExponentialBackoff(base_delay_seconds=10, max_retries=3).
  • The executor applies the Matched → Running → Done transitions in one set_job_statuses call with increasing timestamps, because a direct transition to Done would be rejected by the state machine.
  • All database dependencies (JobDB, JobLoggingDB, TaskQueueDB, and JobParametersDB) and Config are injected through the task framework.
  • The monitor is controlled through:
    • DIRACX_TASKS_DUMMY_JOB_EXECUTOR_ENABLED (default: false);
    • DIRACX_TASKS_DUMMY_JOB_EXECUTOR_INTERVAL_SECONDS (default: 10, required to be positive).
  • run_local.sh explicitly enables the monitor at a 10-second interval. The standard demo deployment is handled by the companion charts PR, fix(demo): enable dummy job executor diracx-charts#286.
  • Deliberately out of scope: JDL handling, real matching or pilots, and real job execution.

Validation

  • focused dummy-executor tests: 6 passed;
  • full diracx-tasks suite: 71 passed;
  • all Pixi-dependent pre-commit hooks passed with --all-files;
  • DIRACX and Gubbins client-generation checks passed with no retained generated-client changes;
  • Basic Tests, Integration Tests, Pixi Lock, and Deployment GitHub Actions workflows all pass.

@read-the-docs-community

read-the-docs-community Bot commented Jul 16, 2026

Copy link
Copy Markdown

Documentation build overview

📚 diracx | 🛠️ Build #33709636 | 📁 Comparing a0b49e2 against latest (74a1a6a)

  🔍 Preview build  

2 files changed
± admin/reference/env-variables/index.html
± admin/how-to/tasks/run-task-manually/index.html

@aldbr
aldbr requested a review from ryuwd July 17, 2026 09:33
@ryuwd ryuwd self-assigned this Jul 17, 2026
Replace the logging-only stub with a working pair of tasks:

- DummyJobExecutorMonitorTask: periodic task (every 10s) that moves
  Received jobs to Waiting and schedules a one-shot executor per job.
- DummyJobExecutorTask: simulates the execution of a single job by
  walking it through the state machine's valid path
  Waiting -> Matched -> Running -> Done via set_job_statuses, holding
  a per-job mutex lock.
@ryuwd
ryuwd force-pushed the issue-964-dummy-job-executor branch from 1068562 to c556409 Compare July 17, 2026 11:30
@ryuwd
ryuwd marked this pull request as ready for review July 17, 2026 11:50
@ryuwd
ryuwd requested a review from chrisburr July 17, 2026 11:50
Comment thread diracx-tasks/src/diracx/tasks/jobs/dummy_job_executor.py Outdated
Comment thread diracx-tasks/src/diracx/tasks/jobs/dummy_job_executor.py Outdated
Comment thread diracx-tasks/src/diracx/tasks/jobs/dummy_job_executor.py Outdated
Comment thread diracx-tasks/src/diracx/tasks/jobs/dummy_job_executor.py Outdated
Comment thread diracx-tasks/tests/test_dummy_job_executor.py
Comment thread diracx-tasks/src/diracx/tasks/jobs/dummy_job_executor.py Outdated
@anpicci
anpicci requested a review from chrisburr July 23, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Dummy Job Executor

3 participants