Skip to content

fix(workflows): reject a non-string prompt in prompt-step validate()#3582

Merged
mnriem merged 1 commit into
github:mainfrom
Noor-ul-ain001:fix/prompt-nonstring-prompt-parity
Jul 17, 2026
Merged

fix(workflows): reject a non-string prompt in prompt-step validate()#3582
mnriem merged 1 commit into
github:mainfrom
Noor-ul-ain001:fix/prompt-nonstring-prompt-parity

Conversation

@Noor-ul-ain001

Copy link
Copy Markdown
Contributor

Summary

PromptStep.execute str()-coerces config['prompt'] and dispatches the result to the integration CLI as the model's instructions:

prompt = evaluate_expression(prompt_template, context)
if not isinstance(prompt, str):
    prompt = str(prompt)   # <-- a list/None/int becomes its Python repr

But PromptStep.validate only checked that prompt was present, not that it was a string — the exact parity gap the sibling ShellStep already closes for its run field (which cites "the command-step input/options and gate options type checks" as precedent).

So a YAML authoring slip like prompt: [review, this] or a bare prompt: (YAML null) passed validation, then execute sent the Python repr ("['review', 'this']", "None") to the LLM verbatim — silently wrong instructions, no error, and a COMPLETED status. The engine does not auto-validate step config (load_workflow explicitly defers validation to validate_workflow()/engine.validate()), so validation is the only place this surfaces before dispatch.

Fix

Extend validate to reject any non-string prompt, mirroring the shell-step run and command-step input/options type checks:

elif not isinstance(config["prompt"], str):
    errors.append(
        f"Prompt step {config.get('id', '?')!r}: 'prompt' must be a "
        f"string, got {type(config['prompt']).__name__}."
    )

A {{ ... }} expression is still a str, so it stays valid.

Testing

  • Adds test_validate_rejects_non_string_prompt (parametrized over null/list/int/dict), mirroring TestShellStep::test_validate_rejects_non_string_run.
  • Adds test_validate_accepts_expression_prompt to confirm an expression prompt still validates.
  • Full TestPromptStep + validation suite passes (85 selected, 0 failures).

🤖 Generated with Claude Code

`PromptStep.execute` str()-coerces `config['prompt']` and dispatches the
result to the integration CLI as the model's instructions. But its `validate`
only checked that `prompt` was *present*, not that it was a string — the exact
parity gap the sibling `ShellStep` closes for `run`.

So a YAML authoring slip like `prompt: [review, this]` or `prompt:` (null)
passed validation, then `execute` sent the Python repr (`"['review', 'this']"`,
`"None"`) to the LLM verbatim — silently wrong instructions with no error and a
COMPLETED status. The engine does not auto-validate step config
(`load_workflow` explicitly defers validation), so validation is the only place
this surfaces before dispatch.

Extend `validate` to reject any non-string `prompt` with the shell-step's
phrasing ("'prompt' must be a string, got <type>"), mirroring the shell `run`
and command `input`/`options` type checks. A `{{ ... }}` expression is still a
str, so it stays valid. Adds regression coverage for non-string prompts
(null/list/int/dict) and confirms an expression prompt still validates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds validation to prevent non-string prompt values from being dispatched as Python representations.

Changes:

  • Rejects non-string prompt values during validation.
  • Tests invalid prompt types and expression-string acceptance.
Show a summary per file
File Description
src/specify_cli/workflows/steps/prompt/__init__.py Adds prompt type validation.
tests/test_workflows.py Covers invalid prompt types and expressions.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Medium

@mnriem
mnriem merged commit b139bd0 into github:main Jul 17, 2026
12 checks passed
@mnriem

mnriem commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants