Skip to content

Alt Text Generation: Integrate with Decorative Image settings #1031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dkotter merged 10 commits into from
Sep 18, 2026
Merged

Alt Text Generation: Integrate with Decorative Image settings #1031

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c8a4151
Alt Text: Integrate with Decorative Image settings
yogeshbhutkar Sep 15, 2026
bc2441b
Introduce Hook to manage focus via callback refs
yogeshbhutkar Sep 18, 2026
f0d0944
Gate generation via `isDecorative` with a notice
yogeshbhutkar Sep 18, 2026
58c676c
Improve code-related docs
yogeshbhutkar Sep 18, 2026
4459897
Update user facing docs
yogeshbhutkar Sep 18, 2026
e5997b4
Add E2E coverage for `is_decorative` path
yogeshbhutkar Sep 18, 2026
3d706d0
Update doc-block for E2E helper
yogeshbhutkar Sep 18, 2026
d777d94
Update Notice UI for images found as decorative
yogeshbhutkar Sep 18, 2026
6f41381
Update doc block for `useAltTextFocus`
yogeshbhutkar Sep 18, 2026
b69779d
Remove redundant `Stack` from `AltTextControls`
yogeshbhutkar Sep 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions docs/experiments/alt-text-generation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ The Alt Text Generation experiment adds an AI-powered "Generate Alt Text" experi

When enabled, the Alt Text Generation experiment adds "Generate/Regenerate Alt Text" controls wherever images are edited:

- **Block editor:** In the sidebar when an Image block is selected, an "AI Alternative Text" panel appears with a button to generate or regenerate alt text. After generation, a textarea shows the suggestion with "Apply" and "Dismiss" options.
- **Block editor:** In the sidebar when an Image block is selected, an "AI Alternative Text" panel appears.
- If the image is already marked decorative, Generate Alt Text is not shown and generation is skipped.
- Otherwise, a button generates or regenerates alt text. After generation, a textarea shows the suggestion with "Apply" and "Dismiss".
- If AI determines the image is decorative, the panel suggests marking it decorative instead of applying alt text, with a **Mark as decorative** action that enables the core Image block setting.
- **Media modal:** When inserting or editing an image via the media library modal (block editor, classic editor, or site editor), a Generate/Regenerate button appears next to the Alt Text field. Generated text is written into the field and core saves it when the modal is closed.
- **Attachment edit screen:** When editing an individual attachment (`Media → Library → Edit`), an "AI Alt Text" meta box or field provides the same Generate/Regenerate button.
- **Bulk action (Media Library list view):** In the list view of the Media Library, a "Generate Alt Text" option appears in the Bulk Actions dropdown. Select multiple images, choose the action, and click Apply. A progress notice tracks generation for each image, and query args are automatically stripped from the URL after completion to prevent re-triggering on refresh.
Expand All @@ -23,14 +26,16 @@ When enabled, the Alt Text Generation experiment adds "Generate/Regenerate Alt T
- Supports both attachment IDs (media library images) and image URLs (including external and data URIs)
- Output is trimmed and cleaned up (surrounding quotes and trailing periods removed)
- Single shared ability (`ai/alt-text-generation`) usable from the UI or directly via REST API
- Skip generation on Image blocks already marked decorative
- When AI detects a decorative image, suggest enabling the core "Mark as decorative" setting instead of applying alt text

### For Developers

The experiment consists of three main parts:

1. **Experiment Class** (`WordPress\AI\Experiments\Alt_Text_Generation\Alt_Text_Generation`): Handles registration, asset enqueuing, block editor and media UI integration, attachment meta box, media modal field, and bulk action registration/handling
2. **Alt Text Generation Ability** (`WordPress\AI\Abilities\Image\Alt_Text_Generation`): Validates input, resolves image references (attachment ID or URL) to a data URI, calls the AI client with a vision model and system instruction, and returns `{ alt_text: '...' }`
3. **Frontend:** React components for the block editor (`AltTextControls`), plus a DOM-based script (`media.ts`) for the media sidebar and attachment edit form that uses `runAbility` (REST when `wp.abilities.executeAbility` is unavailable)
2. **Alt Text Generation Ability** (`WordPress\AI\Abilities\Image\Alt_Text_Generation`): Validates input, resolves image references (attachment ID or URL) to a data URI, calls the AI client with a vision model and system instruction, and returns `{ alt_text: '...', is_decorative: true|false }`
3. **Frontend:** React components for the block editor (`AltTextGeneration`, `AltTextControls`, `AltTextDisabledNotice`, and `useAltTextFocus`), plus a DOM-based script (`media.ts`) for the media sidebar and attachment edit form that uses `runAbility` (REST when `wp.abilities.executeAbility` is unavailable)

The ability can be called directly via REST API for automation, bulk processing, or custom integrations.

Expand All @@ -47,7 +52,7 @@ The ability can be called directly via REST API for automation, bulk processing,
- `bulk_actions-upload` → `register_bulk_action()` adds "Generate Alt Text" to the Media Library list view bulk actions dropdown (gated by `is_enabled()`)
- `handle_bulk_actions-upload` → `handle_bulk_action()` filters selected post IDs to image attachments, checks `upload_files` capability, and redirects with `wpai_bulk_alt_text`, `wpai_attachment_ids`, and a `_wpai_bulk_nonce` signature
- `attachment_fields_to_edit` → `add_button_to_media_modal()` adds an "AI Alt Text" field with Generate/Regenerate button to the media modal
- `src/experiments/alt-text-generation/index.tsx` uses `addFilter( 'editor.BlockEdit', 'ai/alt-text-generation', ... )` to inject `<AltTextControls />` into every `core/image` block when the experiment is enabled
- `src/experiments/alt-text-generation/index.tsx` uses `addFilter( 'editor.BlockEdit', 'ai/alt-text-generation', ... )` to inject `<AltTextGeneration />` into every `core/image` block when the experiment is enabled
- `src/experiments/alt-text-generation/media.ts` finds `.ai-alt-text-media-actions` and the associated textarea (e.g. `#attachment-details-two-column-alt-text`, `#attachment-details-alt-text`, or `#attachment_alt`), wires the Generate button to `runAbility( 'ai/alt-text-generation', { attachment_id } )`, and updates the textarea value and button label on success
- Ability implementation: `includes/Abilities/Image/Alt_Text_Generation.php` (extends `Abstract_Ability`) handles input sanitization, permission checks, image reference resolution (attachment or URL → data URI), and calls `wp_ai_client_prompt()->with_file()->generate_text()` using the system instruction at `includes/Abilities/Image/alt-text-system-instruction.php`

Expand All @@ -62,9 +67,11 @@ The ability can be called directly via REST API for automation, bulk processing,
Enqueueing this script *is* the trigger for the generation run, so the nonce check is load-bearing rather than advisory: without it, any authenticated user who loaded an attacker-supplied `upload.php` URL would start a run that overwrites alt text on attacker-chosen attachments. Alt text has no revision history, so that overwrite is unrecoverable. The per-attachment `edit_post` checks in the ability's permission callback and in the REST media controller still bound *what* a run can touch, but they cannot tell a wanted run from an unwanted one.

2. **Block editor (React):**
- The `editor.BlockEdit` filter wraps the Image block with a component that renders `<AltTextControls />` when the experiment is enabled and the block is `core/image`.
- The `editor.BlockEdit` filter wraps the Image block with a component that renders `<AltTextGeneration />` when the experiment is enabled and the block is `core/image`.
- If `attributes.isDecorative` is set, `AltTextGeneration` renders `<AltTextDisabledNotice />` and does not call the ability. Otherwise, it renders `<AltTextControls />` and calls the ability.
- `AltTextControls` uses `runAbility( 'ai/alt-text-generation', params )` from `src/utils/run-ability.ts`. Params include `attachment_id` or `image_url` and optionally `context`. The helper uses `wp.abilities.executeAbility` when available, otherwise `apiFetch` to `POST /wp-abilities/v1/abilities/ai/alt-text-generation/run` with `{ input: params }`.
- On success, the component shows a textarea with the generated alt text and Apply/Dismiss buttons; Apply calls `setAttributes( { alt: generatedAlt } )`.
- On success, if `is_decorative` is true, the component shows a notice suggesting the image be marked decorative, with **Mark as decorative** and **Dismiss** actions. **Mark as decorative** sets `isDecorative: true` and clears `alt`, `caption`, `href`, `linkDestination`, `linkTarget`, and `rel` so those values are not left behind, matching core.
- Otherwise the component shows a textarea with the generated alt text and Apply/Dismiss buttons; Apply calls `setAttributes( { alt: generatedAlt } )`.

3. **Media modal & attachment edit (DOM):**
- The media script waits for `.ai-alt-text-media-actions` and the corresponding alt textarea (injected by the PHP meta box or `attachment_fields_to_edit`). It attaches a click handler to the Generate button, reads `data-attachment-id`, and calls `runAbility( 'ai/alt-text-generation', { attachment_id } )`. On success it sets the textarea value and dispatches `input`/`change` so core persists the value.
Expand Down Expand Up @@ -120,6 +127,10 @@ array(
'type' => 'string',
'description' => 'Generated alt text for the image.',
),
'is_decorative' => array(
'type' => 'boolean',
'description' => 'Whether the image was determined to be decorative',
),
),
)
```
Expand Down Expand Up @@ -167,7 +178,8 @@ curl -X POST "https://yoursite.com/wp-json/wp-abilities/v1/abilities/ai/alt-text

```json
{
"alt_text": "A red bicycle leaning against a wooden fence in a sunny park"
"alt_text": "A red bicycle leaning against a wooden fence in a sunny park",
"is_decorative": false
}
```

Expand All @@ -189,7 +201,8 @@ curl -X POST "https://yoursite.com/wp-json/wp-abilities/v1/abilities/ai/alt-text

```json
{
"alt_text": "Hero image of a team collaborating in a modern office"
"alt_text": "Hero image of a team collaborating in a modern office",
"is_decorative": false
}
```

Expand Down
183 changes: 90 additions & 93 deletions src/experiments/alt-text-generation/components/AltTextControls.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@
import { Button, TextareaControl, Notice } from '@wordpress/components';
import { update } from '@wordpress/icons';
import { InspectorControls } from '@wordpress/block-editor';
import { useEffect, useRef, useState } from '@wordpress/element';
import { type RefCallback, useState } from '@wordpress/element';
import { __ } from '@wordpress/i18n';
import { dispatch, select } from '@wordpress/data';
import { store as noticesStore } from '@wordpress/notices';
import { store as editorStore } from '@wordpress/editor';
import { Stack } from '@wordpress/ui';
import { getBlockType } from '@wordpress/blocks';

/**
* Internal dependencies
Expand All @@ -27,6 +29,9 @@ interface AltTextControlsProps {
clientId: string;
attributes: ImageBlockAttributes;
setAttributes: ( attributes: Partial< ImageBlockAttributes > ) => void;
generateButtonRef: RefCallback< HTMLButtonElement | null >;
primaryButtonRef: RefCallback< HTMLButtonElement | null >;
requestFocus: ( target: 'generate' | 'notice' | 'primary' ) => void;
}

/**
Expand All @@ -49,68 +54,37 @@ export function getButtonLabel(
return __( 'Generate Alt Text', 'ai' );
}

/**
* Decorative notice component.
*
* Displays a notice when an image is decorative.
*
* @return {React.JSX.Element} The component.
*/
export function DecorativeNotice(): React.JSX.Element {
return (
<Notice status="info" isDismissible={ false }>
{ __(
'This image appears to be decorative. Applying will set an empty alt attribute, which tells screen readers to skip it.',
'ai'
) }
</Notice>
);
}

/**
* AltTextControls component.
*
* Adds a "Generate Alt Text" button to the image block inspector panel.
*
* @param {AltTextControlsProps} props The component props.
* @param {string} props.clientId The block client ID.
* @param {ImageBlockAttributes} props.attributes The block attributes.
* @param {Function} props.setAttributes The function to set the block attributes.
* @param {AltTextControlsProps} props The component props.
* @param {string} props.clientId The block client ID.
* @param {ImageBlockAttributes} props.attributes The block attributes.
* @param {Function} props.setAttributes The function to set the block attributes.
* @param {RefCallback< HTMLButtonElement | null >} props.generateButtonRef The ref to the generate button.
* @param {RefCallback< HTMLButtonElement | null >} props.primaryButtonRef The ref to the primary button.
* @param {Function} props.requestFocus The function to request focus.
* @return {React.JSX.Element|null} The component.
*/
export function AltTextControls( {
clientId,
attributes,
generateButtonRef,
primaryButtonRef,
requestFocus,
setAttributes,
}: AltTextControlsProps ): React.JSX.Element | null {
const { id: attachmentId, url: imageUrl, alt } = attributes;

const [ isGenerating, setIsGenerating ] = useState< boolean >( false );
const [ generatedAlt, setGeneratedAlt ] = useState< string | null >( null );
const [ isDecorative, setIsDecorative ] = useState< boolean >( false );
const [ isFoundDecorative, setIsFoundDecorative ] =
useState< boolean >( false );

const hasGeneratedAlt = generatedAlt !== null;

// Refs used to manage keyboard focus as the suggestion UI appears/disappears.
const generateButtonRef = useRef< HTMLButtonElement | null >( null );
const applyButtonRef = useRef< HTMLButtonElement | null >( null );

// Set when Apply/Dismiss is clicked so focus returns to the generate button.
const shouldFocusGenerateRef = useRef< boolean >( false );

// Move focus when the suggestion UI appears (after generation) or
// disappears (after Apply/Dismiss).
useEffect( () => {
if ( hasGeneratedAlt || isDecorative ) {
// Generation complete: move focus to the Apply button.
applyButtonRef.current?.focus();
} else if ( shouldFocusGenerateRef.current ) {
// After Apply/Dismiss: return focus to the Generate/Regenerate button.
shouldFocusGenerateRef.current = false;
generateButtonRef.current?.focus();
}
}, [ hasGeneratedAlt, isDecorative ] );

// Don't show controls if there's no image.
if ( ! attachmentId && ! imageUrl ) {
return null;
Expand All @@ -128,7 +102,7 @@ export function AltTextControls( {

setIsGenerating( true );
setGeneratedAlt( null );
setIsDecorative( false );
setIsFoundDecorative( false );

// Clear any previous notices.
dispatch( noticesStore ).removeNotice( NOTICE_ID );
Expand All @@ -151,8 +125,10 @@ export function AltTextControls( {
}
);

requestFocus( 'primary' );

if ( result.is_decorative ) {
setIsDecorative( true );
setIsFoundDecorative( true );
setGeneratedAlt( '' );
} else {
setGeneratedAlt( result.alt_text );
Expand All @@ -174,50 +150,73 @@ export function AltTextControls( {
* Applies the generated alt text to the image block.
*/
const handleApply = () => {
if ( isDecorative ) {
setAttributes( { alt: '' } );
} else if ( generatedAlt ) {
if ( generatedAlt ) {
setAttributes( { alt: generatedAlt } );
}
shouldFocusGenerateRef.current = true;

requestFocus( 'generate' );
setGeneratedAlt( null );
setIsDecorative( false );
setIsFoundDecorative( false );
};

/**
* Enables the core Image block "Mark as decorative" setting.
*
* Mirrors core's implementation: set `isDecorative` and clear
* alt, caption, href, linkDestination, linkTarget, and rel fields so
* those values are not left behind.
*/
const markImageAsDecorative = () => {
// Check whether `isDecorative` is registered before setting it.
const supportsMarkAsDecorative = Object.hasOwn(
getBlockType( 'core/image' )?.attributes ?? {},
'isDecorative'
);

setAttributes( {
...( supportsMarkAsDecorative ? { isDecorative: true } : {} ),
alt: '',
caption: undefined,
href: undefined,
linkDestination: undefined,
linkTarget: undefined,
rel: undefined,
} );

requestFocus( 'notice' );
setGeneratedAlt( null );
setIsFoundDecorative( false );
};

/**
* Dismisses the generated alt text suggestion.
*/
const handleDismiss = () => {
shouldFocusGenerateRef.current = true;
requestFocus( 'generate' );
setGeneratedAlt( null );
setIsDecorative( false );
setIsFoundDecorative( false );
};

return (
<InspectorControls group="content">
<div
<Stack
direction="column"
className="ai-alt-text-controls"
style={ { padding: '0 16px' } }
>
{ /* Generated alt text preview */ }
{ hasGeneratedAlt && ! isDecorative && (
<div style={ { marginBottom: '12px' } }>
{ hasGeneratedAlt && ! isFoundDecorative && (
<Stack direction="column">
<TextareaControl
label={ __( 'Generated Alt Text', 'ai' ) }
hideLabelFromVision
value={ generatedAlt || '' }
onChange={ ( value ) => setGeneratedAlt( value ) }
rows={ 3 }
/>
<div
style={ {
display: 'flex',
gap: '8px',
marginTop: '8px',
} }
>
<Stack direction="row" gap="md">
<Button
ref={ applyButtonRef }
ref={ primaryButtonRef }
variant="primary"
onClick={ handleApply }
__next40pxDefaultSize
Expand All @@ -231,57 +230,55 @@ export function AltTextControls( {
>
{ __( 'Dismiss', 'ai' ) }
</Button>
</div>
</div>
</Stack>
</Stack>
) }

{ /* Decorative image notice */ }
{ isDecorative && (
<div style={ { marginBottom: '12px' } }>
<DecorativeNotice />
<div
{ isFoundDecorative && (
Comment thread
yogeshbhutkar marked this conversation as resolved.
<Notice status="info" onDismiss={ handleDismiss }>
<p>
{ __(
'This image appears to be decorative. Consider marking it as decorative so screen readers can skip it.',
'ai'
) }
</p>

<Button
ref={ primaryButtonRef }
variant="secondary"
onClick={ markImageAsDecorative }
style={ {
display: 'flex',
gap: '8px',
marginTop: '8px',
width: '100%',
justifyContent: 'center',
} }
__next40pxDefaultSize
>
<Button
ref={ applyButtonRef }
variant="primary"
onClick={ handleApply }
__next40pxDefaultSize
>
{ __( 'Apply', 'ai' ) }
</Button>
<Button
variant="secondary"
onClick={ handleDismiss }
__next40pxDefaultSize
>
{ __( 'Dismiss', 'ai' ) }
</Button>
</div>
</div>
{ __( 'Mark as decorative', 'ai' ) }
</Button>
</Notice>
) }

{ /* Generate button */ }
{ ! hasGeneratedAlt && ! isDecorative && (
{ ! hasGeneratedAlt && ! isFoundDecorative && (
<Button
ref={ generateButtonRef }
variant="secondary"
onClick={ handleGenerate }
disabled={ isGenerating }
accessibleWhenDisabled
style={ { width: '100%', justifyContent: 'center' } }
style={ {
width: '100%',
justifyContent: 'center',
} }
isBusy={ isGenerating }
icon={ update }
__next40pxDefaultSize
>
{ getButtonLabel( !! hasExistingAlt, isGenerating ) }
</Button>
) }
</div>
</Stack>
</InspectorControls>
);
}
Loading
Loading