IBX-12127: Fixed valid URLs being marked as invalid by ibexa:check-urls

[tbialcz]

🎫 Issue	IBX-12127

Description:

ibexa:check-urls checked links with a bare cURL HEAD request without any headers (since the original implementation, EZP-28505). WAFs like Cloudflare block such requests, so working links were marked as invalid.

Now the request sends browser-like User-Agent/Accept headers, and a failed HEAD is retried once with GET. New optional handler options: method, fallback_to_get, user_agent, headers.

Also fixed in the same code: the https handler was reading http options, and the curl_multi loop could skip trailing URLs.

For QA:

1. Add RichText links to https://www.tiendanimal.es/ and to a URL that rejects HEAD requests (e.g. a local php -S server returning 403 for HEAD), publish.
2. Run php bin/console ibexa:check-urls — both links are Valid in Link manager (before the fix: Invalid).
3. Add a genuinely broken link (404 or dead host), re-run — it is marked Invalid.

Documentation:

Yes — document the new url_checker.handlers.http|https options: method, fallback_to_get, user_agent, headers.

[alongosz]

My general remark - if we're revamping this, can we use something more modern than reinvented-wheel-curl?
Like Symfony HTTP client? Or it doesn't make sense here or it's too much work? Just curious.

Remarks to the current solution, which is still a lot of great work 💪

[alongosz]

Shouldn't this be something more reflecting reality?

Suggested change

	user_agent: 'Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0'
	user_agent: 'Ibexa DXP URL checker'

Or does Cloudfare treat it as suspicious and it's on purpose like that?

[tbialcz]

On purpose — some sites block non-browser UAs (e.g. zillow.com returns 403 instead of 200). The user_agent can be changed if needed.

[tbialcz]

My general remark - if we're revamping this, can we use something more modern than reinvented-wheel-curl? Like Symfony HTTP client? Or it doesn't make sense here or it's too much work? Just curious.

Remarks to the current solution, which is still a lot of great work 💪

Thanks! Agreed, HttpClient would be cleaner, and it’s already a dependency. I kept this PR small as a 4.6 bugfix. I’d do the HttpClient rewrite in a follow-up ticket.

[ViniTou]

this seems to be configured in yaml file, yet we have const here? or those are two different things? cant we at least call const syntax in yaml file?

[tbialcz]

right. i removed the duplication

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud