RobotParser stops at the first find on the disallow/allow list

[cnicodeme]

Bug report

Bug description:

I took Wordpress's robots.txt file for testing : https://wordpress.com/robots.txt

It contains the following:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Testing for "/wp-admin/admin-ajax.php" will return False instead of True

Reproducible sample:

from urllib.robotparser import RobotFileParser

ra = RobotFileParser()
ra.parse('''User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php'''.splitlines())

assert ra.can_fetch('GoogleBot', 'https://wordpress.com/wp-admin/') is False
assert ra.can_fetch('GoogleBot', 'https://wordpress.com/wp-admin/admin-ajax.php') is True  # Fails here

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[picnixz]

This may sound stupid but the docs mention a document for the robots structure that does not specify how "Allow" works (or maybe I missed it).

Also, is there anything about the order of definitions for allow/disallow?

[cnicodeme]

It would be ideal to have a order, because in that case, you stop at the first match. Otherwise, you have to parse the whole list everytime and keep the better match.

I think the best approach would be to list all the lines that matches the current path, get the longest one, and apply its rule (allow/deny), but I'm not sure it's the right approach here.

[picnixz]

I don't know which specs we will (historically) and I don't know if this issue was raised in the past. cc @serhiy-storchaka for historical knowledge (maybe)

I would have expected that exact matches in either list takes precedence though.

[serhiy-storchaka]

This is a duplicate of #83368 which is a part of #138907.