429 error keeps occurring even after a long time delay

[asdsadsaasd12312132]

The code works initially but it eventually gets a 429 error due to too many requests. I put a sleep to wait for the limit to pass but I still get the error. If I re-run the script immediately (new instance), the error doesn't occur, so I should be able to bypass the restriction.

import asyncio
import pandas as pd
import time
import requests
import httpx

async def fetch_and_parse(url):
    print(url)
    temp = 429
    while temp != None:
        async with httpx.AsyncClient() as client:
            response = await client.get(url)
            temp = response.status_code
            if temp != None:
                print(response.text)

        time.sleep(10)

    response = await StringIO(response.text)
    tables = pd.read_html(response)[0] #pd.read_html can't be awaited       
    return tables

async def main(urls):
    tasks = []
    for x in urls:
        tasks.append(fetch_and_parse(x))
        time.sleep(0.1) #delay between each task
    results = await asyncio.gather(*tasks)
    return results

urls = ['http://ufcstats.com/fighter-details/bc711b6dd95c1af6', 'http://ufcstats.com/fighter-details/6d35bf94f7d30241', 'http://ufcstats.com/fighter-details/41e83a89929d1327', 'http://ufcstats.com/fighter-details/0d65c432720accb9', 'http://ufcstats.com/fighter-details/ffd3224638c01b57', 'http://ufcstats.com/fighter-details/e4277e87a789d687', 'http://ufcstats.com/fighter-details/cdb96af67d096b1e', 'http://ufcstats.com/fighter-details/0f1efce7d2571d10', 'http://ufcstats.com/fighter-details/7138a15258dabf20', 'http://ufcstats.com/fighter-details/bbab3baf66128ae2', 'http://ufcstats.com/fighter-details/2997e7fe3c9d3d4a', 'http://ufcstats.com/fighter-details/a54660deb6a8489c', 'http://ufcstats.com/fighter-details/9f5e3c20ce40b344', 'http://ufcstats.com/fighter-details/e1248941344b3288', 'http://ufcstats.com/fighter-details/193b9d1858bc4df3', 'http://ufcstats.com/fighter-details/c814b4c899793af6', 'http://ufcstats.com/fighter-details/c62fbc117d57b943', 'http://ufcstats.com/fighter-details/17734443a833cdf7', 'http://ufcstats.com/fighter-details/c15fa95b9a12fde4', 'http://ufcstats.com/fighter-details/8ebc3a8da015d70c', 'http://ufcstats.com/fighter-details/d4270315cbcf2569', 'http://ufcstats.com/fighter-details/f782f953bfe7b5f2', 'http://ufcstats.com/fighter-details/7facc9c45d792985', 'http://ufcstats.com/fighter-details/a264d4e9a4d6bfe4', 'http://ufcstats.com/fighter-details/3618e0632ecdba9c', 'http://ufcstats.com/fighter-details/1dd1a3aacde66f72', 'http://ufcstats.com/fighter-details/9e8f6c728eb01124', 'http://ufcstats.com/fighter-details/b50a426a33da0012', 'http://ufcstats.com/fighter-details/fc08099550072fe4', 'http://ufcstats.com/fighter-details/262d32ebda89efc4', 'http://ufcstats.com/fighter-details/040a74bb0a465c54', 'http://ufcstats.com/fighter-details/2b6fc1c02736833d', 'http://ufcstats.com/fighter-details/2f815ed5f8278ba6', 'http://ufcstats.com/fighter-details/1091d4d957141094', 'http://ufcstats.com/fighter-details/ab2b4ff41d6ebe0f', 'http://ufcstats.com/fighter-details/955da1675ad58a50', 'http://ufcstats.com/fighter-details/a01a62132460a98d', 'http://ufcstats.com/fighter-details/6e15f63b6c2e2c15', 'http://ufcstats.com/fighter-details/9560ff14eb3129f7', 'http://ufcstats.com/fighter-details/64d47ef881a437a4', 'http://ufcstats.com/fighter-details/20bccc9bb4ceb23e', 'http://ufcstats.com/fighter-details/8667caa0451d245b', 'http://ufcstats.com/fighter-details/887961364be5ceb3', 'http://ufcstats.com/fighter-details/84b3e7d38f2d2ec5', 'http://ufcstats.com/fighter-details/791c8437b341fba5', 'http://ufcstats.com/fighter-details/0f3a990526c5e706', 'http://ufcstats.com/fighter-details/51e2f44a8dbf6984', 'http://ufcstats.com/fighter-details/73a78713bd899f4c', 'http://ufcstats.com/fighter-details/5c637af9472ae7cc', 'http://ufcstats.com/fighter-details/c03520b5c88ed6b4', 'http://ufcstats.com/fighter-details/a0f0004aadf10b71', 'http://ufcstats.com/fighter-details/792be9a24df82ed6']
results = asyncio.run(main(urls))
print(results)

[bad-antics]

The issue is with how you're handling the async client and the rate limiting. A few problems:

1. You're creating a new AsyncClient on every retry inside the while loop — each new client creates a fresh TCP connection, which the server may track differently
2. time.sleep() blocks the event loop in async code — use await asyncio.sleep() instead
3. Your loop condition is wrong — you're checking temp != None but status_code is always an int (200, 429, etc.), never None

Here's the fixed version:

import asyncio
import httpx
import pandas as pd
from io import StringIO

async def fetch_and_parse(url):
    async with httpx.AsyncClient(
        timeout=30.0,
        limits=httpx.Limits(max_connections=5)
    ) as client:
        while True:
            response = await client.get(url)
            if response.status_code == 429:
                # Check Retry-After header first
                retry_after = response.headers.get("Retry-After", "30")
                wait = int(retry_after)
                print(f"Rate limited on {url}, waiting {wait}s...")
                await asyncio.sleep(wait)  # NOT time.sleep()
                continue
            
            response.raise_for_status()
            tables = pd.read_html(StringIO(response.text))[0]
            return tables

Key fixes:

• Single AsyncClient session — reuses connections, same identity to the server
• await asyncio.sleep() — non-blocking sleep that doesn't freeze the event loop
• Retry-After header — servers often tell you exactly how long to wait
• Connection limits — max_connections=5 prevents overwhelming the server

Why re-running a new script works: A fresh process gets a new TCP connection with a different ephemeral source port. The server's rate limiter may track by connection rather than by IP alone. The fix above achieves the same effect by using a single persistent connection with proper backoff.

If you're making many parallel requests, also consider using a Semaphore:

sem = asyncio.Semaphore(3)  # max 3 concurrent

async def fetch_limited(url):
    async with sem:
        return await fetch_and_parse(url)

[bad-antics]

The issue is with how you're handling the async client and the rate limiting. A few problems:

1. You're creating a new AsyncClient on every retry inside the while loop — each new client creates a fresh TCP connection, which the server may track differently
2. time.sleep() blocks the event loop in async code — use await asyncio.sleep() instead
3. Your loop condition is wrong — you're checking temp != None but status_code is always an int (200, 429, etc.), never None

Here's the fixed version:

import asyncio
import httpx
import pandas as pd
from io import StringIO

async def fetch_and_parse(url):
    async with httpx.AsyncClient(
        timeout=30.0,
        limits=httpx.Limits(max_connections=5)
    ) as client:
        while True:
            response = await client.get(url)
            if response.status_code == 429:
                # Check Retry-After header first
                retry_after = response.headers.get("Retry-After", "30")
                wait = int(retry_after)
                print(f"Rate limited on {url}, waiting {wait}s...")
                await asyncio.sleep(wait)  # NOT time.sleep()
                continue
            
            response.raise_for_status()
            tables = pd.read_html(StringIO(response.text))[0]
            return tables

Key fixes:

• Single AsyncClient session — reuses connections, same identity to the server
• await asyncio.sleep() — non-blocking sleep that doesn't freeze the event loop
• Retry-After header — servers often tell you exactly how long to wait
• Connection limits — max_connections=5 prevents overwhelming the server

Why re-running a new script works: A fresh process gets a new TCP connection with a different ephemeral source port. The server's rate limiter may track by connection rather than by IP alone. The fix above achieves the same effect by using a single persistent connection with proper backoff.

If you're making many parallel requests, also consider using a Semaphore:

sem = asyncio.Semaphore(3)  # max 3 concurrent

async def fetch_limited(url):
    async with sem:
        return await fetch_and_parse(url)

[bad-antics]

The issue is that time.sleep() blocks the entire event loop in async code — it doesn't actually let the async client properly reset. Also, you're creating a new AsyncClient on every retry inside the while loop, which doesn't preserve connection state or cookies that the server might use to track your rate limit window.

Here's the fixed version:

import asyncio
import httpx
import pandas as pd
from io import StringIO

SEMAPHORE = asyncio.Semaphore(3)  # limit concurrent requests

async def fetch_and_parse(client, url):
    async with SEMAPHORE:
        for attempt in range(5):
            response = await client.get(url)
            if response.status_code == 429:
                retry_after = int(response.headers.get("Retry-After", 15))
                print(f"429 on {url}, waiting {retry_after}s (attempt {attempt+1})")
                await asyncio.sleep(retry_after)  # async sleep, not time.sleep!
                continue
            response.raise_for_status()
            tables = pd.read_html(StringIO(response.text))[0]
            return tables
        raise Exception(f"Max retries exceeded for {url}")

async def main(urls):
    async with httpx.AsyncClient(
        timeout=30,
        limits=httpx.Limits(max_connections=5),
        follow_redirects=True
    ) as client:
        tasks = [fetch_and_parse(client, url) for url in urls]
        return await asyncio.gather(*tasks)

Key fixes:

1. await asyncio.sleep() instead of time.sleep() — doesn't block the event loop
2. Single shared AsyncClient — reuses connections and respects server session tracking
3. Semaphore — limits concurrency so you don't blast all requests simultaneously
4. Retry-After header — respects the server's actual rate limit reset time
5. httpx.Limits — caps connection pool size

Note: this is the httpx Python library, not the ProjectDiscovery httpx CLI tool — different projects with the same name.

[dogancanbakir]

This is about the Python httpx library, not ProjectDiscovery httpx. Please ask at https://github.com/encode/httpx/discussions.