ScrapingBee is a web scraping API that handles headless browsers and rotates proxies for you. The Python SDK makes it easier to interact with ScrapingBee's API.
You can install ScrapingBee Python SDK with pip.
pip install scrapingbeeThe ScrapingBee Python SDK is a wrapper around the requests library.
Signup to ScrapingBee to get your API key and some free credits to get started.
- HTML API
- Google Search API
- Fast Search API
- Amazon API
- Walmart API
- YouTube API
- ChatGPT API
- Gemini API
- Usage API
The HTML API allows you to scrape any webpage and get the HTML content.
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.html_api(
'https://www.scrapingbee.com',
params={
'render_js': False,
}
)
print(response.content)response = client.html_api(
'https://httpbin.org/post',
method='POST',
data={
'key': 'value'
}
)Scrape Google search results in real-time.
response = client.google_search(
search='web scraping tools',
params={
'language': 'en',
'country_code': 'us',
'nb_results': 10
}
)
print(response.json())Lightweight Google search results in under a second.
response = client.fast_search(
search='pizza in new york',
params={
'country_code': 'us',
'language': 'en',
'page': 1
}
)
print(response.json())Scrape Amazon search results, product details, and pricing.
response = client.amazon_search(
query='laptop',
params={
'domain': 'com',
'language': 'en',
'pages': 1
}
)
print(response.json())response = client.amazon_product(
query='B0D2Q9397Y', # ASIN
params={
'domain': 'com'
}
)
print(response.json())response = client.amazon_pricing(
asin='B0DPDRNSXV',
params={
'domain': 'com',
'light_request': True
}
)
print(response.json())Scrape Walmart search results and product details.
response = client.walmart_search(
query='laptop',
params={
'sort_by': 'best_match',
'device': 'desktop'
}
)
print(response.json())response = client.walmart_product(
product_id='123456789',
params={
'device': 'desktop'
}
)
print(response.json())Scrape YouTube search results, video metadata, and subtitles.
response = client.youtube_search(
search='web scraping tutorial',
params={
'sort_by': 'relevance',
'type': 'video'
}
)
print(response.json())response = client.youtube_metadata(video_id='dQw4w9WgXcQ')
print(response.json())response = client.youtube_subtitles(
video_id='dQw4w9WgXcQ',
params={
'language': 'en',
'subtitle_origin': 'uploader_provided'
}
)
print(response.json())Use ChatGPT with optional web search.
response = client.chatgpt(
prompt='What is web scraping?',
params={
'search': True,
'country_code': 'us'
}
)
print(response.json())Send prompts to Gemini and receive AI-generated responses.
response = client.gemini(
prompt='Best programming languages for data science',
params={
'country_code': 'us',
'add_html': False
}
)
print(response.json())Check your API credit usage.
response = client.usage()
print(response.json())
# {
# "max_api_credit": 8000000,
# "used_api_credit": 1000023,
# "max_concurrency": 200,
# "current_concurrency": 1
# }The get() and post() methods are deprecated and will be removed in a future version. Please use html_api() instead.
# Deprecated
client.get(url, params={...})
# Use instead
client.html_api(url, method='GET', params={...})Here is a little example on how to retrieve and store a screenshot from the ScrapingBee blog.
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.html_api(
'https://www.scrapingbee.com/',
params={
'screenshot': True,
'screenshot_full_page': True,
'window_width': 375,
}
)
with open('screenshot.png', 'wb') as f:
f.write(response.content)The client includes a retry mechanism for 5XX responses.
client.html_api(url, params={...}, retries=5)Scrapy is the most popular Python web scraping framework. You can easily integrate ScrapingBee's API with the Scrapy middleware.