Python grequests: Making Asynchronous HTTP Requests

In the world of web development, making HTTP requests is a common task. Whether you’re fetching data from an API, scraping a website, or communicating with a remote server, the ability to perform HTTP requests efficiently is crucial. Python provides several libraries and modules for making HTTP requests, and one of the most popular ones is grequests, which allows you to make asynchronous HTTP requests.

This tutorial will introduce you to grequests and show you how to use it to perform asynchronous HTTP requests in Python.

What are Asynchronous HTTP Requests?

When you make an HTTP request using the traditional synchronous approach, your program waits for a response from the server before continuing with its execution. This can lead to wasted time, especially if the server’s response time is slow or if you need to make multiple requests.

Asynchronous HTTP requests, on the other hand, allow you to send multiple requests simultaneously and handle their responses as they become available. This can significantly improve the efficiency and responsiveness of your application.

What is grequests?

grequests is a Python library that provides an elegant way to make asynchronous HTTP requests using the popular requests library. It builds on top of the gevent library, which is a coroutine-based networking library. With grequests, you can send multiple HTTP requests in parallel, effectively leveraging the power of asynchronous programming without having to deal with complex concurrency management.

To get started with grequests, you need to install it first. You can install it via pip:

pip install grequests

Once you have grequests installed, you’re ready to start making asynchronous HTTP requests.

Making Synchronous HTTP Requests

Before diving into asynchronous requests with grequests, let’s briefly review how to make synchronous HTTP requests using the requests library, which grequests is built upon. Just to get an idea on how things are normally done, for those who have never used the requests library before.

In this synchronous example, we send a GET request using the get() method, to the specified URL and then wait for the response. The program blocks until it receives a response.

import requests

url = 'https://jsonplaceholder.typicode.com/posts/1'
response = requests.get(url)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print('Request failed with status code:', response.status_code)

Note: The requests library is included within the Standard Python Library

Making Asynchronous HTTP Requests with grequests

Now, let’s see how to make asynchronous HTTP requests with grequests. The basic process involves creating a list of request objects and sending them in parallel. We will walk you through the code for this, step-by-step.

First, we import the module, and create a list of URLs which we intend to fetch some data from. We are using a placeholder site which will actually return us some json data upon a successful request.

import grequests

urls = [
    'https://jsonplaceholder.typicode.com/posts/1',
    'https://jsonplaceholder.typicode.com/posts/2',
    'https://jsonplaceholder.typicode.com/posts/3'
]

We then iterate over this list of urls, calling the get() method from the grequests module on each url. Unlike the requests library however, the get() method will not actually trigger any fetch calls to the specified url. It only creates a request object, and does not send it. To actually execute the requests, we will pass it into the map() method, which runs all of these requests in parallel.

# Create a list of request objects
requests = (grequests.get(url) for url in urls)

# Send the requests in parallel
responses = grequests.map(requests)

The map() method returns a list of responses, which we can then iterate over, and if the request was successful, we will acquire the json data from the response, and print out the data. Otherwise we will trigger an error message to the user.

for response in responses:
    if response.status_code == 200:
        data = response.json()
        print(data)
    else:
        print('Request failed with status code:', response.status_code)

Here is our output after running this script:

{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}
{'userId': 1, 'id': 2, 'title': 'qui est esse', 'body': 'est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla'}
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}

Improving Performance

It is recommended by the authors of the library to use imap over map for speed and performance gains. All you have to do is change the map() method with imap(), and add in the size parameter (which controls how many events are done in parallel).

Here is a different example featuring the imap() method. Avoid keeping the size parameter too high, otherwise you could get blocked or rate-limited by whatever website you are making requests to.

import grequests

codes = range(200, 206)
rs = [grequests.get(f'https://httpbin.org/status/{code}') for code in codes]

for response in grequests.imap(rs, size=5):
     print(response)

<Response [202]>
<Response [203]>
<Response [204]>
<Response [201]>
<Response [200]>
<Response [205]>

Since these requests were sent in parallel, there is no fixed order in which they arrive. As you can see from the output, the response values are mixed around.

Limitations of grequests

While grequests has the ability to make multiple fetch calls (requests) to various urls in parallel, this is still a blocking process. This means that when you call the grequests “map” function, this blocks the execution of your program, until all requests have been completed, and responses have been returned.

This is still useful, as instead of doing each request at a time, it is now being done in parallel (faster execution, and less blocking). However, this is not fully asynchronous, like we would ideally want it to be.

In order to achieve a fully asynchronous way of fetching data, you can make sure of multi-threading or the async library to run the grequests code on a separate thread. Many other possible solutions exist, but none within grequests itself (so far).

This marks the end of the Making Asynchronous HTTP Requests with Python grequests tutorial. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the tutorial content can be asked in the comments section below.

Share on Facebook

Tweet

Follow us