Home Resources Python Series Python: HTTP library: requests: part 1

Python: HTTP library: requests: part 1

The robust HTTP library for Python

January 13, 2019

653

Python requests library takes away all the manual labor in sending HTTP requests. It’s features range from passing parameters in URLs to sending custom headers.

Clone demo from github

Let’s look at a simple example to understand requests. The below code snippet sends a get request to unixutils.com, and print only the headers and URL from the information retrieved from get request.

#!/usr/bin/python
import requests

req = requests.get('https://unixutils.com/')
print "URL:",req.url
print "HEADERS:",req.headers

Output:

URL: https://unixutils.com/
HEADERS: {'Content-Length': '22892', 'X-Powered-By': 'PHP/7.1.18', 'Set-Cookie': 'anspress_session=9b850eb03e2f751a0c93341463c82806; expires=Mon, 14-Jan-2019 13:09:18 GMT; Max-Age=86400; path=/', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding,User-Agent', 'Keep-Alive': 'timeout=5', 'Server': 'Apache', 'Connection': 'Keep-Alive', 'Link': '; rel="https://api.w.org/", ; rel=shortlink', 'Date': 'Sun, 13 Jan 2019 13:09:18 GMT', 'Content-Type': 'text/html; charset=UTF-8'}

Now, we try to take this one step further and pass parameters to the URL, and also extract the URL, HTTP status code and the title of the webpage.

import requests
import re

# Pass parameters to Get request
payload = {'s': 'python'}
req = requests.get('https://unixutils.com/', params=payload)

# print URL and Status code 
print "URL:",req.url
print "HTTP Status code:",req.status_code

result = re.search('<title>(.*)</title>', req.content)
print "Title of the visited URL:",result.group(1)

# req.content returns the HTML content of the page, we use regex to extract the title section from the content.

Output:

URL: https://unixutils.com/?s=python
HTTP Status code: 200
Title of the visited URL: You searched for python - UnixUtils

From the output, we see that the URL has been constructed with parameter “?s=python” which makes a search on string “python” in unixutils.com. The results page has a title “You searched for python – UnixUtils” which is also seen in the output.

HTTP protocol provides different methods of request, which are GET, POST, PUT, HEAD, DELETE, PATCH & OPTIONS, and all of these requests can be sent easily with requests module.

We had a quick introduction to requests module in this post. requests in my opinion is the simplest and robust HTTP library available with amazing capabilities. We will explore its different features and capabilities with examples in future posts.