Pytest for Pentesters
In this blog entry we will discuss how, as a Pen Tester, you can utilize tools and methodologies commonly used by software developers to find API security issues in a quick and efficient manner. You’ll need at least a basic knowledge of Python and REST APIs to get started.
You may have tested REST APIs before. Perhaps you used the Burp OpenAPI Parser plugin, and fed examples in Repeater or Intruder with custom payloads. While this certainly works for smaller or simpler APIs, it can quickly become unwieldy.
What if instead, we utilize some of the methodologies developers use to test their own APIs? Here are a few examples of when using a more code-focused approach could help save you time and provide better coverage:
The service takes a large number of parameters.
The parameters are highly interrelated or dependent (e.g. an order must be placed before payment can be processed)
There are a large number of routes, many with similar parameters (e.g. 15 routes all rely on an ‘order’ parameter).
You’re on an internal team, so you will be testing the same services repeatedly as features are added and bugs are fixed.
First a brief overview of software testing: Most developers think in terms of three types of testing:
Acceptance Testing: This may be completely manual, or partially automated, but it usually involves an actual web browser and tools like Selenium or Postman. This tends to be handled by a QA team.
Unit Testing: Checking a small piece of code with mock data, often a single function.
Integration Testing: Actually validating an API will communicate with the backend, returning valid data and performing expected actions (sometimes this is broken into a separate category, System Testing).
In this blog, we will focus on Integration Testing with a tool called pytest and will test for authorization issues.
Getting Started
The code used in this blog article is located here, the README includes instructions for creating a Virtual Environment and getting credentials.
Then we will create our first test. In our first example we will utilize Damn Vulnerable Web Services to get you familiarized with the process, and then move on to a more complex and realistic example.
We’ll want to make sure the endpoint /api/v2/sysinfo/uname which is supposed to be accessible to administrators only, is indeed only accessible to administrators (Vertical Privilege).
import requests
import pytest
HOST = "http://localhost"
tokens = {}
# Fixture to authenticate users
@pytest.fixture(autouse=True)
def authenticate_users():
tokens['user'] = authenticate_user("mike", "test")
tokens['admin'] = authenticate_user("admin", "letmein")
# Function to authenticate a user and return a bearer token
def authenticate_user(username, password):
data = {"username": username, "password": password}
response = requests.post(HOST + "/api/v2/login", data=data)
bearer = response.json()["token"]
return bearer
# Test for system information endpoint, which should be admin only.
def test_sysinfo():
user_response = requests.get(
HOST + "/api/v2/sysinfo/uname",
headers={"Authorization": "Bearer " + tokens['user']}
)
admin_response = requests.get(
HOST + "/api/v2/sysinfo/uname",
headers={"Authorization": "Bearer " + tokens['admin']}
)
# Assert that the admin can access the page, but the user cannot.
assert admin_response.status_code == 200 and user_response.status_code == 403
This code starts with a fixture. A fixture is the baseline set up we need to run our tests. In this case, we’ll want to authenticate two different users. This code will be run before we run our tests.
Once we authenticate our two users, we will do an HTTP get to an “Administrative Only” endpoint and compare the results. We can then run the test as follows:
As you can see our test fails, no surprise that Damn Vulnerable Web Services are vulnerable.
Useful pytest flags to know are:
-s Turns off capture to display output. For example, if you’re printing debugging statements.
-k Run individual tests that match the substring. Useful if you wanted to run all tests with invoice in the name for example.
-v Verbose mode for troubleshooting.Now you may be thinking “That’s nice, but I could have done that in two seconds with Auth Matrix and Burp. Why would I bother with this?”. So let’s move on to a more realistic example: PayPal with it’s well documented API and Bug Bounty program.
Checking the PayPal API for Invoice IDOR
Below is a snippet of code which again takes two users, this time two PayPal developer users and validates that userB cannot see userA invoices.
import requests
import pytest
from paypal_auth import authenticateUser
import os
HOST = "https://api-m.sandbox.paypal.com"
tokens = {}
@pytest.fixture(autouse=True)
def authenticate_users():
tokens['userA'] = authenticateUser(os.getenv['USERACLIENT'], os.getenv['USERASECRET'])
tokens['userB'] = authenticateUser(os.getenv['USERBCLIENT'], os.getenv['USERBSECRET'])
def test_get_invoices():
user_a_invoices = []
# Get userA's invoices
user_response_a = requests.get(
HOST + "/v2/invoicing/invoices?total_required=true&fields=amount",
headers={"Authorization": "Bearer " + tokens['userA']})
# Populate userA's invoices list
for item in user_response_a.json().get('items', []):
user_a_invoices.append(item['id'])
# Access userB's invoices with userA's token (testing for IDOR)
for invoice in user_a_invoices:
user_response_b = requests.get(
HOST + f"/v2/invoicing/invoices/{invoice}",
headers={"Authorization": "Bearer " + tokens['userB']
})
if user_response_b.status_code != 403:
assert False
assert True
In this case we make an API call as userA to list invoices, and then try to retrieve them as userB. If we do not receive a 403 something has gone wrong. But the real magic of of using pytest is when you need to answer the question "What if userA has no invoices?".
Pytest fixtures and helper functions can be used not only to authenticate prior to our test, but to set up our testing environment. In cases where developers are re-deploying your test environment, or you're re-testing months after the fact, this saves an immense of amount of time. In the case of testing a destructive function (such as one that deleted invoices), it also saves your sanity.
Let's modify our code to create an invoice if one does not exist.
import requests
import pytest
from paypal_auth import authenticateUser
import os
import random
import string
from payloads import INVOICE_PAYLOAD
HOST = "https://api-m.sandbox.paypal.com"
tokens = {}
@pytest.fixture(autouse=True)
def authenticate_users():
tokens['userA'] = authenticateUser(os.getenv('USERACLIENT'), os.getenv('USERASECRET'))
tokens['userB'] = authenticateUser(os.getenv('USERBCLIENT'), os.getenv('USERBSECRET'))
def create_invoice(user):
# Create an invoice and return the ID. Invoice numbers must be unique.
invoice_body = INVOICE_PAYLOAD
invoice_body['detail']['invoice_number'] = ''.join(random.choice(string.ascii_letters) for _ in range(24))
user_response = requests.post(HOST + f"/v2/invoicing/invoices", json=invoice_body, headers={"Authorization": "Bearer " + tokens[user], "Prefer": "return=representation" })
return user_response.json()['id']
def delete_invoice(invoice_id, user):
requests.delete(HOST + f"/v2/invoicing/invoices/{invoice_id}", headers={"Authorization": "Bearer " + tokens[user] })
def test_get_invoices():
user_a_invoices = []
clean_up = False
# Get userA's invoices
user_response_a = requests.get(
HOST + "/v2/invoicing/invoices?total_required=true&fields=amount",
headers={"Authorization": "Bearer " + tokens['userA']})
# Populate userA's invoices list
for item in user_response_a.json().get('items', []):
user_a_invoices.append(item['id'])
# If userA does not have an invoice, let's create one!
if not user_a_invoices:
user_a_invoices.append(create_invoice("userA"))
clean_up = True
# Access userB's invoices with userA's token (testing for IDOR)
for invoice in user_a_invoices:
user_response_b = requests.get(
HOST + f"/v2/invoicing/invoices/{invoice}",
headers={"Authorization": "Bearer " + tokens['userB']
})
if user_response_b.status_code != 403:
if clean_up:
delete_invoice(invoice,"userA")
assert False
if clean_up:
delete_invoice(user_a_invoices[0], "userA")
assert True
In this case, the code will validate if our users has any invoices. If not, we will create one by issuing a POST request as specified in the API Documentation. Good sources of JSON payloads can be Open API Definitions, Postman Collections, or in the case of PayPal’s well documented API their documentation. Code generation features can also be utilized to quickly put these together. Just bear in mind in JSON false is lowercase, and in Python it is uppercase!
To begin, as we did previously, we authenticate our user and query invoices. If there are no invoices, we import our JSON blob and provide a random number. We then add this invoice to our list, test access and then finally delete the invoice.
In this way we can validate our test case even after the environment we've been working in has been destroyed and redeployed. This also opens up the possibility of running these tests as part of a CI/CD pipeline. In this manner developers can ensure after each new deploy they haven’t introduced new security issues or regressions.
There is much more you can do once you have started to test via code, for example we could start feeding our payloads through PyJFuzz or Radamsa to test for more issues. But that is a subject for another blog article.
An Important Caveat: The examples above are lacking things such Exception handling, rate limiting and other features for the sake of brevity and clarity. If you are going to consider using test frameworks repeatedly you should definitely ensure your API connection isn’t timing out or you aren’t accidently DOSing your target. Many of these will end up being re-useable code fragments you can utilize elsewhere.
I hope you have found this article useful, happy pentesting!
