Security autotests for measurable and stable application security processes

1. Introduction
   • The reason behind security autotests
2. Security Autotests
3. How to create security autotest
   • Writing security autotest for validating response header
   • Customising security autotests
4. Security autotests: Fitting use cases
   • Input validation
   • Security Headers are present
   • Verification session token after logout
5. Ready-to-use templates for security autotests
6. Summary

Introduction #

Software security development is a repeatable process, and some steps could be automated to free up the valuable time of security and software engineers. By including security automation steps into the development pipeline, we improve software security posture, allowing for prevention of security regressions through frequent security verification.

Let’s explore how we make security measurable, and ensure a stable security testing process using security autotests. We will provide you with examples of frequently used security autotests and their applications to demonstrate their benefits.

The reason behind security autotests #

During the security assessment of a web application or API, auditors often identify security weaknesses and vulnerabilities, and communicate these findings to the developers for improvements. It is critical to verify if these issues have indeed been resolved. The verification process might be time-consuming, considering the gap between the reporting and fixing the security issue. Security engineers should understand the intricacies of the found issue, review how it manifests, attempt a replication, fix and retest.

However, throughout the development, previously resolved issues may reappear. Detecting these re-emerging issues promptly can be difficult. If left unaddressed, these vulnerabilities can pose significant risks to software security. Detecting those can be a time-consuming process.

Security Autotests #

Security autotests are intended to automate repetitive tasks in order to improve security assessment processes, and they are used by developers and QA engineers for various checks.

Automating vulnerability scans within the CI/CD pipeline offers a double win: It boosts development efficiency and strengthens long-term security. We can achieve this by integrating automated tests and revisiting previously identified vulnerabilities throughout the Secure Software Development Life Cycle (SSDLC).

Read how we built the SSDLC process from scratch and shifted security efforts from firefighting to proactive work, making security more manageable for the development team.

M&A solution

SaaS

Building security for M&A solutions: 5-years of SSDLC

Long-term cooperation with the [REDACTED] company resulted in refined security practices that matched the evolution of their B2B platform. We enabled transition from reactive "firefighting" to a proactive security posture, while simplifying security processes and building security defences.

Read story

Security autotests function as automated vulnerability detectors. They send requests to a web application and compare the responses to pre-defined expectations. This comparison helps to identify potential vulnerabilities and security issues in the application.

For example, we identified a security weakness associated with the ‘Server’ HTTP response header. This header discloses server information and its version, making it susceptible to known attacks.

To automate the detection of this issue, we’ve constructed a test request for the server . The test analysed the ‘Server’ header within the received HTTP response and compared it against a predefined value (specifically, information we don’t expect to see). This approach helps to quickly identify how long a vulnerability has been active, enabling a timely fix.

This is a simple example of a security autotest:

import request 

response = request.get(url="https://cossacklabs.com") 
header = response.headers['Server'].lower()

if "1.2.1" in header or "nginx" in header: 
    print("Not passed") 
else:
    print("Passed")

To put it simply, we’ve developed a code that checks whether a previously identified vulnerability has reappeared or been resolved.

Note: Security autotests may not always be precise, and not all vulnerabilities are worth automating.

Manual checks are more effective for highly critical or complex vulnerabilities (involve making multiple requests, parsing responses, performing MFA, and so on.). Sometimes it’s better to focus on additional inspections than spend more time on automation. It’s all about striking a balance.

Common security issues are perfect candidates for automation. Since they happen so often, it’s worth creating automated tests to catch them. We will discuss those later.

To make the best use of effort invested, we suggest creating clear internal guidelines for building and organising security autotests.

Pay attention to:

• programming language and its version (e.g. python 3.9)
• dependencies/libraries
• code structure (naming, functions, comments, files)
• structure of input and output parameters
• installation process (Dockerise to prevent installation fails)

How to create security autotest #

Creating security autotests should be quick and easy. Here is how we recommend building your own autotests collection.

First: use templates. Instead of starting from scratch each time, we recommend using Burp Suite, a web security toolkit, to capture a relevant request and inject it into a pre-built template. We can then make minor adjustments to fit the specific scenario we are testing.

Second: Python makes work faster. Burp Suite’s “Copy as Python-Requests” extension speeds up the test creation process. It takes a captured request, automatically converting it into Python code using the popular “requests” library. This code has everything you need, including headers, cookies, and the URL body. By automating this step, Burp Suite eliminates the need to write the Python code yourself, saving you even more time.

Note: While tools like DevTools or Postman can be used as alternatives to Burp Suite, this specific scenario requires an extension for Burp Suite. You can use other tools (DevTools or Postman) and find a way to automate the conversion of HTTP requests to Python code or other format.

Example: Writing security autotest for validating response header #

Now, let’s create security autotests to ensure that Cossack Labs’ site doesn’t leak the nginx version.

1. Install “Copy as Python-Requests” from the BApp Store or GitHub.

2. Pick a network request you want to automate for security testing:

3. Paste the copied request into the IDE.

import requests

burp0_url = "https://www.cossacklabs.com:443/search/?query=test"
burp0_headers = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "Accept-Language": "en-CA,en-US;q=0.7,en;q=0.3", "Accept-Encoding": "gzip, deflate, br", "Upgrade-Insecure-Requests": "1", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Te": "trailers"}

response = requests.get(burp0_url, headers=burp0_headers)

We store the server’s response in the ‘response’ variable to compare it later with the expected outcome.

4. Let’s check that the HTTP header Server does not specify the nginx version.

import request

burp0_url = "https://www.cossacklabs.com:443/search/?query=test" 
burp0_headers = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "Accept-Language": "en-CA,en-US;q=0.7,en;q=0.3", "Accept-Encoding": "gzip, deflate, br", "Upgrade-Insecure-Requests": "1", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Te": "trailers"} 

response = requests.get(burp0_url, headers=burp0_headers) 
header = response.headers['Server'].lower() 

if "1.2.1" in header or "nginx" in header:
    print("Not passed") 
else: 
    print("Passed")

Customising security autotests #

The specific checks performed by your tests depend entirely on the situation being tested. In some instances, you might need to confirm the existence of a particular word or phrase within the server’s response. In other scenarios, the focus might be on verifying if a numerical value falls within a predefined range.

This adaptability allows you to tailor the logic used in your tests to address the unique needs of each test case. For more complex situations, you can use regular expressions (regex) to identify specific patterns within the response data. To check the business logic deeper, you might need to include multiple functions, loops and constants, for more thorough and extensive tests.

In the following sections we will provide more examples of autotests, so you will gain a better understanding of how to create and use autotests in your own work.

Security autotests: Fitting use cases #

Automated tests can catch many vulnerabilities, but they aren’t a guaranteed solution.

Let’s take a typical security issue “reflected Cross-Site Scripting” (XSS) as an example. It is technically possible to automate tests for this, but it requires a deep understanding of the specific security measures in place (like character filtering, encoding, HTTP headers manipulation, or other practices).

Until we fully understand what to look for in an automated test (the exact request to make and response to check), manual testing is often more efficient for these types of vulnerabilities.

Here’s another challenge: We can’t predict what security measures developers will implement and what exactly we should test. This means we can’t automate tests for every single vulnerability or its fix.

However, automated tests are perfect for more straightforward and consistent conditions. For example, misconfigurations are usually systemic issues that can be identified through automated scans, HTTP headers can be checked for specific security best-practices, and input validation tests can verify the robustness of the system’s input fields against various forms of input data.

Here is a brief list of frequently repeated automated tests that we’ve found useful in our experience.

Input validation #

Input validation is a cornerstone of security automation testing. Here’s how it works: We bombard the system with various requests aimed at testing its ability to handle potentially risky characters or excessively long text. These requests are designed to trick the system.

Ideally, the server should respond with an HTTP status code of either 400 (Bad Request) or 403 (Forbidden). These codes signal that the server has successfully identified the request as suspicious and rejected it.

But checking the status code isn’t enough. We also examine the server’s response body. Here, we’re looking for a specific match between the data we sent in our request and what the server sends back. This step verifies if the server is processing and responding to our requests correctly.

In some cases, we might send an additional request as part of the validation process. This is particularly helpful when creating a user and the server doesn’t return a response body. We send a separate request to retrieve information about the newly created user. After this, we analyse this information to see if it contains any data that violates our input validation rules.

By thoroughly validating user input, we achieve two key goals: First, we ensure the server can accurately identify and reject malicious requests. Second, we make sure that any associated data is stored and processed correctly. This approach allows us to identify and address a broad spectrum of potential security vulnerabilities, ultimately strengthening our system’s overall security.

Security Headers are present #

Let’s check if the backend service response contains required security headers. We make a request to the server, receive a response, and check headers – simple and fast.

response = requests.get(url="https://site.com/logout")
headers = ['Strict-Transport-Security',
          'Content-Security-Policy',
          'X-Content-Type-Options',
          'X-Frame-Options']

missing_headers = [header for header in headers if header not in response.headers]

if not missing_headers:
    print("Passed")
else:
    print("Not passed")

For a deeper understanding of HTTP Security Headers, their testing procedures, and how to implement them, refer to OWASP HTTP Headers Cheat Sheet. It’s an excellent guide, providing valuable insights into enhancing web application security.

Verification session token after logout #

Many backend developers overlook session revocation during the user logout process. We frequently see this issue when JWT tokens are used, as they are designed to be client-side tokens. Therefore instead of revocation, developers simply delete them from client-side storage, which is insufficient.

Our test verifies whether session cookies or tokens remain valid after the user logs out. This process involves the following steps:

1. Initiating a login request and receiving a session.
2. Storing the received session token in a variable.
3. Logging out.
4. Re-attempting a request with the stored token such as to /api/users/me, ensuring the token is revoked.
5. Validating the response: If the server returns an HTTP 200 OK response with user information in the body, then the token is not revoked after logout, and the security issue is valid.

cookies = login()
requests.get(url="https://site.com/logout", cookies=cookies, headers=headers) #logout
response = requests.get(url="https://site.com/api/me", cookies=cookies, headers=headers)

if str(response.status_code) == "200":
    print("Not passed")
else:
    print("Passed")

Note: This list is not exhaustive. The optimal automated test suite will depend on your application’s unique context and relevant security requirements. Aim for a balanced approach, utilising the strengths of both automated and manual testing to achieve comprehensive security coverage.

Ready-to-use templates for security autotests #

In Cossack Labs GitHub repository you can find ready-to-use security testing scenarios that illustrate the approach discussed.

Check Security Autotests Templates

Check out the repository, copy templates to your workspace, and focus on customising and tailoring tests to your specific needs:

• Base template
• Input validation
• Security headers validation
• Rate limits validation
• User enumeration
• And more.

Here is an example of a complete security autotest template that you can use to automate the security process.

import requests

PASSED = "Passed"
NOT_PASSED = "Not passed"

MESSAGE = '' # Message that will be compared in HTTP responses
verifications = [] # A list that used in verify_response() to collect results of verification (Trues and Falses)

def send_request():
    # There will be your request from Burp
    verify_response(response)

def verify_response(response):
    http = False
    message = False


    # Verification HTTP response status codes 
    # (e.g. if you want the server to respond with HTTP 403 Forbidden  
    # for requests with malicious characters). 
    # You can remove this check if it is not required in your autotests
    if str(response.status_code) == "200":
        http = False
    else:
        http = True


    # Verification for the expected MESSAGE in the HTTP response body  
    # (e.g. if it is vulnerable it will respond "Success", if not the
    # response will not contain that message). 
    # You can remove this check if it is not required in your autotests
    if MESSAGE in str(response.text):
        message = False
    else:
        message = True


    # Appending the results of checks to the verifications[] 
    # so that you can later check it for PASSED or NOT PASSED checks.   
    # You can change the items that will be added to the list, 
    # for example, adding "HTTP_CODE":response.status_code and output   
    # the HTTP code of each check
    verifications.append(
        {
            "HTTP": http,
            "MESSAGE": message
        }
    )

# Checking for PASSED or NOT PASSED checks and printing results
def print_results():
    for verification in verifications:
        if verification["HTTP"] is False or verification["MESSAGE"] is False:
            print(NOT_PASSED, verification)
        else:
            print(PASSED, verification)

send_request()   
print_results()

Summary #

Security autotests are scripts that automate repetitive tasks, saving time, preventing regressions, and ensuring stable security posture. Integrating them into the software development process allows us to revisit previously identified vulnerabilities throughout the Secure Software Development Life Cycle (SSDLC) and make the security process more manageable. You can devote time to advanced security engineering work instead of doing mundane tasks.

However, security autotests are not a foolproof solution, thus striking a balance between automated and manual testing is crucial for comprehensive security coverage. Tailor the testing approach to your specific application and security needs.