Security Testing

Security testing checks that your app can't be made to do things it shouldn't. Passing tests tell you the features work for honest users; security testing asks what happens when someone sends input you never planned for.

Why passing tests aren't enough

A normal test suite checks expected behaviour: log in with the right password and you get in, add an item and the basket total goes up. An attacker doesn't use the app that way. They put a quote mark in the search box, change an ID in the URL, send a request ten thousand characters long, or look for a library with a published weakness.

Security testing covers that side. In practice it is a set of tools, each looking at a different part of the system, plus some way of proving which of their findings are real. The video names the three main families of scanner: SAST, DAST and SCA.

SAST: reading the code

Static application security testing (SAST) reads your source code without running it. It parses the code, then traces where data comes from and where it ends up. Anything a user controls, such as a form field or a query string, is treated as untrusted. If untrusted data reaches a dangerous place, such as a SQL query or a shell command, without being cleaned up on the way, the tool raises an alert.

This is the classic SQL injection that SAST flags:

# The user's input is pasted into the SQL
name = request.args["name"]
query = f"SELECT * FROM orders WHERE name = '{name}'"
cursor.execute(query)

A name like ' OR '1'='1 turns that query into one that returns every order. The fix is a parameterised query, where the driver sends the value separately from the SQL so it can never change the query's shape. With a driver such as psycopg it looks like this:

cursor.execute(
    "SELECT * FROM orders WHERE name = %s",
    (name,),
)

SAST also looks for hardcoded secrets: API keys and passwords written straight into the code. Once committed, a secret is in the repository's history for anyone with access, even after you delete the line, so the fix is to rotate the key and load the new one from the environment:

import os
 
API_KEY = os.environ["PAYMENTS_API_KEY"]

SAST runs early, on every commit or pull request, and points at the exact file and line. Its blind spot is everything outside the code, such as how the server is configured and how the deployed app behaves.

DAST: attacking the running app

Dynamic application security testing (DAST) works from the other end. It never sees the code. It points at a running copy of the app, usually a staging environment, and behaves like an attacker: it crawls the pages and API endpoints, then sends odd requests to each input and watches what comes back.

Those requests are built to break things: a stray quote, a <script> tag, a path like ../../etc/passwd, an empty value where a number should be. A request like this one is typical:

curl "https://staging.example.com/orders?name='"

If the app answers with a 500 error and a database message, the tool reports a likely SQL injection. DAST also catches problems that only exist at runtime, such as missing security headers, cookies without the Secure flag, or error pages that leak stack traces.

Its limits are the mirror of SAST's. It only tests what it can reach, so a page behind a login it can't pass is invisible to it, and it needs a deployed environment to run against. This outside-in approach is black box testing; Black box and White box testing covers the difference in more depth.

SCA: checking the libraries

Most of the code in a modern app was written by someone else. Software composition analysis (SCA) reads your dependency files and lock files, including the dependencies of your dependencies, and compares every package and version against databases of known vulnerabilities.

Most package managers have a basic version built in:

npm audit      # JavaScript
pip-audit      # Python, installed separately

Each finding names the package, the installed version, the published advisory and the version that fixes it. Often the fix is a version bump. Sometimes it isn't, because the patched release breaks something else, which is where Dependency Hell begins.

Here is what each family looks at:

The trouble with scanner output

All three kinds of tool help, and all three share the same problems.

The first is volume. An initial scan of a real codebase can report hundreds of issues, and each one needs a person to read it and decide what to do.

Many of those alerts are false positives. A false positive is an alert for a problem that isn't there. SAST may flag a query built from a variable that, in practice, only ever holds a fixed value. SCA flags a library because one of its functions is vulnerable, even if your code never calls that function. DAST sees a 500 error and suspects injection when the cause was an ordinary bug.

And even a correct alert doesn't prove the flaw can be exploited in your app. An injectable query that only admins can reach is a different risk from one on the public search page.

The cost is alert fatigue. When most warnings turn out to be noise, teams stop reading them, and the one real vulnerability sits in a list of three hundred with no way to tell it apart.

Proving a vulnerability is real

The traditional answer is a penetration test: a person, with permission, tries to break in and stops once they have proof. Pentests confirm what scanners only suspect, but they are slow and expensive, so most teams have one once a year. Code that ships every week spends most of that year untested. Automated Pentesting explains that gap.

The approach in the video, Aikido Infinite, runs pentesting on every release. When new code ships, it runs alongside the SAST, DAST and SCA scanners and launches autonomous pentest agents. The agents only test the areas the change affected. They explore the app, look for attack paths and try real exploits. If a vulnerability is real, Aikido confirms it and suggests a fix.

Two parts of that design deal with the problems above:

• Scoping each run to the change keeps it small enough to happen every release. A change to the checkout doesn't need the blog retested.
• Confirming by exploitation turns 'this might be a problem' into 'this is a problem, and here is how'. A confirmed finding goes to the top of the list, and the unconfirmed ones can wait.

A worked example: one release

Say a team adds a filter to the order history page. The developer builds the SQL with string formatting and, in the same pull request, adds a PDF library to export invoices.

1. SAST flags the new query, because user input reaches cursor.execute without parameters.
2. SCA flags the PDF library, because the installed version has a published advisory.
3. DAST, run against staging, gets a 500 error when it puts a quote in the filter.
4. A pentest step, scoped to the order history page, uses the filter to read another customer's orders. The injection is now confirmed, with steps to reproduce it.

The team now knows which finding to fix first. They switch the query to parameters, re-run the checks, and look at the library advisory to see whether the vulnerable feature is one they use.

Common mistakes

• Treating a clean scan as proof of safety. It means the tools found nothing they know how to find. Logic flaws, such as reading another user's data by changing an ID, often pass straight through.
• Letting alerts pile up. Triage the first big scan once, record why each ignored alert is safe, and keep new alerts at zero.
• Silencing the tool instead of fixing the bug. Every suppression needs a reason someone else can check.
• Deleting a leaked secret and moving on. It is still in the Git history, so rotate it.
• Attacking production, or systems you don't own. Run DAST and pentests against staging, with written permission for anything that isn't yours.

Key takeaways

• SAST reads the code, DAST attacks the running app, and SCA checks the libraries; each sees what the others miss.
• Scanners produce too many alerts and false positives, and an alert isn't proof that a flaw is exploitable.
• A pentest proves a vulnerability by exploiting it, and scoping it to each change lets it run every release.
• Fix confirmed findings first, and keep the rest of the scanner output small enough that people still read it.