A penetration test is three things: the testing, the report, and the debrief. Buy one without the other two and you have bought an exercise nobody can act on.
The testing itself happens out of sight, over a few days, and by the time it matters to you it has finished. What you can judge, act on, hand to a customer or take to a board arrives in the report and the conversation that follows it. Which makes it remarkable how rarely anyone asks what will be in either before signing the order.
Report quality is not a matter of taste
Buyers often assume there is no standard to hold a provider to. There are several, and they broadly agree.
Methodology standards define how the testing is carried out and therefore what the report should be able to describe. The Penetration Testing Execution Standard sets out the phases of an engagement from pre-engagement interactions through to reporting. The Open Worldwide Application Security Project guidance, particularly the Web Security Testing Guide and the Application Security Verification Standard, does the same for web applications and gives findings a common reference. A report that names the methodology it followed lets you judge whether coverage matched the objective, and lets you compare this year’s test with next year’s.
Scheme requirements. Both CREST and The Cyber Scheme set expectations around reporting quality and the competence of the people delivering the work, including that the individuals involved are identifiable rather than anonymous. Buying from an accredited provider is partly buying that assurance.
And the government’s own minimum, which is the most specific published list and a perfectly reasonable bar for any provider, government work or not. The Cabinet Office requires that the report is readable and accessible to the customer, with a clear summary of the number, type and severity of issues identified, and that Common Vulnerability Scoring System base scores are included where possible, with version 3 or 3.1 preferred. It requires the report to name the individuals involved in the work. It requires the background, scope and context to be communicated in full. It requires vulnerabilities to be accurately identified and explained. And it requires that each identified vulnerability is associated with a remedial solution, while noting that the remedy should not be treated as the only way to reduce the risk, since short term measures such as network segregation, limiting access, increased monitoring and further hardening may be appropriate until a strategic fix is possible.
Source: IT Health Check (ITHC): supporting guidance, Cabinet Office. Contains public sector information licensed under the Open Government Licence v3.0. © Crown copyright.
Read that list against the last report your organisation received. If it fails on more than one line, you did not buy what you thought you bought.
What a good report contains
- An executive summary a director can act on. Not a restatement of the findings in smaller type. What was tested, what the overall picture looks like, what the most serious issue means in business terms, and what to do first. If your board reads one page, this is it.
- Scope, stated precisely. What was in, what was out, what was agreed as off limits. A report that does not define its boundaries invites the reader to assume everything was covered, which is how a clean report becomes false comfort.
- Limitations, stated honestly. The environment that was unavailable on the day. The credentials that never arrived. The system that could not be tested without risking production. Every real engagement has some; a report with none has either been sanitised or was never that thorough.
- Methodology, named. Which standard the engagement followed, such as the Penetration Testing Execution Standard for infrastructure work or the Open Worldwide Application Security Project testing guides for web applications. "Industry best practice" is not a methodology; it is a phrase used when there was not one.
- The people who did the work. Named, with their qualifications. Recognised schemes expect it and the government standard requires it, and it is the fastest way to check that the senior tester in the proposal was the person on the engagement.
- Findings with real severity and real impact. A scoring system such as Common Vulnerability Scoring System version 3.1 for consistency, and alongside it, what the finding actually means for your business. A medium severity issue on a system holding client data usually matters more than a high severity issue on an isolated test box.
- Evidence for each finding. Enough to reproduce it: the request, the response, the screenshot, the steps taken. Without evidence your technical team cannot verify the fix, and cannot challenge a finding that is wrong.
- A remedy for every finding. Specific to your environment, not a paragraph copied from a vendor advisory, and ideally split into the immediate containment and the strategic fix.
- The attack narrative. The part no scanner produces: how individually unremarkable findings were chained into something serious. This is usually the most valuable page in the document.
- A prioritised action plan. Ordered by risk to your business, with an indication of effort. A list sorted by severity score is a starting point, not a plan.
Two reports, same test
The difference between a good and a bad report is not length. It is whether someone thought about your organisation.
| Weak report | Report worth paying for | |
|---|---|---|
| Executive summary | Counts of findings by severity | What it means, what to fix first, and why |
| Scope | A list of addresses | What was covered, what was not, and what that leaves unanswered |
| Limitations | Absent | Stated plainly, including what could not be tested |
| Findings | Tool output, lightly edited | Verified, explained in context, false positives removed |
| Severity | Scanner score, unadjusted | Scored consistently, then interpreted against your business |
| Evidence | A screenshot of the scanner | Reproduction steps a technical team can follow |
| Remediation | “Apply vendor patch” | Specific action, plus interim mitigation where a fix takes time |
| Attack paths | None | Chains shown end to end with evidence |
| After delivery | Emailed, then silence | Presented live, questions answered, retest arranged |
Hand the report to someone technical in your organisation and ask whether they could fix the top three findings from it alone. Then hand the summary to a director and ask what they would do about it. If the answer to either is no, the document has failed regardless of how many pages it runs to.
The debrief is part of the deliverable
The debrief is not a courtesy call. It is the third part of what you paid for, and a report emailed without one transfers the work of understanding it onto you at the exact moment you know least about it.
A live debrief, delivered by the person who did the testing, is where the value lands. It is where you find out which finding genuinely worries them, which one looks alarming but does not matter much in your context, and what they would fix on Monday morning if it were their business. None of that survives being written down.
Ask whether the tester will present. Ask whether it is the same person named in the report. And ask what happens afterwards, because the questions arrive in the fortnight after the debrief, once someone has actually tried to fix something.
Retesting is where testing becomes improvement
A finding is not closed because a report says it was fixed. It is closed when someone competent has verified the fix works and has not introduced something else.
Ask what is included before you sign: whether retesting of the original findings is part of the engagement, how long you have to fix things before that window closes, and what a retest report looks like. A provider who charges separately to check their own findings is selling a document. A provider who expects to come back is selling an outcome.
Eight questions to ask before you commission
- Can I see a redacted example report before I commit?
- Who will do the testing, and will they be named in the report?
- What scoring system is used, and is business impact assessed separately from the score?
- Does every finding come with a remedy specific to our environment?
- Are limitations recorded, including anything that could not be tested?
- Will the tester present the findings to us, live?
- Is retesting included, and for how long?
- What format do we receive, and can we share it with a customer or an auditor?
The example report answers most of the others by itself. Any provider confident in their work has one ready.
Where we stand
Our penetration testing is delivered by testers certified through The Cyber Scheme, following established methodology: the Penetration Testing Execution Standard for infrastructure engagements and the Open Worldwide Application Security Project guides for web applications. The report is written by the person who did the work rather than assembled from tool output by someone who was not there. Findings carry evidence and a remedy, limitations are stated, and the debrief is part of the engagement rather than an extra.
If you want to see what that looks like before deciding, ask for the example report. Get in touch or call 01722 445972. We have also written about what you are actually buying when choosing between assessment and testing, and about what the badges on a proposal mean. More detail on the service is on our penetration testing page.