1. What types of tests do you offer?
The answer should be: any type of test you need, on-demand, at scale. Available testing options should span from automated to in-depth manual testing. Industry analysts will tell you no single testing tool can detect all application security vulnerabilities. Vendors should have the ability to utilize multiple best-of-breed tools, with customizations that match your business needs.
Keep in mind: automated testing alone is not sufficient to provide a complete picture of your vulnerabilities. To defend against multi-step attacks or ones that involve social engineering, your vendor should be able to conduct in-depth manual testing to mirror the perspective of a hacker.
2. How will your tests match my risk profile?
Your vendor should have the expertise to apply different testing strategies based on the risk level and unique requirements of each of your applications.
The right vendor will help you create a full inventory of your applications and rank them according to security risk. They’ll design a testing plan so you can focus time and money on the things that matter most.
3. When you find vulnerabilities, how will you help me fix them?
Classic application security testing vendors consider their job to be just that—running tests. Vendors with a holistic approach provide remediation guidance to empower you to fix issues and address causes so fewer security issues ever reach the testing phase.
Make sure you understand how reports are created and verified. You’ll be more confident if you know every testing report is reviewed by a security expert to eliminate false positives. The top vendors will also include contextual remediation guidance in all reports along with the vulnerabilities they find.
Vendors should review findings with you directly. They should include developers in report read-outs to detail causes of vulnerabilities and remediation advice. Even after the initial test read-out, they should provide on-demand live remediation support.
4. How well do you know the security compliance requirements for my industry?
If your applications are subject to industry-specific requirements (PCI DSS, HIPAA, etc.), make sure the vendor includes compliance testing. As regulations are becoming stricter and penalties for non-compliance are increasing, it’s essential that your vendor is proactive in providing guidance to you on any actions you need to make.
Your vendor should help you do more than simply meet minimum requirements for compliance, by including compliance as part of a broader application security strategy.
5. How will you demonstrate success?
To see whether application security testing has been worth the investment, consider how your vendor’s approach will help you answer the following:
- How quickly are tests run compared with any previous system you used?
- Are testing methods identifying vulnerabilities previously missed?
- Are you saving time provisioning secure production applications?
- Are your developers improving over time? What is the defect density of new code?
- Is design improving? Or are tests finding the same flaws over and over?
- Do developers now have time they can use on other projects?