How to benchmark a web application security scanner?

There is a plethora of web application scanner; every one of which claims to be better than the other. It is indeed a challenge to differentiate between them. We need to benchmark the application scanner against hard facts and not marketing claims. Below are some of the most critical metrics against which you would like to benchmark web application scanner.


1. What is the rate of false positives?

False Positives are vulnerabilities reported by a tool that don’t actually exist.  Any web application scanner will throw some false positives.  First we need to understand how false positives are harmful. Even though  they don’t apparently seem to be harmful; it costs money to remove them. Imagine a little bit of sand in your food. You can’t eat that food; similarly you can’t send a report with false positives to developers.


Removing false positives from web application scanner reports takes a lot of time. Hence it adds to your man-power cost and of course the drudgery of doing boring work. I have seen so many organization losing people because the work becomes monotonous.


So, you need to check the percentage of false positives reported by the web application scanner. The flip side however is that a web application scanner can minimize its percentage of false positives by limiting its coverage which leads to the next question.


( Read More: Identity & Access Management (Workshop Presentation )


2. How many classes (or percentage) of vulnerabilities does it cover?

False negatives or vulnerabilities missed out is another critical element. You need to understand the percentage coverage of the web application scanner to ensure that critical vulnerabilities are not missed (particularly at the expense of not having to report false positives). You can use WASC 1, WASC 2 or OWASP as a guideline for what should be covered.


3. Which are the classes web application scanner does not cover?

If a web application scanner does not cover certain classes of test (which is always the case), you should know: which are those classes? How important are the classes of test for your business? Can you live without them?


4. How good is the coverage of the crawler? Is there any benchmark?

Crawlers are the fundamental part of any web application scanner. The first step of any testing is crawling. If a page is not crawled then it is not tested. You can benchmark different web application scanner against the number or the percentage of the pages it could crawl. Fast scanning does not mean good scanning. You need a web application scanner which can comprehensively crawl all the pages.


5. How many scans can run in parallel?

Most organizations today have multiple web applications which need to be tested frequently.  You need aweb application scanner which can scan multiple tests in parallel. Don’t go by the number stated on the product datasheet but how many it can actually run in parallel without significant degradation of performance. So the best thing is to try it and check this out yourself.


( Read More: CISO Platform Top IT Security Influencers (Part 1) )


6. How Flexible are the configuration options of the tool?

Does the tool give you the ability to fine tune what test classes it scans for and let you test your production environment safely? Options that allow you to prevent things like automatic form filling, or limiting the number of concurrent threads etc. can prevent unnecessary disruption to your organization when testing your production environment with a tool.

Few more suggestions by readers and community members


Credits: Simon Bennetts, James McGovern, Keighley Peters

  • How long does it take to run? (Quicker means it could be less comprehensive test. Check for number of tests/hour etc)
  • How long does it take to learn and configure to work effectively?
  • How much does it cost?
  • What are the licensing terms?
  • How many organizations use the tool? How satisfied are they?
  • Are there any industry recognition/analysts mentions (e.g. Gartner)?

The selection of appropriate scanner can be very challenging as every organization has developed their applications differently. By considering the metrics discussed above, organizations can benchmark their application scanner to evaluate the effectiveness of a scanner and make a right choice for their organization.

Views: 206

Join the Discussion ...

You need to be a member of CISO Platform to join the discussion!

Join CISO Platform


Security Trends and Emerging Technologies That A CISO Should Adopt In 2021

Started by Priyanka Aash Mar 3. 0 Replies

What are the challenges you as a CISO have been facing since the last year and share some security trends that are catching up? Help the community by sharing your knowledge and personal views on this subject. Or if you have any specific questions…Continue

CISO as an enabler

Started by Maheshkumar Vagadiya Jul 30, 2020. 0 Replies

Share the instances where you were able to convince the Executive management /board that CISO function is enabler rather then a hindrance.Thanks youMaheshContinue

Has Anyone Evaluated Digital Signature (like Docusign)?

Started by CISO Platform. Last reply by Yogesh Nov 19, 2020. 2 Replies

(question posted on behalf of a CISO member)Has anyone evaluated digital signature (like Docusign), any specific risk/ security areas to be looked into while finalising a vendor? Any and all inputs will be very much appreciated.Continue

What are your strategies for using Zoom in your organization after recent vulnerabilities in news about Zoom platform?

Started by CISO Platform. Last reply by ANAND SHRIMALI May 20, 2020. 4 Replies

(question posted on behalf of a CISO member)What are your strategies for using Zoom in your organization after recent vulnerabilities in news about Zoom platform?Related Question: …Continue

Follow us

Contact Us


Mobile: +91 99002 62585

InfoSec Media Private Limited,First Floor,# 48,Dr DV Gundappa Road, Basavanagudi,Bangalore,Karnataka - 560004

© 2021   Created by CISO Platform.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service

/* */