The Bug Bounty Business: How Solution Providers Are Cashing In

Bug bounty programs are nearly as old as the internet itself. But today they’re big business for solution providers that can offer complementary consulting, triage and remediation services.

Crowdsourced security vulnerability testing was a completely uncharted business opportunity for the channel as recently as four years ago. But where other solution providers saw a competitive threat to the services they already provide, NCC Group saw an opening.

The Manchester, England-based information security assessment and consulting firm is the first solution provider to offer services to help customers establish their own bug bounty programs. Software companies and other businesses use bug bounty programs to compensate outsiders for uncovering and reporting flaws in their hardware or applications. After enjoying some early success, NCC — which has 11 offices in North America — broke out its bug bounty activities into a separate practice in 2015.

Today, the 2,000-employee company brings in millions of dollars in revenue each year by providing consulting, scoping, triage and remediation services for customers that have launched bug bounty programs. The net financial impact of NCC's services is almost incalculable, given that it is helping customers uncover hard-to-find security vulnerabilities and fixing them before those customers become the next big breach headline-maker like Equifax or Sony.

[Related: The 25 Tech Bug Bounty Programs With The Biggest Payouts]

One of NCC's enterprise software customers leveraged a bug bounty program to discover a flaw in its payments API. This bug report revealed that the customer's validation process failed to ensure the account number provided to the API in the final step of the payment process was a real account number, said NCC.

Hackers could have exploited the flaw to supply their own account numbers, which would have resulted in the customer's account appearing to be properly paid, when in reality the funds would be transferred to the hacker's account, a scenario NCC's customer was happy to have avoided in exchange for a $5,000 bounty to the intrepid bug hunter.

"It's all about transitioning clients from fear of vulnerabilities to embracing security vulnerabilities, and helping them understand that there are already bad people looking at their products and not telling them," said Adam Ruddermann, who joined NCC in late 2017 from Facebook, where he led the social media giant's bug bounty program. "It's way better to have that knowledge and insight and be aware of what's really happening out there."

As part of NCC's managed bug bounty service, Ruddermann meets with senior management, product managers and public relations to help them understand what it's like to work with security researchers. NCC also walks the customer through the life cycle of a bug to prepare them from an operations standpoint, he said.

NCC's internal security team, meanwhile, goes through the customer's product in detail to understand it better and capture any issues the customer already knows about, Ruddermann said. The company also examines the history of the product's security architecture so that NCC can better understand what it's looking at when doing triage work.

The core of NCC's managed bug bounty offering is its triage service, which Ruddermann said is focused on determining whether a submitted vulnerability is a misunderstanding of the product, something that's mitigated elsewhere, or a legitimate security flaw. Some foreign researchers also struggle with creating a technical write-up in English, so NCC can help figure out what the researcher is getting at.

From there, Ruddermann said NCC ensures it knows how to properly execute the vulnerability so that the company can fix it correctly. Finally, he said NCC completely rewrites the vulnerability report, providing more details on reproduction steps so that it's easier for a product team not familiar with security tools to re-create the bug.

A look at some of the companies providing platforms on which to run bug bounty programs shows that the industry has hit an inflection point. One such platform, HackerOne, has paid out $23 million in bounties as of early 2018, up from $10 million two years earlier, with the number of registered users on the platform skyrocketing ten-fold during that time.

Competing platform Bugcrowd has paid out $6.4 million to security researchers as of 2017, up 211 percent from a year earlier. The number of hackers on the platform has nearly doubled from 26,800 to 53,300 during that time.

The opportunity to crowdsource the spotting of security bugs comes as the cost of exploited vulnerabilities continues to rise. The average financial impact of a data breach in North America in 2017 was $1.3 million for enterprises and $117,000 for SMBs with 50 or more employees, according to a Kaspersky Lab survey, up markedly from $620,000 for enterprises and $46,000 for SMBs in 2015.

Putting Meltdown And Spectre In The Spotlight

Bug bounty programs are nearly as old as the internet itself. Netscape launched the first-ever "bugs bounty" all the way back in October 1995 when it debuted the beta version of Netscape Navigator 2.0, offering cash prizes to users that reported significant security issues and Netscape merchandise to users finding any security flaws whatsoever.

But it wasn't until the early 2010s that bug bounties really started to gain traction as companies like Microsoft, Google, Facebook and Mozilla rolled out programs of their own. The industry hit more milestones in March 2017 as Google increased its bounty awards by 50 percent, Microsoft doubled its top bounty award, and Intel started paying researchers $30,000 for spotting critical hardware flaws.

Nine months after Intel established its bug bounty program, a team of security researchers from Austria's Graz University of Technology reached out to the Santa Clara, Calif.-based semiconductor giant and let it know that they had found a design-level flaw in its chips that could slow down almost every processor in the world.

The vulnerabilities — which had also been found independently months earlier by Jann Horn from Google Project Zero — became known to the world in January 2018 as Meltdown and Spectre and are found in chips from other vendors such as IBM and ARM as well. Intel awarded the researchers a bug bounty for abiding by responsible disclosure guidelines and not revealing the issue prematurely, although it became public earlier than expected after it was reported by several media outlets.

Six weeks after Meltdown and Spectre came to light, Intel said it was raising its bounty awards across the board and launching a new, limited-time program focused specifically on side-channel vulnerabilities. The side-channel program will run until Dec. 31 and offer awards of up to $250,000 for critical issues.

In addition, the top reward in the regular bug bounty program (for critical hardware flaws) was increased from $30,000 to $100,000. Maximum awards for identifying firmware or hardware vulnerabilities in the regular program — regardless of the bug's severity — have at least doubled, and often tripled.

"We believe these changes will enable us to more broadly engage the security research community, and provide better incentives for coordinated response and disclosure that help protect our customers and their data," Rick Echevarria, Intel's vice president and general manager of platform security, said in a Feb. 14 blog post.

Why Pay For Bug Reports?

Companies without a formal vulnerability disclosure policy often remain in the dark about known flaws in their architecture, with nearly one in four hackers not reporting a vulnerability they've found due in part to fear of reprisal for conducting unauthorized testing on a company's products. And if they do end up reaching out, they might do so through a suboptimal method such as a LinkedIn post or public Tweet.

"It's super-undesirable, because it's visible to the whole world," said Michiel Prins, co-founder of HackerOne, San Francisco. "These things happen if there's no process in place."

Vulnerability disclosure programs are therefore becoming accepted as an industry best practice, and are today recommended by everyone from the U.S. Department of Defense to the Food and Drug Administration.

In fact, one of the Senate's first questions to Equifax following its disclosure of a massive data breach was, "Do you have a vulnerability disclosure program?" said Casey Ellis, founder, chairman and CTO of Bugcrowd, which partners with VARs, systems integrators, managed security service providers and security consultants to provide greater access to the company's offering.

But despite the regulatory guidance, HackerOne has found that just 6 percent of the Forbes Global 2000 companies have a known vulnerability disclosure policy. Still, there are signs of progress, with the share of bug bounty programs from industries other than technology climbing from 28 percent in 2014 to 41 percent in 2016 as more government, media, banking and e-commerce players establish programs.

Bug bounties weren't on the radar of San Antonio-based Denim Group or its customers as recently as 18 months ago. Today, bug bounty-related services account for roughly 5 percent of Denim Group's security revenue.

"It's not yet a significant portion of our business, but it's growing quickly," said Dan Cornell, principal and CTO of the software security advisory company. "It's something that is very interesting to us."

Denim Group's differentiator is fully understanding where a bug bounty program fits into the customer's overall security strategy, Cornell said. As a result, Cornell said Denim Group can effectively determine on a customer-specific basis which security issues should be addressed through threat modeling or internal penetration testing and which should be sent out to the broader researcher community via bug bounty.

Cornell also has tasked some of his team with evaluating and providing context around the vulnerabilities being reported through HackerOne and Bugcrowd. Denim Group doesn't have a dedicated bug bounty practice today, but Cornell said that could certainly change in the next year or two.

"Even with the stuff the bug bounty providers are doing, we've found that a number of organizations still need additional support on top of that in order to provide appropriate context and vetting of the reports that are coming in," Cornell said.

Solution providers can help with sorting the wheat from the chaff, as roughly 30 percent of reports provided through HackerOne's public bug bounty programs in 2017 were marked as spam, not applicable, or something of a similar nature. Plus, vulnerability submissions as a whole have been on the rise, with a 67 percent increase in reports made through the Bugcrowd platform between 2016 and 2017.

Is Your Customer Ready For A Bug Bounty?

The first step any solution provider faces is determining whether or not the customer is ready to set up a paid program.

Bug bounties are typically taken up by customers who are already engaged with specialized penetration testing companies, yet are still struggling to keep pace with changes in the application and infrastructure ecosystem, according to Francesco Faenzi, head of the cybersecurity business platform for Italian solution provider Lutech.

High-profile organizations with mature application security programs are typically the ones to adopt bug bounties, said Denim Group's Cornell. Straightforward or obvious business flaws should be caught through internal testing programs, he said, meaning that the only thing companies are left having to pay bounties on are serious or more subtle vulnerabilities in applications that aren't self-evident.

"Bug bounties are increasingly becoming a material portion of the testing market," Cornell said.

The first couple of security tests against an organization's ecosystem should be carried out by an individual or small number of firms prior to initiating the bug bounty, according to Andrew Howard, CTO at Phoenix-based solution provider Kudelski Security.

Customers that push the bug bounty before they're ready will be inundated with reports and end up paying out for rather obvious errors, Howard said.

Businesses should understand what their overall security environment looks like before embarking upon a bug bounty program, as well as where their potential weaknesses might be, said Lisa Wiswell, a principal at security engineering and consulting firm Grimm. The organizations should have a test environment or virtual lab space where security researchers can poke and prod the ecosystem for flaws without jeopardizing the actual corporate environment, she said.

Well under half of GuidePoint Security's customers would even consider something like a bug bounty program, said Andrew Johnson, vice president of innovative security assessments for the company, No. 134 on the 2017 CRN Solution Provider 500.

Modern, agile, web-centric companies are typically the ones looking into bug bounties, he said, as well as a handful of progressive financial services and health-care firms.

Bug bounties open to the general public are just the tip of the iceberg, though. Some 88 percent of HackerOne's programs and 77 percent of Bugcrowd's programs are actually private, meaning that only researchers that receive an invitation are allowed to participate in — or even know about — the program.

Security vendor Sophos runs both public and private bug bounties, focusing its public effort around inspecting web interfaces and its private program around products, said Craig Paradis, application security architect for the Oxfordshire, England-based security vendor. Products are a better fit for the private program since it requires more time and investment from researchers to install and reverse-engineer them, Paradis said.

Businesses often find it easier to get a private program off the ground with a small number of invited security researchers who are both vetted and more well-known, said NCC's Ruddermann.

Public programs are likely to attract a much greater volume of vulnerability submissions, which Ruddermann said many companies lack the budget or staffing to support from the get-go.

"There's a path to that maturity," Ruddermann said. "Private programs are a great place to start. And public programs might be an ideal place to end up."

Click to expand.

Selling Bug Bounties Outside IT

Although public programs made up just 8 percent of the HackerOne bug bounties launched in the past year, they pay bounties and resolve issues at four times the rate of their private counterparts. But they require buy-in from a much larger share of the organization to get off the ground.

Involvement in an organization's private bug bounty program is often limited to the product and security teams, according to Bugcrowd's Ellis. But with a public program, everyone from legal and purchasing to marketing and public relations must be looped in.

Security teams aren't historically the most public-facing part of an organization, Ruddermann said, and working externally in a way that attracts media attention might be new for them. As a result, he said the security teams sometimes struggle to get senior management and public relations on board with a public program. "A public program could really stress the internal bureaucracy of a company," Ruddermann said.

The biggest obstacle Lutech has encountered in getting bug bounty programs off the ground has been the purchasing office, Faenzi said. No RFP template currently exists for crowd-based vulnerability management services, according to Faenzi, and it's been difficult to fit bug bounties into the purchasing framework used for traditional penetration testing services.

Lutech therefore needs to work with organizations that have a politically strong CSO who's able to convince the purchasing office to avoid all of the RFP issues and move forward without a tender. Once Lutech's bug bounty services offering becomes more established, though, Faenzi believes the sales cycle will become shorter and far less frustrating.

"It's definitely not mainstream yet," Faenzi said.

Organizations also must establish lines of communication and set expectations with the security researcher community before launching a program, said Justin Morehouse, founder and principal at GuidePoint Security. Specifically, Morehouse said researchers must understand the time frame within which the firm expects to work.

Manning The Vulnerability Floodgates

Prospects considering crowd-based security testing often turn to Lutech to ensure that approach makes sense for them, Faenzi said. Lutech is in a strong position to scope the bug bounty proposals since it knows the customer's applications and services as well as their business processes and the potential impact of any vulnerabilities on its operations, he said.

Denim Group's focus out of the gate is on providing appropriate context and evaluation around the vulnerabilities submitted, Cornell said. Denim Group's staff has deep background in the security around software and system building, and Cornell said they're well-positioned to assess how much of an impact the vulnerability would have on systems and users as well as the actual risk associated with the flaw.

"Everybody thinks that their vulnerability is the worst thing that has ever happened, and it needs to be addressed immediately," Cornell said.

Sophos has tasked six individuals around the world with triaging vulnerabilities that come in through the company's bug bounty program and following them through to completion, Paradis said. The members of this team have a strong understanding of application security, he said, and excel at gauging the severity of a flaw as well as the impact it would have within a particular product organization.

As far as remediation is concerned, Johnson said customers are increasingly turning to solution providers such as Guide- Point to serve as a liaison between the security team and the development team to help the customer work through all of the reported vulnerabilities.

"Once these programs go into effect, usually during the first couple of weeks, there's just a ridiculous influx of findings," Johnson said.

Exterminating The Bugs

Some end users are only looking for help facilitating low-level tasks such as managing the ticketing system, Johnson said. But in many instances, Johnson said organizations find that the talented developers they employ aren't all that security-savvy, meaning they need outside guidance on how to actually fix the vulnerabilities.

GuidePoint has expertise across network and application-related issues, including the two most commonly pursued bugs: cross-site scripting, where malicious scripts are injected into trusted websites; and SQL injection, which uses malicious SQL code to steal data. The company can set up conference calls to talk fixes, he said, or walk customers through how to address in more detail if many pieces are affected.

Lutech is well-situated to support remediation efforts for telecom customers since the solution provider is a recognized subject matter expert in the domain. Faenzi said Lutech is very familiar with the best security practices around both the configuration and management of back-end telecom infrastructure and customer-premises equipment.

NCC, meanwhile, can provide customers with a dedicated, on-site program manager to drive vulnerability remediation if the customer is expecting a large volume of bug reports from multiple sources, Ruddermann said. NCC also excels at analyzing highly complex, esoteric bugs to provide the customer with a deeper understanding of the vulnerability itself, he said.

Ruddermann said NCC helps the customer drive security fixes by serving as a point of contact between a customer's security team and their product teams and ensuring the vulnerability is addressed in a time frame that's consistent with its severity.

For its part, Cornell said Denim Group has a lot of back-end experience designing and developing systems, and is therefore able to provide customers with step-by-step advice around how to change code to deal with the vulnerabilities. The company can address both the business implications of the flaw as well as what constitutes proper mitigation from a technical standpoint, Cornell said.

Denim Group stands apart from its peers in that the solution provider is willing to take over code bases and make the requisite fixes themselves, Cornell said. The company steps in when there's no value to the end user maintaining their own knowledge around how to build or evolve the application, which he said typically happens when an application is reaching end-of-life or isn't under active development.

"One of the things that's rare about our services is that we will actually take over and fix vulnerabilities for organizations where that's a good fit," Cornell said.

’Good Dollars And Sense'

One of the biggest selling points of bug bounties over hiring outside consultants to assess an organization's security posture is the cost associated with each, said Grimm's Wiswell.

Before joining Grimm in April 2017, Wiswell worked at the U.S. Department of Defense and created the federal government's first bug bounty program, known affectionately as "Hack the Pentagon." The Defense Department spent roughly $150,000 on "Hack the Pentagon," with security researchers finding 138 flaws in the five publicly facing web domains included in the scope of the program. By chaining together a number of vulnerabilities, Wiswell said hackers were able to identify issues that were meaningful or mission-critical to the government.

"It's really starting to make good dollars and sense," Wiswell said.

Before launching "Hack the Pentagon," Wiswell said the Defense Department paid a third-party vulnerability defense firm $150,000 to find and report security flaws on the same five websites, but the company averaged just 11 submissions annually over a three-year period.

The Defense Department tasked internal employees to think like adversaries and go after the same sites in a "red team" scenario, but Wiswell said they only ended up reporting 12 or 13 flaws during the three-week simulation. Breach simulations typically take between two and four weeks, though eight- to 12-week exercises aren't uncommon at larger firms, solution providers said.

Bounty size factors significantly into which organizations security researchers decide to target, said HackerOne's Prins. Organizations might pay as little as $1,000 for critical flaws when first launching a bounty since hackers often enjoy considerable success finding issues in a program's early days, he said.

But as organizations fix more vulnerabilities and their attack surface hardens, Prins said they also need to mature their bounty structure in order to keep the researcher community's interest. Google Chrome has exceled at this, HackerOne found, with the business increasing its top bounty from just $3,000 to $100,000 over the course of little more than five years.

"Every time the bounty increases, a new caliber of researcher becomes interested in you," Prins said.

Bugcrowd, meanwhile, advised in 2016 that highly mature security organizations pay $15,000 for critical vulnerabilities, somewhat mature companies pay $5,000 for critical flaws, and less mature firms from a security standpoint pay $1,500 for critical issues. Kudelski Security's Howard said companies need to offer a five-digit payout for major vulnerabilities to attract the interest of major research firms.

Payouts for both average and critical vulnerabilities are up across the industry. The average overall vulnerability payout clocked in at $451 in 2017, Bugcrowd found, up from $295 in 2016 and nearly double the typical payout in 2015. Bounties associated with critical vulnerabilities, meanwhile, increased to $1,923 in 2017, HackerOne found, up nearly 20 percent from a typical dispersal of $1,624 in 2015.

What's Next For Bug Bounties?

Going forward, the greenest pastures for bug bounty programs might be beyond North America. Some $17.2 million — or 72.6 percent — of the $23.6 million in bounties paid thus far through HackerOne's platform come from companies based within the U.S. or Canada.

Lutech started looking into providing crowd-based vulnerability management services in Italy last quarter, Faenzi said. The Cologno Monzese, Italy-based solution provider has actively explored the opportunity with two prospects, and Faenzi said the feedback from their high-level technical personnel has been strongly positive.

Faenzi believes Lutech will be able to provide customers with a more intimate experience during the vulnerability validation phase since his staff will be able to converse with clients in fluent Italian. The company is therefore searching for opportunities where it can measure the effectiveness of its approach in the growing field.

"I really want to try and be successful in this kind of service," Faenzi said. "It's one of the goals I have in mind for 2018." Like both Bugcrowd and HackerOne, new NCC bug bounty services leader Ruddermann is based in San Francisco. That's fairly par for the course, Ruddermann said, with California companies being the first major adopters of bug bounty and North American-based businesses receiving most of the attention thus far.

But NCC has a truly global footprint, with 11 offices in the U.K., eight sites in Continental Europe, and four locations across Asia and Oceania. This leaves Ruddermann with a golden ticket to bring NCC's unique bug bounty services into some largely untapped markets.

"There's tons of opportunity globally to expand the number of bug bounty programs in more traditional, non-technology companies overseas," he said.