Attack Surface Management: The Case for Project Discovery

Defining and assessing the attack surface of the typical enterprise has never been an easy task. The enterprise IT environment has gotten more dynamic, and thus more difficult to assess, with each phase in the infrastructure’s evolution. As organizations transition to the cloud, the attack surface becomes even more dynamic and amorphous, encompassing thousands of applications and services, along with their supporting protocols and innumerable APIs. Innocent misconfigurations of cloud services can create an instant vulnerability, while CI/CD pipelines add, delete, and modify services, interfaces, and protocols continuously. 

Because new risks arise quickly and without warning, attack surface management in the cloud requires continuous asset discovery and assessment operations, ideally leveraging the cloud’s automation and scale. Unfortunately, assessment tools haven’t kept pace with the increasing dynamism that cloud environments enable. While there are more than a few commercially available assessment tools--such as vulnerability scanners—they haven’t evolved significantly in over a decade, imposing the following limitations:

  • Inflexibility: Engineers can’t design custom workflows, matching their assessments to their environments, priorities, and issues. 

  • Slow response to new problems: Adding newly discovered vulnerabilities to assessment frameworks can be a complex and painfully slow process, requiring vendors to update software, security engineers to write complex testing scripts, or both. 

  • Lack of automation and extensibility: Frameworks don’t easily accommodate the multi-staged nature of vulnerability testing that security engineers need to perform. 

  • Lack of integration and scale: While databases such as Shodan and CenSys provide excellent resources, few products offer large-scale integration capabilities with these and the other tools security professionals need to do their jobs. Engineers are forced to cobble together brittle workarounds to get the job done.  

Simply put, today’s tools lack the levels of automation, integration, and orchestration that security engineers need to perform attack surface assessments in the cloud. 

The Mother of Invention

The founders of ProjectDiscovery were all dealing with the cumbersome and challenging nature of asset reconnaissance as a part of their day jobs as application security engineers, bug bounty hunters, and analysts. Frustrated with the lack of functionality in commercially available products, they turned to the open-source community. They met each other as leading contributors to the GitHub repository for Subfinder, a subdomain discovery tool. 

Subfinder was started by Ice3man543, the handle for Nizamul, a bug bounty hunter involved in several open-source projects. Nizamul, Sandeep Singh, Marco Rivoli, and Rishiraj Sharma all started talking about the tooling they’d like to see. They decided to build a set of integrated open-source tools that gave them the functionality and flexibility they needed to get their jobs done. Ice3man543 moved the Subfinder repository into the newly created ProjectDiscovery repository on GitHub, and the group began building tools, essentially in the order that they needed them in their own workflows.

None of the developers started ProjectDiscovery intending to build a company. They worked with the open-source community in their spare time, creating the tools they needed to do their jobs. But as other security engineers caught wind of the project – and the functionality that the tools had to offer – the GitHub repository started getting more and more attention. After posting the second tool (Naabu, a port enumeration utility), the project got 100 stars in just a few weeks. The team kept up their part-time work, posting additional tools. ProjectDiscovery had 10,000 stars and several hundred contributors to a suite of attack surface management tools within a year.

VCs began to call, of course, but the team largely ignored them, preferring to keep their day jobs and work on the tools part-time. But then Caleb Sima, vice president of security at Databricks and a user of the tools, got involved, mentoring the team and helping them find investors experienced in funding open-source companies. ProjectDiscovery was born as a company, and after a seed round, the founders went full-time in February of 2021. 

The company and its investors are wholly committed to the open-source approach, ensuring that the core tools will be free to use as they evolve, benefitting from the continued improvement that open source contributors make to the project. Today, the company remains focused on building the core set of open source tools. Over the longer term, however, the company will create a platform service that will integrate those tools, providing a common interface and higher levels of functionality for enterprise customers. 

ProjectDiscovery’s Approach

ProjectDiscovery built its attack surface management tools from the ground up, based on the founder’s deep experience with what it takes to discover and assess cloud assets continuously. The suite of tools the team continues to develop brings new levels of functionality to attack service management through:

  • Automation: Like the cloud environment itself, ProjectDiscovery’s tools allow engineers to automate tasks, often in multiple stages, to bring scale and speed to bear on the problem. Users can write custom workflows. 

  • Extensibility: Using simple YAML templates, security engineers can quickly customize the vulnerability scanner to meet their specific needs.

  • Integration and Scale: ProjectDiscovery uses a distributed microservices architecture for flexibility and scale. Engineers can leverage public or private data sources as needed. Discovery data are stored in the cloud and are indexed and searchable. A custom query language allows security engineers to perform sophisticated data operations. Tools can send notifications to Slack and Discord.

  • Community Development: The community around ProjectDiscovery contributes new templates that others can reuse. The simplicity around the YAML-based template process ensures that the team can make most templates available in the repository with 24 hours of submission. 

The Tools

Today, ProjectDiscovery includes some 34 repositories, but six primary tools form the foundation of the suite. As of this writing, those six tools collectively have over 11,000 stars on GitHub. The tools all work with each other in various combinations, depending on the needs of the user. The following figure shows an example workflow using those tools.

CLI-Workflow.jpg

Those tools are:

  • subfinder: A subdomain discovery tool that uses passive online sources. It leverages the DNS data in the Chaos service as well as third-party resources. 

  • naabu: A port scanner focused on simplicity and speed. 

  • proxify: An HTTP/HTTPS proxy that supports request/response dump, filtering, and manipulation. It includes a replay utility.

  • dnsx: A multi-purpose DNS toolkit that supports multiple probes, performing multiple DNS queries with a list of user-supplied resolvers.

  • httpx: An multi-purpose HTTP toolkit that supports multiple probes.

  • nuclei: A vulnerability scanning engine. Instead of writing vulnerability checks into the product, the team created a scalable engine based on a simple, YAML-based template syntax. Users can quickly customize the engine to fit their needs, and a carefully curated (and growing) repository of templates is available.

Future Directions

As we said earlier, ProjectDiscovery remains focused on building and improving the core tools, working with the open-source community as it always has. The InfoSec community is collaborative by nature, and both the company and its investors see continuing the collaboration through the open-source model as a core goal for the company. Now that they are full-time, the founders expect to accelerate the improvements to the toolset. (The team released Nuclei v2.3 in early March.) 

As those tools continue to mature, ProjectDiscovery will build a cloud-based platform service for continuous asset discovery and vulnerability detection. The platform will integrate the individual tools – which will always be free to use on their own – into a common interface, providing higher levels of integration and scale for companies with significant needs. (If you’re interested, you can join the waitlist.) 

Conclusion

The sheer volume of interest in and contributors to ProjectDiscovery is a clear indication of the market need for better attack surface management tools for cloud environments. The overwhelmingly positive response to the tools ProjectDiscovery is creating demonstrates that the team not only understands the need but how to meet it with effective tools, leveraging the collaborative nature of the InfoSec community. Those dynamics drove our investment in the business, and we’re thrilled to see the project move into its next phase.

Jamie Lewiscloud security, Attack surface management, cybersecurity