In this series of posts, Victoria Willis explores how OSINT (Open Source Intelligence) can be applied in the areas of Cyber Threat Intelligence, IT Asset Discovery, Security Assessments and Attack Surface Monitoring. In this third post of the series, the focus is on the relevance of OSINT when performing security assessments and penetration tests. Check out the previous posts on using OSINT for Cyber Threat Intelligence and OSINT for IT Asset Discovery.

Identifying a target’s digital footprint should be a critical step in all security assessments. Whether you’re preparing for a vulnerability scan or pen test, Open Source Intelligence (OSINT) is a valuable resource that provides organizations with a full picture of what their assets (and possible exposures) look like through open data sources that are available to defenders and attackers alike. By collecting OSINT, shadow IT elements can also be pinpointed ahead of time and added to scope for a more complete assessment.

Lost or long-forgotten shadow IT assets may be so outdated that they can be immediately recognized as unpatched and vulnerable. Beyond forgotten servers, poorly configured S3 buckets and unintended exposures can be identified through OSINT collection, as well.

Determining a target’s full digital footprint provides security operations with actionable intelligence on low-hanging fruit, allowing them to rectify these issues ahead of a vulnerability scan or pen test — in addition to providing teams with necessary information on what should be included in scope.

Assessing Security With OSINT

Employing OSINT collection as an initial step in security assessments is crucial to identifying potential areas of exposure which may not otherwise be included in a vulnerability scan. For example, the information gathered through OSINT may reveal exposed S3 buckets, previously breached accounts, leaked credentials, sensitive information that’s been indexed by search engines and systems that aren’t being properly maintained or unintentionally exposed to the outside world.

Poorly configured S3 buckets in particular are a common problem and may leave sensitive information vulnerable. Through the first few months of 2020 there have been multiple instances of customer data getting exposed through misconfigured S3 buckets; in June, 20 million files were exposed via open S3 buckets across a number of different dating apps. And in July, it was reported that a fitness brand accidentally exposed one million files — including personally identifying information on nearly 100,000 people. Leaked credentials are another major concern; recent estimates suggest that there are over 15 billion credentials from over 100,000 data breaches available on the dark web.

After identifying new or exposed assets and leaked credentials through OSINT, security operations can expand the scope of their vulnerability management program to incorporate newly identified assets, and take action to mitigate malicious activity where necessary. This is where automation becomes pivotal.

Asset Discovery and Vulnerability Identification

Asset discovery is a critical early phase in any comprehensive security assessment. Through OSINT collection, organizations can get a more complete picture of what their public-facing assets look like and what shadow IT may exist; this allows for more complete vulnerability scanning, and gives organizations insights as to what newly identified assets may need protection or taken offline.

Since we are dealing with entities like hostnames, e-mail addresses and others and obtaining much of the data in structured form, using an automated OSINT collection tool can be extremely helpful in this arena. Hundreds of different sources of OSINT exist and manually compiling and sorting data from each source can be an overly complicated and time-consuming process. SpiderFoot provides users with the ability to automate much of the labor involved with OSINT collection, allowing security operations to focus more on analysis and remediation.

Once collected, the data can be used to generate threat intelligence and fed into SIEM tooling, vulnerability scanning efforts, incident response processes and much more. Automation provides organizations with the ability to seamlessly translate OSINT into actionable intelligence that can improve overall cybersecurity posture.