Imagine having the ability to efficiently collect valuable intelligence from a wide range of platforms without the hassle of switching between tools—this is precisely what makes theHarvester stand out in the world of cyber reconnaissance.
While some premium tools may offer similar features, few match the flexibility and openness of theHarvester. As an open-source solution, it is incredibly user-friendly and can extract data from numerous sources, both free and premium. One of the most notable features is the seamless integration with a large number of tools and APIs, allowing user to gain even deeper insights during security assessments.
theHarvester, a tool crafted by the Edge-Security team, is a key utility for open-source intelligence (OSINT) gathering, helping professionals identify and assess potential security risks. Built on Python and designed for command-line use, it assists in the initial phase of cyber investigations, offering insights into an organization’s exposure on the internet.
Originally intended for penetration tests and red team operations, theharvester has evolved into a versatile tool used across different cybersecurity teams. Its passive data collection capabilities provide essential insights for both offensive and defensive security efforts. Additionally, its ease of integration with various data sources helps streamline reconnaissance efforts.
Here are the passive and active reconnaissance sources leveraged by theharvester, including some unique facts about each:
Passive Reconnaissance Sources:
Active Reconnaissance Sources:
These reconnaissance capabilities make theHarvester an indispensable tool for understanding the scope of a company’s exposure in the digital world, offering invaluable insights for security teams looking to secure their networks or assess vulnerabilities.
There are several ways to set up theHarvester, making it adaptable to various environments. The four primary methods of installation include:
For this tutorial, we will focus on the third option, setting up theHarvester from source. It’s recommended to use a secure environment, such as a virtual machine (VM), container, or isolated test server, when installing new software. In this case, we are using Ubuntu 20.04 and MacOS Sequoia 15.5 for the installation, though these instructions should work for most Debian-based systems, with minor adjustments needed for others.
Start by preparing the system and installing necessary dependencies:
sudo apt update
sudo apt upgrade
sudo apt install git python3-venvNext, we create a Python virtual environment to isolate the required dependencies:
python3 -m venv harvest
cd harvest/
source bin/activate
git clone https://github.com/laramies/theHarvester
cd theHarvester/
python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install .You can also use poetry to install dependencies:
pip install poetry
potry installBut it is important to note that the pyproject.toml file does not contain the project version, which can cause errors when installing dependencies. To avoid them, add the version manually.
After completing the installation, you can verify it by running the following command:
python theHarvester.py -hThis will display the help menu and confirm that the tool is ready for use.
Lastly, don’t forget to configure the API keys needed for certain functions. Harvester uses a mentioned YAML file called api-keys.yaml to store these keys. For testing, we’ll use free or freemium APIs, which can be added using a text editor like nano or vim:
cd theHarverster/data
nano -w api-keys.yamlThis step ensures theharvester is equipped to access the APIs and perform OSINT tasks efficiently.
How to Install theHarvester on MacOS
The method described above also works when installing the utility on MacOS. However, in addition to this, the tool can be installed much more easily using the Homebrew package manager:
brew install theharvesterThe api-keys.yaml file in this case will be located in the /Users/user/.theHarvester directory.
In this demonstration, we’ll leverage theHarvester to dig deeper into publicly available data by performing a basic search on a domain using various data sources.
For this test, let’s start with a simple query for the domain “target[.]com”, and always limiting our results to 50 entries.
To enhance our reconnaissance efforts, we can use theHarvester in conjunction with some powerful tools. Let’s execute the following command to gather data about the domain “target[.]com” using the SecurityTrails API as our data source:
python theHarvester.py -d target.com -l 50 -b securityTrailsThere was a problem here. It was possible to find one IP address associated with the target domain, but due to problems with response processing, it was not received. The utility returned the following results:
[*] Target: target.com
Read api-keys.yaml from /Users/user/.theHarvester/api-keys.yaml
Done Searching Results
[*] Searching SecurityTrails.
[*] IPs found: 1
-------------------
An exception has occurred while adding: _rank to ip_list: failed to detect a valid IP address from '_rank'
[*] No emails found.
[*] No people found.
[*] No hosts found.Not a pleasant result, but we will be able to find other data later. For now, let’s remember that using the SecurityTrails API is associated with a small bug.
The above result highlights an important aspect of OSINT tools: the availability of data can vary across different platforms. Sometimes, a particular source might not have information related to a specific domain, especially if the domain is not widely discussed or indexed on that platform. This is a natural limitation in the world of open-source intelligence, but it also underscores the flexibility of theHarvester, which allows users to continuously explore other data sources in pursuit of valuable information.
Here are a few points to consider when working with theHarvester and other OSINT tools:
Even with some sources not yielding results, theHarvester’s open-source nature ensures that it remains a powerful and adaptable tool in any OSINT investigation.
To start, we used theHarvester to perform an OSINT search for the domain “target[.]com” through the UrlScan API, limiting the results to 50 entries:
python theHarvester.py -d target.com -l 50 -b urlscanThis time the result was much more interesting. We managed to find the following resources associated with the domain: 3 ASNs, 3 interesting URLs, 24 IPs.
[*] Target: target.com
[*] Searching Urlscan.
[*] ASNS found: 3
--------------------
AS16509
AS27589
AS54113
[*] Interesting Urls found: 3
--------------------
https://678doo7bt5p9mx7n.u5hqmkfdt8rxif1w.www.fastly.gcp.ephemeral.target.com/
https://www.target.com/
https://www.target.com/bp/hyde%2Band%2Beek%21%2Bboutique%E2%84%A2
[*] IPs found: 24
-------------------
108.138.128.114
108.138.128.14
108.138.128.6
...
3.167.99.21
54.230.228.105
54.230.228.20
54.230.228.57
[*] No emails found.
[*] No people found.
[*] Hosts found: 0Primary OSINT sources like UrlScan are excellent for identifying IPs and hosts linked to a domain. One of the major advantages of theHarvester is its access to a wide range of data sources, so we’ll now extend our tests by using additional third-party services.
Next, we tried using ThreatMiner and RapidDNS:
python theHarvester.py -d target.com -l 50 -b threatminer
python theHarvester.py -d target.com -l 50 -b rapiddnsThreatMiner returns an error, but RapidDNS has allowed us to find over 2,500 hosts associated with the domain. Some of them are listed below:
[*] Target: target.com
[*] Searching Rapiddns.
[*] No IPs found.
[*] No emails found.
[*] No people found.
[*] Hosts found: 2532
---------------------
1view.target.com:161.225.130.61
2daymail.target.com:66.216.75.58
2daysweepstakes.target.com:rdr.target.com
2daysweepstakes.target.com:rdr.target.com.
2daysweepstakes.target.com:rdr3.ewips.target.com
Connectti.target.com:connectti.ewips.target.com
...
xyzshared.pf.target.com:xyzoffice365.pf.target.com
xyzshared.pf.target.com:xyzoffice365.pf.target.com.
xyzsso.pf.target.com:xyzoffice365.pf.target.com
xyzsso.pf.target.com:xyzoffice365.pf.target.com.An interesting tool for searching open data is also GitHub. You can search it as follows:
python theHarvester.py -d target.com -l 50 -b github-codeThis time we also managed to find several hosts, but besides them, for the first time we found email:
[*] Target: target.com
Read api-keys.yaml from /Users/user/.theHarvester/api-keys.yaml
Searching 0 results.
Searching 38 results.
[*] Searching Github-code.
[*] No IPs found.
[*] Emails found: 1
----------------------
[email protected]
[*] No people found.
[*] Hosts found: 8
---------------------
.target.com
admin.target.com
gsp.target.com
intl.target.com
login.target.com
tech.target.com
test.target.com
weeklyad.target.comTo broaden our investigation, we ran a comprehensive scan of the “target[.]com” domain across all available sources:
python theHarvester.py -d target.com -l 50 -b allAs a result of scanning we managed to get 1 email, 3210 hosts, 252 IP addresses, 3 ASNs and 3 URLs. There could have been more data, but for some tools requiring authorization, API codes were intentionally not entered.
It is also important to consider that some sources may block your IP address if you scan too often. To bypass this restriction, theHarvester offers a proxies:
By adding multiple proxy IPs to the proxies.yaml file, we can bypass some of the restrictions caused by frequent queries.
One feature we found a bit confusing was the --shodan option, which is not a data source but a flag that must be added for domain searches. After some experimentation, we found the correct usage:
python theHarvester.py -d target.com -l 50 -b urlscan --shodanShodan requires an associated IP address to work, and once it found one, it returned data on subdomains related to the domain. In our example, we first used Urslcan to find IP addresses, after which they will go to Shodan’s input.
The result was to obtain both the list of addresses shown above, and responses for some of them. It looks something like this:
[*] Target: target.com
[*] Searching Urlscan.
[*] ASNS found: 3
--------------------
AS16509
AS27589
AS54113
[*] Interesting Urls found: 3
--------------------
https://678doo7bt5p9mx7n.u5hqmkfdt8rxif1w.www.fastly.gcp.ephemeral.target.com/
https://www.target.com/
https://www.target.com/bp/hyde%2Band%2Beek%21%2Bboutique%E2%84%A2
[*] IPs found: 24
-------------------
108.138.128.114
...
54.230.228.57
[*] No emails found.
[*] No people found.
[*] Hosts found: 0
---------------------
[*] Searching Shodan.
Searching for 108.138.128.114
Read api-keys.yaml from /Users/user/.theHarvester/api-keys.yaml
{
"asn": "AS16509",
"domains": [
"cloudfront.net"
],
"hostnames": [
"server-108-138-128-114.jfk50.r.cloudfront.net"
],
"ip_str": "108.138.128.114",
"isp": "Amazon.com, Inc.",
"org": "Amazon.com, Inc.",
"ports": [
80,
443
],
"product": "",
"server": "CloudFront",
"technologies": [
"Amazon Web Services",
"Amazon CloudFront"
],
"title": "ERROR: The request could not be satisfied"
}
...Finally, we attempted to perform DNS brute forcing and take screenshots of the results with the following command:
python theHarvester.py -d target.com -l 50 -b securityTrails -c --screenshot /Users/user/Desktop/scsThe DNS brute-force approach wasn’t particularly useful, as we had already gathered this information from other sources. The screenshot feature, unfortunately, did not function as expected, and we encountered errors each time we tried.
This testing process highlights the versatility of theHarvester in collecting OSINT data across multiple platforms. While some sources may not yield results, the tool’s wide range of options ensures that users can gather valuable insights from the data available.
In conclusion, theHarvester proves to be a highly valuable resource for performing basic OSINT investigations. Its ability to gather intelligence passively from a wide range of sources provides a solid foundation for any cybersecurity reconnaissance effort.
While the tool excels in functionality, one area that could be improved is its documentation. Many general guides overlook specific features or advanced configurations, but in-depth analyses like ours shed light on the full potential of theharvester. Additionally, as with most open-source software, we encountered a couple of minor bugs during testing. These issues are not uncommon but do not diminish the overall usefulness of the tool.
Here are a few key points about theharvester’s impact on cybersecurity reconnaissance:
All things considered, theHarvester is a robust asset for digital security experts aiming to discover openly accessible data throughout their investigative efforts. Its robust functionality, though occasionally hindered by minor issues, makes it an essential asset in the OSINT toolkit.
Book Your Netlas Demo
Chat with our team to explore how the Netlas platform can support your security research and threat analysis.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。