A few decades ago, gathering intelligence about your target required sophisticated techniques and large budgets. Nowadays, one can achieve the same result by just having a computer and an internet connection.
In today's information age, people became more dependent on computing technology and the internet in their daily lives. For instance, people are using the internet to communicate with each other, send email messages, socialize, shop, and online banking, among other things. On the other hand, Corporations use internet technology to communicate with customers/vendors and increase their overall work efficiencies - and ultimately, their bottom line. The sum of organizations and people's interactions with the internet produce vast amounts of digital data. Tuch data can be investigated using various methods to glean useful intelligence about any subject imaginable.
Open-source intelligence (OSINT) refers to all information available publicly without breaching any copyright or data protection law. OSINT includes both online and offline resources; however, with the growth of internet and web technology, most OSINT resources are now available online.
OSINT is used extensively by adversaries; for instance, criminals and black hat hackers use OSINT techniques to gather information about their targets. On the other hand, law enforcement, other government agencies, businesses, and non-profit organizations use OSINT, legally, to support their operations.
OSINT gatherers use a plethora of tools and techniques to search online; nevertheless, there are many tools developed specifically to aid investigators in finding and analyzing information online. The latter is the real focus of this article.
The Best OSINT Software to use in Online Investigations
There are tons of tools and online services and public databases that can aid OSINT gatherers. The following section lists the most popular tools and important services of which OSINT gatherers should be aware.
Start The "Open Source Intelligence (OSINT) Fundamentals" Course Today >>
First: OSINT Links (www.OSINT.link)
The best way to begin an OSINT search is to know what tools/services are available out there for this task. OSINT.link contains a repository of links to hundreds of OSINT data sources and tools. Sources are categorized, and each group contains the relevant tools or web services.
Second: Maltego (https://www.maltego.com/products)
Maltego is a data mining software used for computer forensics and OSINT gathering. It provides a set of extensions (known as transforms using Maltego naming) to harvest information from various online repositories. It then presents the findings visually in a graph format.
Maltego focuses on finding and analyzing relationships between entities online, whether these entities are individuals, groups, documents, domains, networks, internet infrastructure, or social media affiliations.
Maltego extends its search capability by partnering with various public sources (OSINT) and commercial vendors or integrating your sources. Some data sources available with Maltego are DNS records, whois records, search engines, online social networks, Shodan, DomainTools, VirusTotal, and FireEye iSIGHT Intelligence.
Maltego uses Java to run, so it is already supported on all major operating systems such as Windows, Linux, and Mac. Keep in mind, to use Maltego; you need to register for a free account on .maltego.com.
Third: Shodan (https://www.shodan.io)
This is the first search engine used to find Internet of Things (IoT) devices. If you think Google can play the same rule of Shodan, then you are wrong. To clarify the differences between Google and Shodan, let us give this simple example: Google search is used primarily to index and find information on the visible web, such as webpages and documents. On the other hand, Shodan is used to investigate the hidden areas of the web related to IoT devices that Google can not find, such as webcams, servers, routers, printers, surveillance cameras, traffic lights, smart home appliances, vehicles, health care devices and any Internet of Things (IoT) device.
Fourth: TheHarvester (https://github.com/laramies/theHarvester)
TheHarvester is a python tool designed to find emails, subdomains, hosts, people names, open ports, and banners from various online repositories, such as Shodan, and other public sources, such as the popular search engines (e.g., Yahoo, Google, Bing).
This tool is excellent at the early stage of OSINT gathering. It is used to find initial information about the target (e.g., target email address and full name) before expanding the search.
Fifth: ExifTool (https://exiftool.org)
Used primarily in digital forensics examination, Exiftool is a great tool to investigate images for insightful information embedded in the metadata. During the OSINT gathering process, gatherers will certainly encounter image files. These files should be examined carefully for any useful information buried in their metadata.
Exiftool support finding metadata for the following image formats: EXIF, GPS, IPTC, XMP, JFIF, GeoTIFF, ICC Profile, Photoshop IRB, FlashPix, AFCP, and ID3, as well as the maker notes of popular digital cameras manufacturers.
Sixth: SpiderFoot (https://www.spiderfoot.net)
SpiderFoot is an open-source tool for automating the OSINT gathering process. It allows you to search a wide variety of parameters such as IP addresses, domain names, e-mail addresses, usernames, names, subnets, and ASNs from many sources such as AlienVault, HaveIBeenPwned, SecurityTrails, SHODAN, and more.
Seventh: Censys (https://censys.io)
Censys is a public search engine for finding information about hosts and networks that compose the internet. It aggregates its information from three main areas:
- Hosts on the Public IPv4 Address Space
- Websites in the Alexa Top Million Domains
- X.509 Certificates
Eighth: BuiltWith (https://builtwith.com)
This is a great online service to discover the type of technologies used on a particular website, such as hosting provider, email provider, CMS used to create the website, JavaScript frameworks and libraries, external fonts, web server type, and SSL certificate providers.
Ninth: Google Dorks
Google is the leader in the search engines market, although Google only indexes a small portion of the web (about 4%); however, almost no OSINT gathering task can be completed without using Google.
Google offers advanced search operators (consists of special characters and commands); when used properly, it can help the searcher dive deep and locate information that a normal Google search cannot find. The following are some Google advanced search operators:
- Site: Limit your search to a single site.
- Intext: Instruct Google that you want results where the search keywords appear in the page's body.
- Intitle: Instruct Google that you only want results where pages include the search keywords in their meta title tag.
- Inurl: Instruct Google only to return results where the search keywords are included in the URL.
- Filetype: search for specific file types.
Tenth: FOCA (Fingerprinting Organizations with Collected Archives) (https://www.elevenpaths.com/labstools/foca/index.html)
FOCA searches a particular website and scans all public documents (PDF, MS Office files) within it and extracts the metadata and other hidden info (users, folders, emails, software used, operating system, and other useful information). FOCA uses three search engines to aggregate its results (Google, Bing, and DuckDuckGo). You can also use FOCA to extract metadata from files stored on your local computer.
Summary
As we saw, there are many programs and online services for gathering intelligence from online public sources. In this article, we have mentioned only 10; however, you will certainly discover more tools that aid you in your search effort as you progress in online investigations.
