OSINT
Open Source Intelligence

Intro

The focus of this section is to provide helpful resources for OSINT and Passive reconnaissance on a given target. There will be certain tools and sites you might be familiar with that could be applicable in this section, that I have omitted and with good reason.
    1.
    This section is "Passive" recon, which does not entail touching or interacting with your target in any way. For offensive operations, staying off radar is key. But can we check with other sources that may have already scanned out target? You bet.
    2.
    There are many research tools that provide similar output to the ones listed in the later sections. The ones that I have specifically omitted (and will document in another section) are tailored more to defensive operations and contain information like reputation data and historical activity.

OSINT Resources

Formal OSINT Guides
Blogs/Communities
Bookmarks/Tool Collections
Training
Misc

Formal OSINT Guides

Specific for what you need to look for and how to find it, during the passive recon phase of a penetration test, or the proper way to OSINT.
    ​Pen Test Standard - Great guides for every step of a penetration test, but this section is especially useful as a reference here.
    ​Security Sift - This write up is a great guide to Passive recon when preparing for a penetration test. For building up your own workflow, start with this.
    ​IntelTechniques - One of the best resources for OSINT has been Michael Bazzell's OSINT book and his website. I highly recommend you order his book. The HTML search tools I reference here come from his collection, available on his website for free. He also runs the Privacy and Security Podcast which is a highly recommended resource for both OSINT techniques and personal privacy.

Blogs/Communities

Good for looking for what ever you need.
    ​OSINTCurious - Great community and training for those who are interested in OSINT skills and tools.
    ​OSINT Techniques Blog - Fantastic site with tool lists, video guides, and blog on the latest techniques.
    ​Osintion - OSINT and Social Engineering master Joe Grey's website. Resources, OSINT Courses, and consultation services.
    ​OSINT Dojo - A project that provides those new to OSINT a number of free resources and simple challenges that build on one another to provide a simple road map for learning more about the field and polishing up related skills while also earning badges to show off your hard work.
    ​Bellingcat's OSINT How-To - Bellingcat is a collective of researchers and journalists that use OSINT tools and techniques for a variety of purposes and that have come to gether to share thier latest and greatest tools and techniques. They have a slow of guides for researching specific things with OSINT.

Misc Resources

Search Engines

Cyber search engines are a beautiful set of tools that allow us to do research on specific targets as they relate to cybersecurity. Searching by IP, domain, vulnerability, etc. allows us to perform tasks ranging from passive recon as part of a penetration test, to threat intelligence research. Search.html - Your OSINT starting place.
When initiating a search on anything, the first stop is a bit of HTML whipped up by OSINT master Michael Bazzell. This handy tool allows an easy search of multiple sources with one click. I prefer this to other others as it allows me to manually parse the search page and go down rabbit holes from there.
Included Search Engines: Google, Google Date, Bing, Yahoo, Searx, Yandex, Baidu, Exalead, Duckduckgo, Startpage, Newsgroups, Blogs, FTP Servers, Indexes, Scholars, Patents, Qwant, News, Wayback, and Ahmia. Google is the most powerful and scary information gathering tool today. Beyond searching for basic keywords, adding in advanced commands can refine your results and reveal incredible amounts of information about your target.
These are by no means exhaustive, but they are a handy place to start.
Search.html
15KB
Text
Google
Dorking Cheatsheet
Cyber Search
Dark Web
Misc

​Google​

Google is the most powerful and scary information gathering tool today. Beyond searching for basic keywords, adding in advanced commands can refine your results and reveal incredible amounts of information about your target.
    ​Google Advanced Search - Google search with multiple special options for your search parameters
    Google search commands (Dorks)
    Google Dorking CLI Tools
      ​Goohak - Automatically launch google hacking queries against a target domain to find vulnerabilities and enumerate a target.
      ​Googd0rker - GoogD0rker is a tool for firing off google dorks against a target domain, it is purely for OSINT against a specific target domain.
    ​Keyword Tool - Tool for assisting in analyzing the efficacy of searching for certain keywords
    ​Google keyword monitor - An awesome tool that can alert you on new search hits on certain keywords.
    ​ISearchFrom - Tool that allows you to search google as if you are in different locations to analyze the differences in results.

Training

Google Dorking Cheatsheet

    @[Search term] Searches a keyword on social media
    β€œSearch term” Searches an exact match
    β€œSearch * term” Searches the * for any wildcard
    (+) (-) (β€œ) (.) (*) (|) (β€œString” | String) Force inclusion of something common Exclude a search term Use quotes around a search phrase A single-character wildcard Any word boolean β€˜ORβ€˜ Parenthesis group queries 06 cache:[url] Searches for cached versions of a site or page
    numrange[#]..[#]
    daterange:startdate-enddate Must be expressed in *Julian time (and only in integers)
      The number of days that have passed since January 1, 4713 B.C. unlike Gregorian days (those on the calendar)
    link: [url] Shows links to the URL and helps determine site relation- ships and more importantly trust relationships; this gets treated like normal search text (not a modifier) when com- bined with other search terms though.
    related: [url] Searches related to your search term
    intitle: string to search Show only those pages that have the term in their html title
    allintitle:[string] Similar to intitle, but looks for all the specified terms in the title
    inurl: [string] Searches for the specified term in the url; for example inurl:”login.php”. (Can also do :port)
    allinurl:[url] Same as inurl, but searches for all terms in the url
    intext:β€œString to search” Searches the content of the page and similar to a plain Google search; for example intext:”index of /”.
    allintext: β€œString to search” Similar to intext, but searches for all terms to be present in the text 07 filetype: [xls] Searches for specific file types; filetype:pdf will looks for pdf files in websites.
    phonebook:[name]
    [URL]&strip=1 Added to the end of a cached URL only shows Google’s text, not the target’s; perform a Google search, right-click copy/ paste the link and then paste the URL adding &strip=1
    site.com/search?q=inurl:admin.PhP&start=10 Changing your query to vary the extension case and modifying the query can help defeat some of Google’s blockers which work to defeat your search query
    site.com/[email protected] Searching for email addresses
    site:site.com -site:obivousresult.com Eliminates obvious results, reducing most public, top β€˜ranked’ unwanted results and bringing more useful results to the top of the search; you are looking for the relation- ship of links in both inbound and outbound directions
    inurl: Port scanning, can be combined with the site operator
    inurl:8080 -intext:8080 Servers listening on port 8080 removing results with 8080 in the page
    filetype:inc intext:mysql_connect filetype:sql + β€œIDENTIFIED BY” -cvs Search combinations that goes after files with cleartext SQL passwords and credentials
    intitle:”VNC viewer” Example of a search for sites that launch a VNC client

Cyber Search Engines

    ​Shodan - Shodan is often called the "Hacker's Search Engine". Shodan has servers scanning the entire internet for devices. Once it finds them, it maps their ports and collects other useful information. Shodan has advanced search commands similar to google dorks. Shodan also has a flexible API that can be leveraged into many other tools.
    ​Spyse - The Internet Asset Search Engine. Spyse has some advanced scanning features that can allow you to discover seen vulnerabilities by CVE score or subdomains, on top of detailed metadata on your target. With its heavy toolset and API functionality, it is a popular choice for automated enrichment.
    ​Maltiverse - A search engine for threat based indicators. Maltiverse also has multiple threat feeds you can ingest into your intel platform for alerting.
    ​Onyphe - A Cyber Defense Search Engine for open-source and cyber threat intelligence data collected by crawling various sources available on the Internet or by listening to Internet background noise. ONYPHE does correlate this information with data gathered by performing active Internet scanning for connected devices and also by crawling Web site URLs.
    ​IntellX - Search Engine that allows searching with selectors, i.e. specific search terms such as email addresses, domains, URLs, IPs, CIDRs, Bitcoin addresses, IPFS hashes, etc. It searches in places such as the darknet, document sharing platforms, whois data, public data leaks and others. It also keeps a historical data archive of results, similar to how the Wayback Machine from archive.org stores historical copies of websites.
Searching the Dark/Deep web is a great intelligence activity that can yield a multitude of different treasures. This is a great resource for spotting early indicators of a breach or getting the latest trends in cyber crime.
*Darknet and .onion sites change frequently. This list may not be up to date.

Historical/Cached pages

Sometimes the page you are trying to find is no longer available. But it still may exist in web archives or cached data. Be sure to check these when you are getting stuck.
    ​OnRender Archive/Cache Search - Amazing search tool that checks multiple search engine caches for the page you are looking for as well as a few web archive tools.
    ​Wayback Machine - The gold standard web archive. if you are looking for a version of a web page at a specific place and time, check this!

Misc.

This section contains miscellaneous search engines and utilities.

IP, Domain, Username, and Email Address

IP address
Domains
Username/EmailAddresses

IP Address

When researching IP addresses, it is important to know the context of the search you are performing. There are a multitude of sources to research IP addresses and they can vary depending on what information you want to learn about them. For offensive security, threat hunting, and attack surface mapping, we want current registration data, and any associated data points that our searches may return. These can include hosted domains, ASN, associated network artifacts, etc.
For defensive operations, such as those of the security blue team, you would be looking for historical data and activity data of the IP address. These tools will be detailed in another section about threat research. The below links and tools are specifically for offensive intelligence gathering and reconnaissance.
IP.html
IP.html is another handy little tool created by Michael Bazzel that makes initial research of an IP address quite easy. This tool will populate multiple searches automatically for you to see what information you can gather about your target.
Sites Include: Bing, Reverse IP, Locate IP, Port Scan, IP Whois, TraceRoute, Who.IS IP, Cynsys, ThreatCrowd, Shodan, ZoomEye, Torrents, "That's Them", WeLeakInfo, Dehashed, and UltraTools IP
IP.html
10KB
Text
Whois Vs. RDAP
Whois is a great tool for gathering registration data for IP addresses and domains. The only problem with it is that there is not a clearly defined structure to organize registration data points and keep them maintained. Enter RDAP. A new Standard as of 2019, RDAP lookups will quickly replace WHOIS lookups.
Is this a Tor Node?
Maybe? Check it with this! https://metrics.torproject.org/exonerator.html​
Torrent IP addresses - https://iknowwhatyoudownload.com​
IP Location Info
There are a several ways to find geolocation of a user: HTML5 API, Cell Signal and IP Address to name a few. If you have an IP Address and want to find the geolocation data for the target, the below sites use various methods to determine that data.
*Note: It is recommended that you use as many tools as possible for a consensus determination on the location. Some times results will show the location of the registrant, but not the location of the IP in use.

Domains

Domains, more than almost any other target, have one of the largest assortments of associated data points. The most important that we will look for out of this section is the Registration data, the hosting data, site information, archived data, and analytics.
Domain.html
Domain.html is a tool that allows us to research multiple data points associated with a domain that might be handy during an investigation.
    Registration Data - This tool will check the domain for whois based registration data against multiple sources to get the most up to date data.
    Hosting Data - This is the information that shows which provider is physically hosting the domain. Be sure to look for indicators if the target domain is hosted by a hosting provider, or self hosted by your target.
    Exposed Data - Any information that may be exposed to the public. (Other sources are better)
    Archive Data - When researching a domain, sometimes you can find older cached or saved versions of the website that may yield valuable information. These include Google Cache, Archive.is, and the WayBack Machine.
    Analytics data - This is a grab bag of handy searches, ranging from general site details and analytics, to similar sites on the web, or checks for backlinks to from other sites.
    Threat Data - Discussed under the Blue - Threat Data section
    Shortened URL metadata.
Domain.html
28KB
Text
Domain Toolboxes
These next few tools are collections of utilities focused around domains. Some can be used for research on other network artifacts like IP addresses and email records, but DNS records and domain related metadata is really where they shine.
    ​ViewDNS - Huge toolbox with various utilities for enumerating information about a domain.
    ​DNSDumpster - Free domain research tool that can discover hosts related to a domain.
    ​MXToolbox - Checks MX information for the given domain
    ​W3DT - W3dt.Net is an online network troubleshooting site dedicated to providing relevant real-time data regarding networks, websites and other technical resources.
    ​DNSLytics - Find out everything about a domain name, IP address or provider. Discover relations between them and see historical data. Use it for your digital investigation, fraud prevention or brand protection.
Whois Vs. RDAP
Whois is a great tool for gathering registration data for IP addresses and domains. The only problem with it is that there is not a clearly defined structure to organize registration data points and keep them maintained. Enter RDAP. A new Standard as of 2019, RDAP lookups will quickly replace WHOIS lookups.
Sub-domains
There are tons of highly effective tools for subdomain enumeration and brute forcing, but they can be quite noisy. During the Passive Recon phase of a penetration test, we can start with any subdomains recorded by other sources to plan out our attack/test.
Domain Certificates
Domain Certificates are an interesting and useful item to research when mapping out a target domain. Beyond the various attacks that can be performed by exploiting these certificates, looking up the domain certificates can lead to discovery of hosts, sub-domains, and related targets that were previously undiscovered.
Web Site Change Tracking
Some times a target will change a website and you will want to be notified right away, usually to see what has changed and how you can exploit it.
    ​Follow that page - Follow That Page is a change detection and notification service that sends you an email when your favorite web pages have changed.
    ​Visual Ping - Tool that can track multiple different kinds of changes in a particular webpage and alert on specific conditions.
Misc. Utilities
    ​DNPedia - Domain Name Solutions, Statistics, Scripts, News and Tools
    ​Google's Online Dig command - Online version of the Dig command
    ​SimilarWeb Traffic Analytics - Compare meta data about domains and traffic to other elements on the web
    ​Backlink Checker - Tool to easily monitor backlinks for a particular domain.
    ​DomLink - DomLink is a tool that uses a domain name to discover organization name and associated e-mail address to then find further associated domains.
    ​https://dfir.blog/unfurl/ - Easily breakdown and visualize the elements of a URL link.

Usernames and Email Addresses

Corporate usernames are beginning to be obnoxiously easy to guess and build. The standard of [email protected] is so common, it's ridiculous. Even more so when account management tools will simply take the first half of the email and reuse it as a username. We can use schemes like this to our advantage to search for a multitude of treasures like accounts on other services with the same username, credentials found in breaches, and associated sites or tools. When searching for usernames, you can uncover linked social media accounts and tons of relevant intelligence.
Username.html and Email.html
These two tools often go hand in hand with results often overlapping. Still, it is good habit to run the searches for both the username and the email address in case there is a discrepancy between the two. These two tools check for two things: presence of the username/email on a given platform, and any public/leaked info connected to them.
Email.html
13KB
Text
Username.html
19KB
Text

Username Search Tools

    ​https://usersearch.org/ - Search Engine for Usernames
    ​Lullar search - Search tool specifically for names
    ​Name Check - See if a username is available across multiple platforms
    ​Sherlock - Hunt down social media accounts by username across social networks​
    ​Stalker - OSINT tool for automated scanning of social networks and other websites, using a single nickname.

Email Address Search Tools

    ​Public Mail Records - Search public email records for a given email address.
    ​MXToolBox - Collection of online tools that can gather multiple points of data surrounding an email address or domain.
    ​Email Format - Find the email address format for a given company or domain.

Email Verification

Some times it helps to perform a quick check to see if an email is even valid or registered.
    ​Tru Mail - Prevent bounced emails and low-quality users with free professional grade email verification
    ​Email Hippo - Email address verification technology from Email Hippo that connects to mailboxes and checks whether an email address exists.
    ​Verify email - This email verification tool actually connects to the mail server and checks whether the mailbox exists or not.
    ​Email Checker - Email Checker ensures that an email address is correct and active in real-time without ever needing to send a message.

Email CLI Tools

    ​TheHarvester - This tool is the defacto standard for email intelligence gathering. It checks a large array of sources to pull together information. It can leverage APIs of other services such as Spyse or Shodan to improve the search. Remember these will require an API key to use. I have found that between the above html tools and this, it will satisfy your email searching needs.
    ​Infoga - Infoga is a tool gathering email accounts informations (ip,hostname,country,...) from different public source (search engines, pgp key servers and shodan) and check if emails was leaked using haveibeenpwned.com API.
    ​Match Email to Phone number - email2phonenumber is an OSINT tool that allows you to obtain a target's phone number just by having his email address.
    ​GHunt - Google account info scraper -

People Search/PII

This section will get stalker-ish real quick. While limited in usefulness for a penetration test, it can help you discover all the interesting data surrounding a person, or link data you have found back to an individual.
Name Search
Social Media
Telephone
Business/Gov Records
Name search records can get muddy really quickly when dealing with names like "John Smith", however many tools will allow you refine the search with other data points such as location or other context. As with any search tool, the more data you feed it, the more accurate your results will be. As always I like to start with one of Michael Bazzell's handy tools, Name.html. This tool will search databases of names that will return associated data points for building a full profile of your target. From here, the next step is to start looking for any public records that may be associated with the name you are searching for. Remember to use location to aid in context.
    ​Xlek - Searches millions of online records for a given name.
    ​Thats Them - Find all sorts of information about a person including address, email, even their cars VIN number
    ​Public Records - Search public government records for entries relating to your target.
    ​Peekyou - Popular people search engine
    ​BeenVerified - People search engine that can return people, vehicle, property and contact info.Volunteer OSINT
Name.html
16KB
Text

Social Media

Facebook.html
44KB
Text
Instagram.html
9KB
Text
LinkedIn.html
5KB
Text
Twitter.html
23KB
Text
Communities.html
21KB
Text

Telephone Numbers

    ​Carrier lookup - Enter a phone number and returns the carrier name and whether the number is wireless or landline.
    ​Number Validator - Search phone number format and origin
    ​CallerID check - A database of caller names used to identify the name of a caller when receiving an inbound call from the United States or Canada.
    ​TrueCaller Caller ID check - One of the best CallerID utilities
    ​Spy Dialer - Reverse phone number lookup for cell phones, VOIP and landlines.olunteer OSINT
Telephone.html
37KB
Text

Government and Business Records

The US government loves paperwork, and thanks to many initiatives like the Freedom of Information Act, so much of what they gather is available for you to parse through. Business.html searches through multiple sources like public records, voter record, and court documents.
    ​Public Records - Search public government records for entries relating to your target.
    ​Background checks - Search for mentions of a person in court cases, contact information, assets, police records and much more!
    ​SEC filings - All companies, foreign and domestic, are required to file registration statements, periodic reports, and other forms electronically through EDGAR. Anyone can access and download this information for free. Here you'll find links to a complete list of filings available through EDGAR and instructions for searching the EDGAR database.
Business.html
8KB
Text

Files, Media, Breach Data, and Code Repositories

Files/Documents
Images/Video
Breach Data
Code Repositories

Files and Documents

Files and Media are one of the more juicy targets to look for when planning a penetration test. For companies that publish things to the web on a regular basis, there is constantly information that is overlooked and should not have been sent out of the organization. I have found things like email distribution lists, Internal only email addresses perfect for phishing, personnel information, client communications, etc. Dont forget public facing FTP servers. They always seem to have something juicy hidden in them.
Documents.html
Documents.html is a tool that allows you to take a search term related to your target, and search for various file types associated with the term. The term should be something as unique as possible, but still related to the target: company name, platform, application, client, etc. Perform multiple searches for various terms for the best coverage.
Documents.html
14KB
Text

File/Document Search tools

    ​Napalm FTP Indexer - Search for documents in public FTP servers
    ​MMNT - Russian FTP indexer
    ​GreyhatWarfare Public AWS Buckets - Search for publically acessible AWS S3 buckets.
    ​MS Azure Portal - Search for public blobs -
    ​PowerMeta - PowerMeta searches for publicly available files hosted on various websites for a particular domain by using specially crafted Google, and Bing searches. It then allows for the download of those files from the target domain. After retrieving the files, the metadata associated with them can be analyzed by PowerMeta. Some interesting things commonly found in metadata are usernames, domains, software titles, and computer names.

Images/Videos

While having other functionality Michael Bazzell's Images.html and Videos.html tool helps search for image terms across multiple platforms. Looking for faces of employees? Maybe a picture of thier security badge you can copy? Image of a target you can extract metadata from later? Start with a good image search. Google is hard to beat for this but there are other platforms that can lead to some interesting discoveries.
*Note - this tool is only to find images associated with a search term. If you have an image and you would like to find out more information about it, that will be discussed under the Forensics section.

Tools and Resources

Images.html
11KB
Text
Videos.html
17KB
Text

Breach/Leak data

Looking for easy creds? Linked data? Password hash? Breaches can be a trove for low hanging fruit for those targeting those not diligent with their cyber hygiene. Often times, the credentials found in large data breaches will turn into password lists such as the infamous rockyou.txt password list that came from a sizeable breach in 2009.
The below tools and links can be used to parse data in known data breaches and leaks, or be used for detection and alert for the presence of credentials when new breach data is reported. Paste sites like Pastebin have recently changed their ability to be parsed. Pastebin itself has removed the ability to to search its pastes. However, with a bit of clever google dorking, you can still search for breach data by submitting your search along with "insite:pastebin.com"
Breaches.html
15KB
Text

Tools and Resources

Code Repositories

Ah the gold mine of git repositories. So at the time of writing this, we are still in the golden age of security ignorance in coding. DevSecOps has not yet fully caught on, and software engineers everywhere post up this tid-bits of insecure code for storage later, or post a bit if their config file on a forum asking for help. Little did they realize that in that bit of the config file, they accidentally posted their creds! These are a few examples of the fun things we can find when checking code repositories. Now searching for these is usually limited to the context of a penetration test against an organization where you know they have software engineers bust creating the next great thing.
There are many great options out there for code repositories, but there are 4 that are the gold standard for checking.
You can manually parse these by user or subject but there are some handy tools that can help search and keep track.

Misc. OSINT

Bot Hunting
Automotive
Cryptocurrency
Location
Utility

Bot Hunting

    ​https://botsentinel.com/ - Bot Sentinel is a free non-partisan platform developed to classify and track inauthentic accounts and toxic trolls.
    ​https://botometer.osome.iu.edu/ - Botometer (formerly BotOrNot) checks the activity of a Twitter account and gives it a score. Higher scores mean more bot-like activity.
    ​https://hoaxy.osome.iu.edu/ - Visualize the spread of a tweet to determine if it is artifically propagated.
    ​https://csmr.umich.edu/projects/iffy-quotient/ - The Iffy Quotient is a metric for how much content from β€œIffy” sites has been amplified on Facebook and Twitter.
    ​https://www.io-archive.org/#/ - The Information Operation Archive hosts publicly available and rigorously attributed datapoints from known Information Operations on social media platforms.
    ​http://twittertrails.com/ - A tool that allows members of the media to track the trustworthiness of stories shared on Twitter

Cryptocurency

Currencies.html
13KB
Text

Location

Location.html
24KB
Text

Misc Utility

    ​Hunchly - Hunchly is a an interesting tool that passively captures the web pages as you browse. This can be handy when you are parsing through a large volume of pages and you want to keep track of them, or if you are concerned a page may change after you visit.
    ​Carbon14 - This is an OSINT tool for estimating when a web page was written. Common CMS's easily permit to change the displayed date of content, affecting both websites and RSS feeds. Moreover, the dynamic nature of most web pages does not allow investigators to use the Last-Modified HTTP header. However, most users do not alter the timestamps of static resources that are uploaded while writing articles. The Last-Modified header of linked images can be leveraged to estimate the time period spent by the writer while preparing a blog post. This period can be compared to what the CMS shows in order to detect notable differences.
    ​Wigle - A website for war driving. Maps of wireless networks and their locations.
    ​JupyterPen - For those who love Jupyter notebooks, this is a project that started as an OSINT framework built around JupyterNotebooks and has expanded into a full penetration testing tool.
    ​CardPwn - OSINT Tool to find Breached Credit Cards Information

Misc Techniques

How to check a short link instead of being redirected:
    bit.ly - add + at the end
    cutt.ly - add @
    tiny.cc - add =
    tunyurl.com - add "preview." to the beginning of the url.
​

Volunteer OSINT

There are a few interesting organizations out there that take OSINT researchers and have them help with certain public good tasks like finding missing children or stopping pedophiles. It can be a heavy ask but can really do some good in the world with the skills that we have. Please check out https://www.tracelabs.org/ and https://www.innocentlivesfoundation.org/ If you can donate some time to help, please do!
​
Last modified 1d ago