Category Archives: Tool

How to Extract links from a Webpage

Turn this…


into a neat file with clean links…


Things you need:

  1. A web browser
  2. A webpage from which to extract links
  3. An online tool to automatically extract the links
  4. A program to open .csv files (Excel/Google Sheets/free alternative, etc.)

Hint: If you only need links from a portion of the page use Firefox.  It allows you select a portion of the page and view only the “selection source” instead of the entire page.


  • Navigate to the desired webpage in your favorite browser
  • View and copy the page source
    • Chrome: “View page source”
    • Firefox: “View Page Source” or “View Selection Source”
  • Paste the copied html into an online tool and click the button to generate a CSV file with the resulting links
  • Download, open, & enjoy!

Once the file is open you can do more other things in Excel like sorting by Domain or removing duplicate links.

Online Tool for Extracting Links from webpages:

How To Check If Your Site is Blacklisted by Google

Occasionally, you should verify that your site is still appearing as it should in search engine results (especially if you don’t use managed hosting or another site scanning service).

Getting de-listed from Google or any other search engine could be devastating.

In the search bar type in where you replace with your domain name.  Example:

Your main domain should be in the top search results along with other popular or top-level pages.

Going directly to your website ie might still appear normally as some malware redirects only affect search engine referral traffic.

Signs that your sight might be hacked, serving SPAM, or de-indexed

What to watch for:

  1. Significant drops in traffic per analytics reports
  2. A warning in search engine results stating that “This site might be hacked”
  3. Long connection times in FTP access
  4. Large backups or backups that fail to complete
  5. Security alerts from Google Search Console (formerly Webmaster Tools)

Other blacklist sites to check:


Other search engines to check:

  1. Bing/MSN:
  2. DuckDuckGo:
  3. Yahoo:
  4. AOL:

Inspired by:

Troy Hunt’s ultimate list of security links

Below is the list of security related links Troy Hunt posted on September 26, 2015.

Also included are links posted by his readers in the Comments section.

Troy Hunt’s ultimate list of security links


  1. Is TLS fast yet – A great site debunking the myths of SSL/TLS speed cost
  2. Firesheep – A watershed moment for SSL by demonstrating the ease with which unprotected traffic can be intercepted and sessions hijacked
  3. Qualys SSL Labs – Tests a variety of attributes of the SSL implementation by pointing it at any URL
  4. CloudFlare – Get SSL for free on any website
  5. Let’s Encrypt – It’s coming, and it promises to fix the current mess that is CAs and configuring certs
  6. Betsy’s free wifi – Shows a young girl standing up a rogue wifi hot spot
  7. Chromium HSTS preload list – All the sites submitted for HTTP strict transport security preload (a depressingly small number of them)
  8. HTTP Shaming – Sensitive data sent insecurely? Name and shame!


  1. Krista’s professional DDoS service – Video of an innocent teenager promoting a DDoS service
  2. Norse – Totally awesome real time map of DDoS attacks that’s absolutely mesmerising to watch
  3. Booter promotional video – Very professional advert for a “booter” service (complete with “Epic DDoS interface”)
  4. – Example of a DDoS service… protected by CloudFlare… the world’s largest provider of DDoS defences…

SQL injection

  1. sqlmapThe tool for mounting SQL injection attacks tests against a running site
  2. Drupal 7 SQL injection flaw of 2014 – great example of how impactful it still is (patch it within 7 hours or you’re owned)
  3. Ethical Hacking: SQL Injection – If you really want to go deep, here’s five and a half hours worth of Pluralsight content


  1. XSSposed – List of sites found to be vulnerable to XSS (including attack vector)
  2. Dutch banks doing the Harlem Shake – Video collage of a number of Dutch banks with XSS risks being made to do the Harlem Shake via a script reflected from the URL
  3. XSS Filter Evasion Cheat Sheet – Because XSS payload filtering is almost always insufficient
  4. </xssed> – Heaps of XSS news and lists of vulnerabilities

Security scanners

  1. NetSparker – My favourite dynamic analysis tool due to ease of use and practicality (especially good for developers who may not live in security land)
  2. OWASP Zed Attack Proxy (ZAP) – Great tool for dynamic analysis security testing and ha a whole raft of other users too (oh – and it’s free!)
  3. Burp Suite – Seriously powerful with a heap of different tools and a freebie version to get you started
  4. Fiddler – Not a security tool per se, but I use it extensively to inspect website behaviour, tamper with requests and modify responses on the wire
  5. Acunetix – Popular dynamic analysis tool similar to NetSparker but is let down a bit in the usability stakes IMHO
  6. Nikto2 – Freebie open source app scanner sponsored by NetSparker

Exploit databases and breach coverage

  1. – Heaps of exploits consolidated from various bug tracking lists
  2. Exploit DatabaseVery comprehensive list of vulnerabilities
  3. PunkSPIDER – Lots of vulnerabilities of all kinds all over the web (about 90M sites scanned with over 3M vulns at present)
  4. Data Loss DB – Good list of breaches including stats on number of records compromised
  5. Information is Beautiful: World’s Biggest Data Breaches – Fantastic visualisation of incidents that give a great indication of scale

Cracking software

  1. HashcatThe tools for cracking hashed passwords; totally free with a great supportive community
  2. John the Ripper – Also top notch password cracking software with some different approaches to Hashcat
  3. RainbowCrack – Rainbow tables are becoming less relevant in the era of fast GPUs and tools like Hashcat, but it’s worth a mention anyway
  4. Aircrack-ng – For all your 802.11 WEP and WPA-PSK key cracking needs

Hacking and penetration testing tools

  1. Metasploit – The canonical pen testing tool; seriously advanced and enormously powerful
  2. BeEF – The Browser Exploitation Framework offering remote control over a target’s browsing session
  3. Kali Linux – All your pen testing bits in one image!
  4. Backtrack-linux – Fallen out of favour a bit as Kali has emerged, but still deserves a mention
  5. Nmap – For all your mapping of network things needs
  6. Wireshark – When you need to down to monitoring at the packet level

Vulnerability definitions

  1. The OWASP Top 10 Web Application Security Risks – The canonical categorisation of the top risks on the web today
  2. SANS 20 Critical Security Controls – Great consolidation of security controls presented in an easily consumable fashion

Security headers

  1. Fiddler extension for CSP – Massively streamlines your creation of a CSP by building the policy as you browse
  2. – Everything security header related and a great place to assess your current state
  3. Report URI – Analyse your CSP and HPKP headers plus log your exception reports there
  4. Make any website do the Harlem Shake – if you can run this in the console against a website, they almost certainly don’t have a CSP prohibiting arbitrary content from being loaded into the site


  1. OWASP Password Storage Cheat Sheet – There are plenty of bad ways of doing it, this is a great resource documenting the good ways
  2. Jimmy Kimmel “What is your password” – video of interviewing people and engineering them into disclosing their password
  3. Diceware – A popular method of creating strong pass phrases suitable for use as a password

Password managers

  1. 1Password – Still my favourite password manager; client based, runs on all devices and the keychain is syncable via multiple mechanisms
  2. LastPass – A web based password manager (albeit with rich clients as well), one of the big players in password managers
  3. KeePass – A popular free alternative to commercial password managers

Account management

  1. Adult Friend Finder password reset – Enumeration done wrong; initiate a password reset for any email address and be told if they’re a member of a highly personal site
  2. Entropay password reset – A great example of not disclosing the existence of an account (try resetting an account that isn’t registered on their system)
  3. Botnet brute force attack against GitHub – I regularly use this as an example of how hard it can be to defend against brute force

Personal security

  1. F-Secure’s Freedome – My VPN of choice with lots of exit nodes around the world and a promise of no logging
  2. – This is an Aussie version so do find one local to you if you’re not down under, but identity protection and credit alerts is a “must have” today IMHO


  1. Google Hacking Database – Great collection of Googledorks categorised by various classes of expose data
  2. Google Hacking for Penetration Testers – In case you prefer books over web pages

Other tools and links

  1. Have I been pwned? – How could I not include this?! My own tool, now being put to particularly good use by large enterprises monitoring tens of millions of people
  2. Mailinator – create temporary email addresses for testing
  3. Shodan – Find devices connected to the web (cameras, SCADA systems, etc.)
  4. Reitre.js – “What you require you must also retire”: Helps identify JavaScript libraries with known vulnerabilities
  5. – Analyses web-delivered malware by inspecting an individual URL and identifying malicious behaviour
  6. Phish5 – I’m yet to use them but I hear good things; phishing attacks are enormously effective and these guys help you test your organisation for how well equipped people are to recognise the attacks
  7. Plain Text Offenders – Been emailed your password? Name and shame!
  8. Kaspersky Real Time Threat Map – Very cool visualisation of the real time threat Kaspersky is seeing
  9. Tor Browser Bundle – Access the underwebs and browse anonymously

Security statistics reports

  1. Verizon Data Breach Investigations Report – The annual DBIR is based on real world security incidents and is a great resource for evidence-based security metrics
  2. WhiteHat Security Statistics Report – Based on findings in the websites they monitor with their security products so another good evidence-based report
  3. Trustwave Global Security Report – Another annual report driven from real world investigations (plus they use the terms “threat intelligence”, “seedy criminal underground” and “data defender” so you know it’ll be good!)
  4. Websence Threat Report – Created by Websense Security Labs, a fairly high level overview of the threat landscape
  5. HP Cyber Risk Report – More cyber, more statistics, more reports

Noteworthy books

  1. We are Anonymous – Still one of my favourite security books, a look inside Lulzsec and how it all unravelled
  2. Ghost in the Wires – The story of Kevin Mitnick’s early days and an absolutely fascinating read
  3. Data and Goliath – Just because you’re paranoid doesn’t mean they’re not after you! Excellent read on data collection by Bruce Schneier

Other things you should be reading

  1. What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text – Because encoding is one of those things you just need to know

Awesome people you want to read and follow

  1. Mikko Hypponen
  2. Brian Krebs
  3. Jeremiah Grossman
  4. Scott Helme
  5. Bruce Schneier
  6. Kevin Mitnick
  7. Swift on Security
  8. Brian Honan
  9. Graham Cluley
  10. Rob Graham

Links from the Comments (with some shameless and not shameless plugs)

  1. / @troyhunt
  2. (Crypto relatively easy explained)
  3. (Also crypto, by Dan Bernstein)
  4. (Detailed “How-to”s to a wealth of security stuff, from DNS over Hash extension to ROP)
  5. (Especially the txt documents – that’s where a lot of the kernel security today comes from – ASLR anyone?)
  6. (Twitter of one of the grsecurity devs, most interesting for Linux users)
  7. (Google’s Project Zero)
  8. <- big list of “recommended Reading”
  10. dont panic fist aid kit (more for devops, but some links could fit in here too)
  11. “ASP.NET Security Secrets Revealed” course
  13. Also open source training:
  14. Edgescan stats report:
  17. IBM X-Force Exchange Threat Intel Platform –
  18. IBM X-Force Threat Intelligence Quarterly –
  19. Book – The Cuckoos Egg by Cliff Stoll
  22. Password Manager –
  31. Book –

Original content courtesy of Troy Hunt licensed under a Creative Commons Attribution 3.0 Unported License.

Work on this page is also licensed under a Creative Commons Attribution 3.0 Unported License.

Tools for internet archivalists

The internet is a very dynamic place.  Sites, images, and resources are always coming and going.  Controversial content is purposely removed by originators or site hosts.  Site redesigns more often than not break links to images or navigational elements.

A good place to start:

The Internet Archive has a tool called the Wayback Machine that takes snapshots of webpages as they appear when the page was saved.  By now many people are aware of it’s existence, but I would bet that few people know you can manually innate a  capture with their “Save Page Now” feature.  Only caveat is that it is “only available for sites that allow crawlers.”



Wayback Machine – See how a page appeared when a snapshot was taken.

Wayback Machine “Save Page Now” – Capture a web page as it appears now for use as a trusted citation in the future.

DownThemAll! – add-on for Mozilla Firefox that “lets you download all the links or images contained in a webpage and much more” – an online tool that saves an entire webpage (better than stitching together multiple screenshots). Save in JPEG, PDF, TIFF, BMP, PNG, PS, or SVG formats.