24
2012
Bulk Site URL Enumeration With GooDork
An article by connection
Yesterday I was tasked with mapping IP/Hostnames to Google Search Results. While I was initially going to use SearchDiggity I decided to use GooDork. I’m super glad I did. This post will discuss how you can use GooDork for Bulk Site URL Enumeration.
When I saw Keith Makan working on GooDork I knew I’d have to give it a try one of these days. While working on a recent assessment I had over 200 IPs/Hostnames which I wanted to run through a Google “site:” search to see what Google had indexed in order to give me some potential jump off points. Someone recommended I use SearchDiggity but it really wasn’t my cup of tea and figured that GooDork might be a better tool for the job. I’m glad I went with GooDork.
Alright let’s jump right into it.
First of all the GooDork banner, while cool, is a bit of a pain in the ass so let’s nuke it to make grepping out data from the results easier later on.
echo "" > Banner
Alright now that that is done make sure you put your IP addresses/hostnames into a file, I named mine “urls.txt”.
Execute the following command:
for i in `cat urls.txt` ; do echo -e "site: $i\n==========\n `python GooDork.py "site:$i"`"; done >> urls && sed 's/step.*//g' urls |sed 's/===================================//g' |sed 's/\[\]//g' |sed 's/Found.*seconds//g' |awk '{if(length($0) > 1) print $0}' |awk '{gsub(/site/,"\nsite");print}'
Alright let’s break this command down a bit for those that aren’t the most BASH savvy.
The first part of the command uses a for loop to grab the IPs/hostnames from the urls.txt file. Inside of the for loop we print out “site : IPaddress/Hostname” and then some equal signs to act as a separator and then pass in the results of GooDork under the line of equal signs. Once we finish with all of the IPs/hostnames in the list we throw them into the urls file and immediately start the magic. First things first we want to get rid of the output from GooDork; this include stuff like what step it is on, the equal sign separation lines GooDork adds in, some brackets GooDork has in the output and the time it took to find the results. Now we use awk to only print lines that aren’t blank and then use awk again to clean up the output by adding a newline before the site: IP/hostname line.
This leads to output like this:
site: http://hacktalk.net ========== Results: http://hacktalk.net/profile/%3Farea%3Dstatistics%3Bu%3D350 http://blog.hacktalk.net/tweets-for-2011-12-20/ http://blog.hacktalk.net/tweets-for-2012-01-08/ http://blog.hacktalk.net/xss-helper/ http://hacktalk.net/other/five-fun-facts-in-programming/ http://hacktalk.net/community-projects/ http://hacktalk.net/tutorialsguides/intro-to-defeating-aslr/ http://hacktalk.net/general-discussion-1/fun-in-the-field-my-exciting-social-engineering-tales/ http://hacktalk.net/profile/%3Fu%3D412%3Barea%3Dshowposts http://blog.hacktalk.net/tag/bitst0rm/ http://hacktalk.net/*nix-support/ http://blog.hacktalk.net/advanced-google-searching-and-google-fu/ http://hacktalk.net/phpjavascript/apache-logs-blocking/ http://hacktalk.net/tutorialsguides/hiding-files-in-ntfs-alternatedatastreams(ads)/ <truncated>
site: http://blog.hacktalk.net ========== Results: http://blog.hacktalk.net/tweets-for-2011-12-20/ http://blog.hacktalk.net/tweets-for-2012-01-08/ http://blog.hacktalk.net/xss-helper/ http://blog.hacktalk.net/honda-admits-to-data-breach/ http://blog.hacktalk.net/exploit-db-ettercap-and-more-hacked/ http://blog.hacktalk.net/page/71/ http://blog.hacktalk.net/how-private-is-private-mode/ http://blog.hacktalk.net/fbi-servers-40-search-warrants-in-anonymous-crackdown/ http://blog.hacktalk.net/tweets-for-2012-01-15/ http://blog.hacktalk.net/google-uk-street-view-wi-fi-data-deleted/ http://blog.hacktalk.net/portsmouth-city-council-rapped-for-data-breach/ http://blog.hacktalk.net/tag/bitst0rm/ <truncated>
Now you can also add >> Output.txt to the end of the line to get all this outputted to your file of choice.
I spoke with Keith about this little hack I made using his tool and he really liked the idea of it and said that he is considering implementing this functionality into GooDork and as such this post may very well become useless in the near future but I hope this post helps to illustrate that fact that using fairly basic BASH skills you can take an already awesome tool and make it do something the author of the tool hadn’t even thought of.
If you can make my little one-liner shorter feel free to leave a comment as I’m always looking for ways to get more compact scripts to improve my commandline-fu.
Keep your hacks up and your head down; cya in cyberspace.
