How to build a spider… uh, well an email scraper

What is a spider? A spider is a program that automatically navigates web pages to find informations. These informations can…

November 26, 2013

What is a spider?

A spider is a program that automatically navigates web pages to find informations. These informations can be of any kind, Google, for example has a spider named Googlebot, which is the most famous and ubiquitous spider ever existed, it scans pages and copy contents to Google servers to let your pages be found on Google search engine. After copying pages, other programs build indexes, analyze your contents and determine page rank for each keyword of your page. Search engines, like Google, Bing, Yahoo! can exist only because there are spiders.
Spider are good. Bad can be evil… and Googlebot reminds me to Agent Smith, do you remember Agent Smith in Matrix? :)

So, in spiders developing there are often two steps:
1) collect data from urls
2) analyze and use data

The second step could go outside the scope of a spider if it is too complex.

There are different names for spiders. They are called also web crawlers, scrapers, bots, harvesters. Programs that go on the web are called web bots (or simply bots) when perform small tasks, whereas when they collect a lot of data and parse thousands of pages and make complex things they are called spiders.

Building a spider or a web bot

Building a spider/bot means thinking at web in a different way, it’s not only a place where you can read articles, log in to your starred social network or buy something on a e-shop. It’s a place where you can use programs to make things.
For example, you can set up a spider to check if the page of your preferred artist changes and send you a push notification on your device if it happens. You can use spiders to track views of your competitors and build a rank to monitor them.
You can build a spider that logs in on a site and makes something as a normal user. There are a lot of possibilities.
A spider can also use APIs to find data and perform tasks inside third parties sites.

How to build an email scraper

A very simple spider to build is an email scraper. Which, if used to send spam, becomes a Spambot, wich is a BAD THING. As the name say an email scraper is a program that crawl the links of an entire site, and find emails.
The emails are than stored in a database and can after be used to contact users with (maybe unsolicited) emails.

I really don’t like spam and spammers, but building a part of a Spambot (the part that find emails) for didactic purposes can show you how simple is to take emails out from a site, and you can easily understand why there is so much spam in our inboxes today.

So, I will show the code of an email scraper which collects emails from a web site.

You will also understand, as a developer why it is so important to never show emails clearly in web pages.
In the next paragraphs I will explain the code of a simple mail scraper that is made using my Mini Bots PHP Class.

Be polite and keep the control of your spider

Since every spider needs time to run, it’s really important to have control on it. This means that you know what you’re doing and that you can cause problems to servers and developers behind those servers. So you have to control your code, and be able to stop your spider and limit it.

This because if you send a lot of http calls to a web site, you can overload it and “kill” it.

The simplest way I’ve found to run a spider in PHP completely under my control is to let it work by chunks inside a web browser. Step by step means that it makes a job that lasts a small number of seconds, than stops and waits an amount of time, than it restarts again and makes another little chunk of work. And so on…
This way respects also the rule of politeness, since you will make a small number of calls to the target server, not to close. This rule is very important and is a policy that developers must follow.

The crawler starts from a “seed“, that is to say a beginning URL, and finds all the links in that page and store them in a database. Then it searches for emails in that page and, if there are any, proceed to store them in the database. It’s very easy to find emails since you need just one regular expression to extract them from html. At the end of this process the crawler marks the analyzed page as read and the spider stops.

Since this mail scraper was thought to run in a browser, at the end of the process it waits for a random interval of time and then reload throught javascript.
When it restarts it choose another seed from the previous saved links and not yet marked a read.
This process lasts until all the urls in the database are read.
With a script like this you can scrape an entire site and find emails.


Here is a very simple database for our email scraper:

  `id` int(10) unsigned NOT NULL auto_increment,
  `saved` datetime NOT NULL,
  `url` varchar(255) NOT NULL,
  `visited` tinyint(3) unsigned NOT NULL default '0' COMMENT '0=not visited,1=visited',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `url` (`url`)

  `cd_spider` int(10) unsigned NOT NULL,
  `email` varchar(150) NOT NULL,
  UNIQUE KEY `email` (`email`)

As you can see there are only two tables, one to store the links found and mark them visited or not.
And then the emails table which collects the found data.
Here is the code, remember that to let it work you need my Mini Bots PHP Class, available for just 7$ on CodeCanyon.

PHP source code

This code uses these methods of the Mini Bots PHP Class:

To grab a page from a url, masking your bot as a browser

To grab the list of links in a page, fix absolute paths and remove duplicates and filter internal links.

To extract all the emails in a web page

So, build the tables in your database, copy the code and configure it with your database and your $SEED and go.
You can close the browser and reopen it, or stop the automatic refresh of the scraper by pressing ESC.
To restart it just reload the page.

header('Content-type: text/html; charset=utf-8');
ini_set('default_charset', 'UTF-8');
setlocale(LC_CTYPE, 'it_IT.UTF-8');

define("WEBDOMAIN", "localhost");
define("DEFDBNAME", "dbname");
define("DEFUSERNAME", "username");
define("DEFDBPWD",  "password");
if(!Connect()) {die("err db");} else {mysql_query("SET NAMES 'utf8';");}

// this is the seed, the starting
// page.
$SEED = "";

// includes the Mini Bots PHP Class
$mb = new Minibots();

// this is the main function
// that does everyhting
function goSpiderGo($target="") {
	global $mb;
	$f = (integer)execute_scalar($sql = "select count(*) from spider where url='".addslashes($target)."' and visited=1");
	if($f>0) {
		$target = execute_scalar("select url from spider where visited=0 order by saved limit 0,1");
	if(!$target) {
		return 0;

	echo "target: <code>".$target."</code><br>";
	$web_page = $mb->getPage($target);
	echo "length: <code>".strlen($web_page)." bytes</code><br>";
	$links = $mb->findLinks($target, $web_page, false, "pdf,zip,jpg,gif,png");

	$countUrls0 = execute_scalar("select count(*) from spider");
	foreach($links as $link){saveUrl($link,0);}
	$countUrls1 = execute_scalar("select count(*) from spider");
	$countUrlsV1 = execute_scalar("select count(*) from spider where visited=1");

	echo "Urls added: <code>".($countUrls1-$countUrls0)."</code><br>";
	echo "Urls Totali: <code>".($countUrls1)."</code><br>";
	if($countUrls1>0) echo "Urls scraped: <code>".($countUrlsV1)." (".round($countUrlsV1/$countUrls1*100)."%)</code><br>";

	$spider = saveUrl($target,1);
	echo "record scraped: <code>".$spider."</code><br>";

	$countEmails0 = execute_scalar("select count(*) from emails");
	$emails = $mb->findEmails($web_page);

	foreach($emails as $email){
		mysql_query($sql = "insert ignore into emails (cd_spider,email) values (".(integer)$spider.",'".addslashes($email)."')") or die(mysql_error().$sql);
	$countEmails1 = execute_scalar("select count(*) from emails");
	echo "Emails added: <code>".($countEmails1-$countEmails0)."</code><br>";
	echo "Emails totali: <code>".($countEmails1)."</code><br>";

	return 1;

// database related functions
// -------------------------------------------------
function Connect() { if (@mysql_connect( WEBDOMAIN, DEFUSERNAME, DEFDBPWD ) && @mysql_select_db( DEFDBNAME)) return 1; else return 0; }
function saveUrl($url,$visited=0) { 
	$f = execute_scalar("select id from spider where url='".addslashes($url)."'");
	if(!$f) {
		mysql_query("insert ignore into spider (saved,url,visited) values (NOW(),'".addslashes($url)."',".(integer)$visited.")") or die(mysql_error());
		$f = mysql_insert_id();
	} else {
		if($visited==1) mysql_query("update spider set visited=".(integer)$visited." where url='".addslashes($url)."'") or die(mysql_error());
	return $f;
function execute_scalar($sql,$def="") {
	$rs = mysql_query($sql) or mysql_error().$sql;
	if (mysql_num_rows($rs)) {$r = mysql_fetch_row($rs);mysql_free_result($rs);return $r[0];}
	return $def;
function execute_row($sql) {
	$rs = mysql_query($sql) or mysql_error().$sql;
	if (mysql_num_rows($rs)) {$r = mysql_fetch_array($rs);mysql_free_result($rs);return $r;}
	return "";
// -------------------------------------------------

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" ""><html>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="content-language" content="it" />
<title>Email Scraper</title>
	code {color:#777;background-color:#f0f0f0;padding:0 3px;}
echo date("Y-m-d H:i:s");
echo "<br>";

$refresh = goSpiderGo($SEED);

if($refresh) {
	// stops and wait a small
	// amount of time to be polite with
	// the target server.
	$timer = rand(1,3)*1231;
	echo "reloading in... <code id='timer'>".$timer."</code> milliseconds";
	echo "<script>
		document.location.href = document.location.href;
		if(q-10>0) q = q-10; else q=0;


I'm a software engineer, an everyday web developer and a maker. I usually build sites with PHP, within or without WordPress. I build Internet of Things with Arduino and ESP8266. I'm the founder of and and I'm actually the Chief Technical Officer of Better Days web agency.


How to use photos on your site

UPDATE: 2013-12-04 I’ve made a method in the Mini Bots PHP Class that lets you retrieve images from instagram without…

August 18, 2011

get MySpace events with a PHP function

Here is a function to read the concerts for a myspace band page. This code retrieves the “shows page” for…

February 21, 2011

PHP bot to get wikipedia definitions

Wikipedia, the collaborative and multilingual encyclopedia project, has a lot of usefull terms defined in its database, you can find…

August 29, 2010

Send push notification to iPhone with PHP and

UPDATE: 2013/11/18 This code no longer works, but you can check a similar function here: Sending push notifications with php…

August 9, 2010

PHP to post on a Facebook page

Hi, I’ve modified the Mini Bot Class, I’ve fixed the Facebook status update and I’ve implemented the function to post…

July 28, 2010

Parsing Flickr Feed with PHP tutorial

I’ve spent about 30 minutes to find a javascript embed to print out a custom thumbs list of flickr photos,…

May 30, 2010