PHP Web page to text function

I’ve found this nice small bot on the www.php.net site, thanks to the author of the script on the preg_replace…

January 16, 2010

I’ve found this nice small bot on the www.php.net site, thanks to the author of the script on the preg_replace page.
This bot returns the text content of a url and it could be used to take text from a site and find relevant words to search.

It’s nice because it uses CURL and let us see some nice stuff that CURL does:

  1. hide himself presenting with a specific user agent, to seem a browser and not a spider
  2. follows redirect (this means that you can call this function with a tiny url to retrieve the text in the real page!)
  3. use very powerful regular expression to remove html tags, but also javascript and styles (they remain if you simply do a strip_tags)

This script will be included in the next version of the Mini Bots PHP Class.

function webpage2txt($url) {
	$user_agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";

	$ch = curl_init();    // initialize curl handle
	curl_setopt($ch, CURLOPT_URL, $url); // set url to post to
	curl_setopt($ch, CURLOPT_FAILONERROR, 1);              // Fail on errors
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);    // allow redirects
	curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); // return into a variable
	curl_setopt($ch, CURLOPT_PORT, 80);            //Set the port number
	curl_setopt($ch, CURLOPT_TIMEOUT, 15); // times out after 15s

	curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);

	$document = curl_exec($ch);

	$search = array('@<script[^>]*?>.*?</script>@si',  // Strip out javascript
		'@<style[^>]*?>.*?</style>@siU',    // Strip style tags properly
		'@<[\/\!]*?[^<>]*?>@si',            // Strip out HTML tags
		'@<![\s\S]*?–[ \t\n\r]*>@',         // Strip multi-line comments including CDATA
		'/\s{2,}/',
	);

	$text = preg_replace($search, "\n", html_entity_decode($document));

	$pat[0] = "/^\s+/";
	$pat[2] = "/\s+\$/";
	$rep[0] = "";
	$rep[2] = " ";

	$text = preg_replace($pat, $rep, trim($text));

	return $text;
}

echo webpage2txt("http://www.rockit.it");

Author

I'm a software engineer, an everyday web developer and a maker. I usually build sites with PHP, within or without WordPress. I build Internet of Things with Arduino and ESP8266. I'm the founder of Rockit.it and Dailybest.it and I'm actually the Chief Technical Officer of Better Days web agency.

Comments on “PHP Web page to text function”

2 thoughts

  1. Canyoun says:

    hello friend

    Thanks for this one

Comments are closed

Recommended

How many times a web link has been shared on Twitter

Twitter share button and Facebook share button are the most used buttons to share links on Internet. You can read…

October 19, 2012

get MySpace events with a PHP function

Here is a function to read the concerts for a myspace band page. This code retrieves the “shows page” for…

February 21, 2011

PHP to get twitter infos and avatar

I’ve just updated the Mini Bot Php Class with an improved version of the twitterInfo function, here is the code…

March 1, 2010

New version of Mini Bots PHP Class (v.1.4)

I’ve added three more bots to the Mini Bots Php Class, now the version number is 1.4 and it has…

January 20, 2010

Bot that retrieves url meta data and other infos

From a given url this function retrieves page title, meta description, keywords, favicon, and an array of 5 images to…

January 12, 2010

PHP bot to grab meteo information from Google

Google has many usefull functions that give you data fast, such as cinema infos, or for meteo forecasts. I think…

December 24, 2009