Feb 15 2010

Mixing bots to gain new services

Category: Php,Spiders & web botsGiulio Pons @ 12:39 pm

Spiders and bots let you take services from other web sites, this could be very cool, but also this could become a problem (you are using stuff made from other people, is it correct? they know what you’re doing, are there any bandwidth problems you can cause? are your bots ok with copyright?).

Well let’s go over all this problems and try to make spider’s work even more cool: you can mix two or more spiders to create something new, in this example I’ve mixed geographic ip reference bot and meteo bot to get a meteo service localized for the user who connect at your site.
This is a geographic meteo as the ones you can find on smartphones.

Have you any ideas about other mix you can do? You can grab restourants and show localized restourants, shops… There are many applications that do this on iPhone… the problem is, how much good is the result, and this depends on how good are the sources. But this geo mixes already sounds old, we have to find new mixex.

Share

Tags: , , , , ,


Dec 24 2009

PHP bot to grab meteo information from Google

Category: Php,Spiders & web botsGiulio Pons @ 3:37 pm

Google has many usefull functions that give you data fast, such as cinema infos, or for meteo forecasts. I think that Google grabs those informations from the many sites indexed with his bots.

As I did on a previous post for words spelling you can retrieve those informations with some mini bots. The mini bot I’ve made for meteo retrieves informations from italian Google servers about weather forecast of a specified city (not only italian cities).

Since google gives only 4 days meteo if you ask for a date too much in the future it will return an empty string.

Below you can find the PHP source code, and here is a working demo!

This function is included in the Mini Bots Class.

function dayadd($days,$date=null , $format="d/m/Y"){
	//add days to a date function
	return date($format,strtotime($days." days",strtotime( $date ? $date : date($format) )));
}

function attr($s,$attrname) {
	//get the attribute value of an html tag
	preg_match_all('#\s*('.$attrname.')\s*=\s*["|\']([^"\']*)["|\']\s*#i', $s, $x);
	if (count($x)>=3) return $x[2][0];
	return "";
}

function doGoogleMeteo($q,$date) {
	if ($date>dayadd(3,date("Y-m-d"),"Y-m-d"))return "";

	// grab google page with meteo query
	$web_page = file_get_contents( "http://www.google.it/search?q=meteo+" . urlencode($q) );

	//parse html to find data, and store them in an array
	preg_match_all('#<div class=e>(.*)</table>#Us', $web_page, $m);
	if (count($m)>0) {
		$p = array();
		preg_match_all('#<img([^>]*)?>#Us', $m[0][0], $img);
		for ($i=0;$i<count($img[0]);$i++) {
			$tag = str_replace("src=\"/","src=\"http://www.google.it/",$img[0][$i]);
			$p[dayadd($i,date("Y-m-d"),"Y-m-d")]["title"] = attr($tag,"title");
			$p[dayadd($i,date("Y-m-d"),"Y-m-d")]["img"] = attr($tag,"src");
		}
		preg_match_all('#<nobr>(.*)</nobr>#Uis', $m[0][0], $nobr);
		for ($i=0;$i<count($nobr[1]);$i++) {
			$temp= explode("|",$nobr[1][$i]);
			$p[dayadd($i,date("Y-m-d"),"Y-m-d")]["min"] = trim($temp[1]);
			$p[dayadd($i,date("Y-m-d"),"Y-m-d")]["max"] = trim($temp[0]);
		}
		return $p[$date];
	}

	return "nada.";
}

print_r ( doGoogleMeteo("milano","2009-12-25") );
//Array (
// [title] => Rovesci
// [img] => http://www.google.it/images/weather/rain.gif
// [min] => -4°C
// [max] => 7°C
//)
Share

Tags: , , , , , , , , ,