Mini Bots PHP Class [OPEN]

A powerfull PHP class to perform a lot of small jobs in your PHP programs

This project has moved to Github, now it’s free and open source, you can find it here: https://github.com/giuliopons/minibots-php-class

This PHP class is a growing collection (since 2010!!) of small spiders and web bots that go out on the web and make some small useful jobs. This software can help to build your spider PHP scripts and easily brings new functionalities to your works.

You can do very nice works with CURL using php and this class, this small web bots can be included in your own works with a few line of code:

Get GPS coordinates from an address (geocode an address)

If you need to show a place on a map you need to convert the address from street, number and country, to a couple of GPS coordinates. You can use this getLatLong method of MiniBots class.

include("minibots.class.php");
$mb=new Minibots();
$gps = $mb-getLatLong("Piazza Duomo, Milan, Italy","APIKEYHERE");
print_r($gps);

And this will be the output, which you can include and use in a Google Maps:

Array ( 
[lat] = 45.465454000000001 
[long] = 9.1865159999999992 
)

Convert an IP address to a GPS location (Georeferencing IPs)

If you need to perform some action on users coming from a specific geographical location, you can use the MiniBots Class to convert an IP address to a geographical location, so you can for example choose what to show based on the country of the user. There are two methods that performs the same job but using different free services, here is the ipToGeo call:

include("minibots.class.php"); 
$mb=new Minibots(); 
$obj = $mb-ipToGeo(); 
print_r($obj);

And you’ve got this:

stdClass Object ( 
  [ip] = 93.51.121.24 
  [country_code] = IT 
  [country_name] = Italy 
  [region_code] = 25 
  [region_name] = Lombardy 
  [city] = Milan 
  [zip_code] = 20121 
  [time_zone] = Europe/Rome 
  [latitude] = 45.4667 
  [longitude] = 9.2 
  [metro_code] = 0 
)

Get informations from WikiPedia with PHP

In this class there is this method called wikiDefinition which allows you to retrieve basic informations from WikiPedia, for example you can search a famous people name and find the picture to show its face. Or you can search for a monument and get its GPS latitude and longitude coordinates.

The call is very simple and the results are packed into an object that you can easily parse to get everything you need, for example here is a call to retrieve informations about the small island of Favignana in Mediterranean sea:

include("minibots.class.php"); 
$mb=new Minibots(); 
$obj = $mb-wikiDefinition("favignana"); 
print_r($obj);

And this will be the output object, available in your code:

stdClass Object ( 
  [title] = Favignana 
  [extract] = Favignana is a comune including three islands (Favignana, Marettimo and Levanzo) of the Aegadian Islands, southern Italy. It is situated approximately 7 kilometres (4 miles) west of the coast of Sicily, between Trapani and Marsala, the coastal area where the Stagnone Lagoon and the international airport of Trapani, are sited.  
  [thumbnail] = stdClass Object (  
    [source] = https://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Favignana.jpg/320px-Favignana.jpg [width] = 320 [height] = 240 [original] = http://upload.wikimedia.org/wikipedia/commons/c/c8/Favignana.jpg 
    ) 
  [...cutted...] 
)

In the following list there are all the methods included in this class.

List of all features included:

  • get ip address of the user
  • add days to a date
  • functions to easily parse attributes and tags from html
  • get remote pages also with POST data
  • get all links and all emails from a page
  • extract text from a page
  • function to make urls from local to absolute (useful during spidering)
  • get the size of a remote file and check if a remote url exists
  • copy a remote file to your server
  • make SMTP validation for an address
  • get informations on any URL (extract meta data, works also with Amazon products)
  • get informations on YouTube and Vimeo URLs (also Youtube views count)
  • read Facebook sharing counters for any URLs with Facebook App
  • read Facebook fan counters for Facebook pages, without any facebook integration
  • get tweet counters for shared counters
  • get Google Plus counters
  • get Twitter user informations (with number of followers)
  • get images from Instagram both with username and with hashtags
  • get user counters for an Instagram user
  • get sharing counters for Pinterest
  • get sharing counters for Linkedin
  • get Google suggest for a bad spelled keyword
  • convert an address to GPS coordinates
  • get informations from Wikipedia
  • geocode an ip address to a GPS coordinates and textual informations (region, country…)
  • get money exchange rates
  • get images for search engine (Picsearch) with any words
  • get Gravatar pic for an email
  • interact with Pingomatic service
  • interact with NotifyMyDevice service

This is a growing and changing list.

Some methods could stop working, so please check if the method you need is working with the interactive demos of MiniBots Class. If the method you’re searching isn’t available in the interactive page you can try the complete demo test page.

Documentation, support and limitations

Every method is easy to call and documented. The documentation of the class explicitly tells to pay attention to use this class especially for production systems. It comes as it is and even if I give you support through CodeCanyon, sometimes free services are closed and can’t be scraped anymore.

Moreover, sometimes services automatically blocks spiders that makes many requests and couse heavy traffic. Remember alsways that heavy traffic means heavy costs for servers, so don’t make too many requests and store data locally when possible. Check the services you’re using (in documentations you will find the name of them) and if you have to make heavy usage contact them and buy their services.

If you’re interested on making spiders, here is a tutorial on how to build a spider.

Download MiniBots Class FREE

Now open source on Github Minibots PHP Class.

Comments on “Mini Bots PHP Class [OPEN]”

64 thoughts

  1. seeal ha detto:

    love this class thx :D

  2. Matteo ha detto:

    Great class , thanks :)

  3. Greg Avola ha detto:

    Great work dude – way to go.

  4. holyguard ha detto:

    Utilissimo!!!

    Puoi usare l’exchange rate di Google invece di quello della banca d’Italia, che te ne pare?

  5. Tod Wedge ha detto:

    Just landed on this place via Google seek. I love it. This post change my percept and I am taking the RSS feeds. Cheers Up.

  6. webcarlo ha detto:

    complimenti però sappiate che Google non gradisce che si facciano query nei suoi servizi che non prevedono API, potrebbe penalizzare un sito che serve i dati con il grabbing

  7. admin ha detto:

    Credo che l’importante sia non abusarne, anche Google fa grabbing a tutti i livelli, non sono sicuro che penalizzi queste attività, dove l’hai letto?

  8. Jorge Garifuna ha detto:

    I just tried updating my twitter status with the mini bot’s twitterSetStatus() function and the functioned returned false which indicates that my status was not updated. I verified on twitter and no update was set to my status.

    Does this have anything to do with the oAuth implementation of twitter?

    Any ideas to get twitter update status to work in the near future?

  9. Giulio Pons ha detto:

    Yes. Since 31, 2010 twitter made basic autenthication deprecated, I will soon make available a different version of the function (I hope).

  10. Jorge Garifuna ha detto:

    Thanks a million. I’m currently using the facebook status update and it works great!

  11. Jorge Garifuna ha detto:

    Any ideas to add hi5.com to this class?

  12. Ronan ha detto:

    Bel lavoro davvero !

    peccato che tu non abbia scelto una licence per questo script.
    The Free Software Foundation has recommended that the GNU AGPLv3 be considered for any software that will commonly be run over a network

    Saluti dalla Bretagna,
    Ronan

  13. Giulio Pons ha detto:

    Ciao Ronan, non conosco bene le licenze, mi leggerò quella che mi consigli. Ciao!

  14. Douglas Ianitsky ha detto:

    Hey, Thanks for class.

    I modify the googleGetImages function for get a especify quanty.

    public function googleGetImages($k, $quant = 20) {
    		$x = array();
    		$next = 0;	
    		while ($next < $quant) {
    			$url = "http://www.google.com.br/images?q=##query##&gbv=2&sout=1&tbs=isch:1,itp:##query##&sa=N&start=##next##&ndsp=20";
    			$web_page = file_get_contents( str_replace(array("##query##", "##next##"), array(urlencode($k), $next), $url ));
    			$tieni = stristr($web_page,"dyn.setResults(");
    			$tieni = str_replace( "dyn.setResults(","", str_replace(stristr($tieni,");"),"",$tieni) );
    			$tieni = str_replace("[]","",$tieni);
    			$m = preg_split("/[\[\]]/",$tieni);
    			
    			for($i=0;$i<count($m);$i++) {
    				$m[$i] = str_replace("/imgres?imgurl\\x3d","",$m[$i]);
    				$m[$i] = str_replace(stristr($m[$i],"\\x26imgrefurl"),"",$m[$i]);
    				$m[$i] = preg_replace("/^\"/i","",$m[$i]);
    				$m[$i] = preg_replace("/^,/i","",$m[$i]);
    				if ($m[$i]!="") {
    					array_push($x,$m[$i]);
    					if (++$next >= $quant)
    						break;
    				}
    			}
    		}		
    		return $x;
    	}
    
  15. Douglas Ianitsky ha detto:

    Another small change

    // get list of images from google images
    //$k == query to search
    //$quant == total of itens that you need
    //$check == check if image link is broken (depending on the amount of images, the time to check them becomes too long. Use only if absolutely necessary.)
    public function googleGetImages($k, $quant = 20, $check = false) {
    	$x = array();
    	$next = 0;	
    	while ($next < $quant) {
    		$url = "http://www.google.com/images?q=##query##&gbv=2&sout=1&tbs=isch:1&sa=N&start=##next##&ndsp=20";
    		$web_page = file_get_contents( str_replace(array("##query##", "##next##"), array(urlencode($k), $next), $url ));
    		$tieni = stristr($web_page,"dyn.setResults(");
    		$tieni = str_replace( "dyn.setResults(","", str_replace(stristr($tieni,");"),"",$tieni) );
    		$tieni = str_replace("[]","",$tieni);
    		$m = preg_split("/[\[\]]/",$tieni);
    		
    		for($i=0;$i<count($m);$i++) {
    			$m[$i] = str_replace("/imgres?imgurl\\x3d","",$m[$i]);
    			$m[$i] = str_replace(stristr($m[$i],"\\x26imgrefurl"),"",$m[$i]);
    			$m[$i] = preg_replace("/^\"/i","",$m[$i]);
    			$m[$i] = preg_replace("/^,/i","",$m[$i]);
    			if ($m[$i]!="") {
    				if ($check) {
    					if (@getimagesize($m[$i])) {
    						array_push($x,$m[$i]);
    						if (++$next >= $quant)
    							break;
    					}
    				} else {
    					array_push($x,$m[$i]);
    					if (++$next >= $quant)
    						break;
    				}
    			}
    		}
    	}		
    	return $x;
    }
    
  16. Giulio Pons ha detto:

    Thank you Douglas!

  17. Michael ha detto:

    Wow really amazing, very usefull collection! Thanks!!

  18. nina ha detto:

    can you make function for parse facebook wall feeds ?

  19. santiago ha detto:

    Thanks!!

  20. Putranto ha detto:

    Just what i needed, twitter status change without messy oAuth !!! Thanks bro, excellent !!!

  21. Stefano ha detto:

    Sto provando il codice di Douglas per recuperare l’immagine da Google Images, ma ottengo sempre un errore di sintassi T_PUBLIC. Ho tolto il public alla funzione e non restituisce più l’errore ma non ritorna niente, pagina bianca (test effettuato tramite XAMPP).

    prima della classe ho semplicemente inserito una riga così:
    $k = “MC544FD/A”;
    Dopodichè ho modificato quantità in 1 ed abilitato il check dell’esistenza dell’immagine.

    Devo inserire altro codice per visualizzare l’immagine?

    Inoltre è possibile inserire un controllo “di pertinenza”? Mi spiego, se il primo risultato ha un nome estremamente diverso da quello cercato (e lo chiedo perchè io sto cercando dei codici di prodotto) è possibile dire allo script di skippare il grab dell’immagine?

    Grazie mille!

  22. Joseph ha detto:

    I will use this class in my collection of web directories and submission form. Thanks.
    Directory Gist – Directory of Web Directories –
    http://www.directorygist.com
    ———–
    Searcheable database of 7000 free web directories, Search Engines, social BookMarking Sites, Article Directories. Also online submission tools to ease your website promotion.

  23. SAYIM ha detto:

    Great class.
    However, I need the page URL as well in the result.
    For example, I need http://www.example.com/the_page.php as well as http://www.example.com/the_page/imageX.jpg

    Thanks,

  24. Short Links ha detto:

    Great class. Please could you add the short URL service at http://101.gs to “decode any small url to the original long url”

    Thanks

  25. Hung Nguyen ha detto:

    So great,
    Your Bot class so useful.
    Thanks

  26. C. Davis ha detto:

    Hi,

    I am trying to use your dosmtpvalidation function but am not able to build a form to pass variables to that function. Could you demonstrate what the form looks like? I tried to copy example at https://www.barattalo.it/demo/dosmtpvalidation.php including jquery and css file but submit button does nothing when I copy it exactly to my website.

  27. alex ha detto:

    hi

    Demo for minibots.class.php Does not work

  28. Kazamboo ha detto:

    googleGetImages class didnt work

  29. Jack ha detto:

    some of features are not working now.

  30. real0m ha detto:

    google get images class not working

    https://www.barattalo.it/demo/googlegetimages.php

  31. James ha detto:

    Great Class! Any chance of getting the source to the demo pages? I got the twitter class working, with _post[“u”] etc, but I think what I have can be done better. Plus when I submit it, it just displays a double of my twitter form where the result should say, bad, ok, etc.

    Could I possibly get the code for the twitter demo page?

  32. alper c ha detto:

    googlegetimages not working =(

  33. Renga ha detto:

    This is very helpfull for me.
    Thanks

  34. hi i like your work but how to use it i m new to php, what i whould to do is to get link of first 3 image ( and medium size)

    $k = ‘demo’;
    $images = googleGetImages($k, $quant = 3, $check = false)
    print_r $images;

  35. sampath ha detto:

    Hii,
    I want to get latitude and longitude from address.
    The demo which you have shown is mapping exactly.
    Can you provide me the entire source.
    Im new to PHP and not able to link it.
    Please help me

  36. Giulio Pons ha detto:

    google has changed its code and I have to fix it.

  37. gary ha detto:

    Do you ever tried to curl dropbox.com ?

    Looks impossible but i don’t know why….

    Love your bot class by the way

    thanks

  38. Giulio Pons ha detto:

    I think dropbox has api libraries, probably it’s better use the apis.

  39. Inarus ha detto:

    Awsome class Thanks!
    I notced that in minibots Ver.3.61 since 14.03.2017 getInstagramPicsByTag method does not works (Undefined offset on line 1055). –> https://www.barattalo.it/demo/minibots/test2.php

  40. Giulio Pons ha detto:

    Thanks, unfortunately Instagram has just closed the public access to the tag exploration pages. So now it’s not possible to retrieve the images by tag.

  41. Giulio Pons ha detto:

    The method to get Instagram pics by tag works again!

  42. Seb ha detto:

    Hi, it looks like Instagram methods aren’t working anymore.

  43. Giulio Pons ha detto:

    Hi Seb, I see it working on https://www.barattalo.it/instapostcards

  44. Seb ha detto:

    Ok Giulio, I see it working too in instapostcards. Maybe it is just related to GetInstagramPic method. I have a few scripts based on that method and they don’t work anymore since this morning.

  45. Giulio Pons ha detto:

    Ok, fixed!

  46. Seb ha detto:

    Awesome Giulio!

  47. Seb ha detto:

    Codecanyon shows today’s update but lets me download the previous version. Is it normal?

  48. Giulio Pons ha detto:

    I’ve just uploaded it to codecanyon, they need one or two days to approve it before you will be able to download the new version.

  49. Toby Woby ha detto:

    Hi, I think readFacebookPageCounters is not working properly anymore, I tried even with the last version 3.68!

  50. Giulio Pons ha detto:

    Hi, since August 2016 the readFacebookCounters method needs also to configure an app on Facebook and use appid and app secret. The readFacebookPageCounters (for facebook pages) I see that have big restrictions on usage, so sometimes it works and sometimes it doesn’t. I’m going to investigate a little bit more on it.

  51. Giulio Pons ha detto:

    You can try it here: https://www.barattalo.it/demo/minibots.php
    The method is working, probably there are some limitations.

  52. Avcılar Çilingir ha detto:

    Thank you eery much ercan zengin

  53. bartosz glen ha detto:

    I can with this class download photo from instagram to my FTP and description of same photo to SQL? after looking and take all photos from a #hashtag?
    if yes I buy it!

  54. Giulio Pons ha detto:

    yes, but you have to code it. the class allows you to get filenames and description taking photos by hashtag, but: it doesn’t save file locally (you have to do it), it doesn’t make queries to your db (you have to do it), it doesn’t get all the photos for a tag but only the first 12 pictures of the most recents image and the 12 most recent popular images (if you want to get all future images you have to make multiple queries with a timer and save new pictures everytime you find one).

  55. Bartosz bartosz glen ha detto:

    Thanks for fast answer. I already tested ur script and is rly cool! Big thanks! I have some question:
    1. How I can show numbers of count edge_followed_by? because I cant show same way as like $show[“user”].
    I mean how to shows this part of code:

                [edge_followed_by] => stdClass Object
                    (
                        [count] => 3787
                    )
    

    2. Did u think about update ur script to looking relation from Instagram where is one special tag?

    3. Do you have meybe a some example what exactly you mean with multiple queries with a timer?
    Here I need to add ur script download much more than 24 photos form #tag :)

  56. Vijay Dalvi ha detto:

    Hey, how can I installed mini boat on server, and how much url copy & save my db in a day.

  57. Giulio Pons ha detto:

    I don’t understand your question, sorry.

  58. Vijay Dalvi ha detto:

    (Hey, how can I installed mini boat on server, and how much url copy & save my db in a day.)
    1. How add this class in my php search engine project codding.
    2. this class how much url crawl in single day & all data can save my own database directly.

  59. prio ha detto:

    Hi, I want take information about:
    – USERNAME, COMMENTS, LIKE, FLOWERS
    when I give link for photo like this:
    https://www.instagram.com/p/BoES6mRBNPN/

    Now we have option only when I give for script $user or $tag
    How I need to change script for put WHOLE link for photo and then take information from photo and user?

  60. Giulio Pons ha detto:

    There is also a method getInstagramPic to let you get data from a picture url.

  61. Giulio Pons ha detto:

    It’s up to your coding skills. The Minibots class gives you a few method to search and retrieve data from the last pictures of a user, last pictures for a tag, or from a url of an instgram post. It doesn’t crawl, it doesn’t store in a db.

  62. Giulio Pons ha detto:

    1) If you have an item of an array which is an object you can get the variables of the object simply with => notation. echo $item["edge_followed_by"]=>count.
    2) what do you mean with “special tag”
    3) I don’t have an example, sorry. You have to make a cron which calls your php script. Your script should perform the search and if it finds new images store them. You can set cron to be called, for example, every hour. This way you have a script that every hour search for new images of a tag and you can monitor all the new images coming from a tag.

Lascia un commento

Your email address will not be published. Required fields are marked with an *