PHP - Multi Curl Optimized For Speed
Iv'e been looking in to methods of scraping data from pages and has found several examples of using multi-curl to achieve this. But i am not used to curl and is not completely sure how it works and i need to find the fastest reliable (i do need all, or close to all, pages every run) method of getting the content of a number of pages (about 160).
Here is an example i got from searching the web which i managed to implement: <?php /** * *@param $picsArr Array [0]=> [url], *@$picsArr Array will filled with the image data , you can use the data as you want or just save it in the next step. **/ function getAllPics(&$picsArr){ if(count($picsArr)<=0) return false; $hArr = array();//handle array foreach($picsArr as $k=>$pic){ $h = curl_init(); curl_setopt($h,CURLOPT_URL,$pic['url']); curl_setopt($h,CURLOPT_HEADER,0); curl_setopt($h,CURLOPT_RETURNTRANSFER,1);//return the image value array_push($hArr,$h); } $mh = curl_multi_init(); foreach($hArr as $k => $h) curl_multi_add_handle($mh,$h); $running = null; do{ curl_multi_exec($mh,$running); }while($running > 0); // get the result and save it in the result ARRAY foreach($hArr as $k => $h){ $picsArr[$k]['data'] = curl_multi_getcontent($h); } //close all the connections foreach($hArr as $k => $h){ $info = curl_getinfo($h); preg_match("/^image\/(.*)$/",$info['content_type'],$matches); echo $tail = $matches[1]; curl_multi_remove_handle($mh,$h); } curl_multi_close($mh); return true; } ?> Since time is critical in my script i would ask if you think this is a good implementation or if you can point me in the direction of one that will save me noticeable run-time. Similar Tutorialsgood day dear community, i am workin on a Curl loop to fetch multiple pages: i have some examples - and a question: Example: If we want to get information from 3 sites with CURL we can do it like so: $list[1] = "http://www.example1.com"; $list[2] = "ftp://example.com"; $list[3] = "http://www.example2.com"; After creating the list of links we should initialize the cURL multi handle and adding the cURL handles. $curlHandle = curl_multi_init(); for ($i = 1;$i <= 3; $i++) $curl[$i] = addHandle($curlHandle,$list[$i]); Now we should execute the cURL multi handle retrive the content from the sub handles that we added to the cURL multi handle. ExecHandle($curlHandle); for ($i = 1;$i <= 3; $i++) { $text[$i] = curl_multi_getcontent ($curl[$i]); echo $text[$i]; } In the end we should release the handles from the cURL multi handle by calling curl_multi_remove_handle and close the cURL multi handle! If we want to another Fetch of sites with cURL-Multi - since this is the most pretty way to do it! Well I am not sure bout the string concatenation. How to do it - Note I want to fetch several hundred pages: see the some details for this target-server sites - /(I have to create a loop over several hundred sites). * siteone.example/?show_subsite=9009 * siteone.example/?show_subsite=9742 * siteone.example/?show_subsite=9871 .... and so on and so forth Question: How to appy this loop into the array of the curl-multi? <?php /************************************\ * Multi interface in PHP with curl * * Requires PHP 5.0, Apache 2.0 and * * Curl * ************************************* * Writen By Cyborg 19671897 * * Bugfixed by Jeremy Ellman * \***********************************/ $urls = array( "siteone", "sitetwo", "sitethree" ); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i]=curl_init($url); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout curl_multi_add_handle ($mh,$conn[$i]); } do { $n=curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $res[$i]=curl_multi_getcontent($conn[$i]); curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); print_r($res); ?> I look forward to your ideas. I have an array of proxies and I need one to be chosen randomly and applied to each curl handle. This script runs successfully when proxy support is removed, but as soon as I try to add a proxy it refuses to connect. My proxies are good. I have private proxies that validate by confirming that the IP address of the requester is on a whitelist of IPs I configured. I've also tried working public proxies with no validation, with the same result. The error it returns is that it "cannot connect to google.com:3238". Why it's trying to connect to that port is beyond me, the proxies I use, their port is 3238, but it should be connecting to IP:PORT then Google.com:80. Not sure why it's mixing them up. Here's my code: Code: [Select] function multi($urls,$proxies,$agents,$referer) { // create the multi curl handle $mh = curl_multi_init(); $handles = array(); $i = 0; foreach ($urls as $url) { // create a new single curl handle $ch = curl_init(); // setting several options like url, timeout, returntransfer // simulate multithreading by calling the wait.php script and sleeping for $rand seconds curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_PROXY, $proxies[rand(0,count($proxies))]); curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1); curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); curl_setopt($ch, CURLOPT_REFERER, $referer); curl_setopt($ch, CURLOPT_USERAGENT, $agent); // add this handle to the multi handle curl_multi_add_handle($mh,$ch); // put the handles in an array to loop this later on $handles[] = $ch; $i++; } // execute the multi handle $running=null; do { curl_multi_exec($mh,$running); // added a usleep for 0.25 seconds to reduce load usleep (250000); } while ($running > 0); echo 'Curl error: ' . curl_error($ch)."<br />".print_r(curl_getinfo($ch)); // get the content of the urls (if there is any) for($i=0;$i<count($handles);$i++) { // get the content of the handle $output[] = curl_multi_getcontent($handles[$i]); // remove the handle from the multi handle curl_multi_remove_handle($mh,$handles[$i]); } // close the multi curl handle to free system resources curl_multi_close($mh); // echo the output to the screen return $output; } I only post on forums as a last resort. Typically a few Google searches solves the problem, but I've spent too much time researching this one with no progress. I'm assuming it's simply a parameter that needs to be added to the CURL handles, but I've tried every proxy related config option out there and still no dice. i want to do it with cURL-Multi since this is the most pretty way to do it! Well i am not sure bout the string concatenation. How to do it - Note i want to fetch several hundred pages: see the some details for this target-server sites - /(i have to create a loop over several hundred sites) siteone_dot_com?show_subsite=9009 siteone_dot_com?show_subsite=9742 siteone_dot_com?show_subsite=9871 and so on and so fort how to appy this loop into the array of the curl-multi <?php /************************************\ * Multi interface in PHP with curl * * Requires PHP 5.0, Apache 2.0 and * * Curl * ************************************* * Writen By Cyborg 19671897 * * Bugfixed by Jeremy Ellman * \***********************************/ $urls = array( "siteone", "sitetwo ", "sitethree" ); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i]=curl_init($url); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout curl_multi_add_handle ($mh,$conn[$i]); } do { $n=curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $res[$i]=curl_multi_getcontent($conn[$i]); curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); print_r($res); ?> Hey guys, Recently I bumped into a little coding problem. I'm trying to submit information to a multi-part form (including an image upload). The receiving server couldn't read my input, figured I had a nill object somewhere (RoR). First I thought the problem was in my curl setopts, but I think they are fine. It looks like the problem is with the data I'm submitting. I read a bit about multi part posting, and every article I read advised to make a postfield array, which curl would interpret and send out correctly. I did and this was my resulting array: $postfield = array ("social[comments]" => "Test", "avatar_file" => "", "friend_avatar_file" => "", "forum_avatar_file" => "", "social[signature]" => "", "forum_signature_file" => "", "save_profile.x" => "25", "save_profile.y" => "9"); That failed. So I tried making it into a URLENCODED string: $postfield = "social%5Bcomments%5D=".stripslashes($_POST['textarea'])."&avatar_file=&friend_avatar_file=&forum_avatar_file=".$file."&social%5Bsignature%5D=&forum_signature_file=&save_profile.x=25&save_profile.y=9"; This finally worked. I was able to submit the info I wanted. I'd like to send a file with it this time though. I'd need to put this into a postfield array to work though. if(!$file = upload( )) { $file = ''; } else { $file = '@'.$file; } function upload( ) { if(isset($_FILES['file']) && !empty($_FILES['file']['name'])) { $uploaddir = "temp"; if(is_uploaded_file($_FILES['file']['tmp_name'])) { move_uploaded_file($_FILES['file']['tmp_name'],$uploaddir.'/'.$_FILES['file']['name']); $searchfile = $uploaddir.'/'.$_FILES['file']['name']; return $searchfile; } } return false; } So, I got the uploading covered. I need to put generate the postfield array though. I have the feeling the problem is with the brackets [ ] I'm using. I tried urlencoding the keys in the array, but that didn't work either. Does anybody have an idea on how to handle brackets in post (multi-part) data? Thanks in advance! Hi - good evening! thanks for the answer and all the help so far Oncemore the "Fetching-part" for a little parser-script: Here we go - new target urls: see the overview: http://dms-schule.bildung.hessen.de/index.html http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html Search by pressing the button "type" and then choose all schools with the mouse! Results 2400 schools: Here i can provide some "more help for getting the target!" - btw: see some details for this target-server: http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9009 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9742 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9871 well - you see i have to iterate over the sites - with a function /(a loop) http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=1000 to 10000 BTW - after fetching the page i have to see which one are empty - those ones do not need to be parsed! Well - i want to do this with curl-multi since this is the most advanced way to do this: I see i have an array that can be filled I have to try it out...!! Well, I'm not pretty sure - but judging by the low level of PHP i have: I think that i may need a double array to define the URLs like: Can i do like so?: $urls = array( "siteone" => "suche_schul_db.html?show_school=%i", "sitetwo" => "suche_schul_db.html?show_school=%i", "sitethree" =>"suche_schul_db.html?show_school=%i" ); $params = array ( for ($i = 1; $i <= 10000; $i++) { // body of loop } // well i have to define the variables in a open form like abvove nand not below: // better than doing it like so: "siteone" => array(9009, 9742, 9871), ); then pass them to curl-multi as: foreach ($urls as $id => $url) { foreach ($params[$id] as => $param) { $i = $id . $param; $finalurl = sprintf($url, $param); $conn[$i] = curl_init($finalurl); I don't know if that's fits the needs tip that i am looking for but I hope so ;-) Can i do like so [see above]? What do yu think. i send you many greetings martin hello dear php-friends i currently work on a little parser project i have to find solutions for the a. fetching part b. parser part here we go - the target urls: see the overview: http://dms-schule.bildung.hessen.de/index.html http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html Search by pressing the button "type" and then choose all schools with the mouse! Results 2400 schools Here i can provide some "more help for getting the target!" - btw: see some details for this target-server: http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9009 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9742 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9871 well - you see i have to itterate over the sites - with a function /(a loop) http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=1000 to 10000 BTW - after fetching the page i have to see which one are empty - those ones do not need to be parsed! Well - i want to do this with curl-multi since this is the most advanced way to do this: I see i have an array that can be filled -... but i have to think about the string-concatenation - i guess that i have make some sophisticated string concatenation. this one does not fit - for($i=1;$i<=$match[1];$i++) { $url = "http://www.example.com/page?page={$i}"; and besides this i have an array - i c an fill the array. can you help me how to run in a loop with <?php /************************************\ * Multi interface in PHP with curl * * Requires PHP 5.0, Apache 2.0 and * * Curl * ************************************* * Writen By Cyborg 19671897 * * Bugfixed by Jeremy Ellman * \***********************************/ $urls = array( "http://www.google.com/", "http://www.altavista.com/", "http://www.yahoo.com/" ); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i]=curl_init($url); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout curl_multi_add_handle ($mh,$conn[$i]); } do { $n=curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $res[$i]=curl_multi_getcontent($conn[$i]); curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); print_r($res); ?> How to draw the path of individual planes that avoid the intersection of their paths for safety and optimization of routes?!
Note:
Same number of planes. ( = M )
Same number of coordinates for airports ( junctions ). ( = N )
Add new points to avoid Intersection of paths of the planes!
flights 12 1.JPG 16.44KB
0 downloads
flights 12.JPG 16.08KB
0 downloads
Edited by Vamsi19, 17 September 2014 - 10:57 PM. I have this php page that based on what the user chooses shows them the appropriate photo galleria and i was wondering if their was any way i could speed this up this is the php code for selecting the galleria and I'm not showing the javascript that does the slide show because it s the galleriffic jquery plugin Code: [Select] <div id="Info"> <div id="page"> <div id="container"> <?php $g = mysql_real_escape_string($_GET['g']); $query = "SELECT * FROM pinkpanther_games WHERE Gallery_no = '$g'"; $results = mysql_query($query) or die("Query failed ($_query) - " . mysql_error()); $row = mysql_fetch_assoc($results); echo "<h1>" . $row['Day_Played'] . " vs " . $row['Opponent'] . "</h1>"; ?> <!-- Start Advanced Gallery Html Containers --> <div id="gallery" class="content"> <div id="controls" class="controls"></div> <div class="slideshow-container"> <div id="loading" class="loader"></div> <div id="slideshow" class="slideshow"></div> </div> <div id="caption" class="caption-container"></div> </div> <div id="thumbs" class="navigation"> <ul class="thumbs noscript"> <?php $x = $row['no_Pics']; $y = 1; $year = $row['Year_Played']; $day = $row['Day_Played']; $scheck = $row['Sessions']; if ($scheck == 1){ $sessions = "Session1"; }else{ $sessions = "Session2"; } if ($x == 0){ echo "<li> <a class='thumb' href='../images/nopics.jpg' title=''><img src='../images/nopicsthumb.jpg ' /></a></li>"; }else if ($x == 10000){ echo "<li> <a class='thumb' href='../images/coming.jpg' title=''><img src='../images/comingthumb.jpg ' /></a></li>"; }else{ while ($y <= $x){ echo "<li> <a class='thumb' href='../images/Sections/pinkpanthers/" . $year . "/" . $sessions . "/" . $day . "/" . $y . ".jpg' title=''><img src='../images/Sections/pinkpanthers/" . $year . "/" . $sessions . "/" . $day . "/thumbs/" . $y . ".jpg ' /></a><div class='caption'><div class='download'><a href='../images/Sections/pinkpanthers/" . $year . "/" . $sessions . "/" . $day . "/big/" . $y . ".jpg ' target='_blank' />Download</a></div></div></li>"; $y ++ ; } } ?> </ul> </div> <div style="clear: both;"></div> </div> </div> <div id="footer"></div> Hi everyone, I was discussing this topic with one of my friends and both of us can't give a real answer to this. Example: class test { function a(){ $obj = new DB_TableObject(); //blabla bla $this->b($obj); } function b($object){ $object->getResults(); } } class test { function a(){ $obj = new DB_TableObject(); //blabla bla $this->b($obj); } function b(){ $obj = new DB_TableObject(); $object->getResults(); } } Which one is better? The first or the second solution? Im going to use a large array of arrays, each of one having a lot of values and some sub arrays. My question is... is faster to use arrays or is better to have a object to acces using methods and all? i suppose objects are slower... Also i was planing in use arrays with string keys in nearly all places, normally these are slower, but in php hashes and arrays are the same tipe so i dont know... this is pretty decent right? Code: [Select] mySQL_array SPEED TEST - [ 30,000 Numbers ] - SERIALIZED with BASE64 ENCODING AND GZ COMPRESSION 52.8KiB WRITE ARRAY in: 0.15 seconds. 52.8KiB READ ARRAY in: 0.03 seconds. 52.8KiB ROUND TRIP in: 0.18 seconds. (approx: 3.41 milliseconds per KiB) --- running on FIREFOX + EASY PHP 5.3.8.0 - ON ASUS G51VX - 2GHz dual core (x64) - 4GB RAM --- When pulling exif data from an image I have pretty much everythiung nailed down except for the Shutter Speed value. When pulled from the image this is returned as:- 'Shutter' => string '0.0015625' Now I know that the actual shutter speed at the time of shooting was 1/640 sec; does anyone have any ideas about how one would convert... 'Shutter' => string '0.0015625' to 1/640 sec... I have been scratching my head over this problem for a few days now and am stiull no wiser Is there any php function I can test how fast my server is performing, like how fast its carrying out a certain action so I can see its resource usage. Complete beginner here so just really looking for pointers on where to start. Been doing a bit of bug fixing on one of our sites because the past 3 programmers we've hired have disappeared on us. The big issue I'm looking to solve is the load time of a search. Our website (Love-Rugs) is taking about 6 or 7 seconds to perform a blank search (basically a quick browse) whereas out other site (Little-Persia) takes about a second. It wouldn't be so bad if it was just the initial search but going from one page (only 10 products listed per page) to the next takes the same amount of time. There seems to be an awful lot of queries (around 130-150) on the searches - however - when using some of the search options e.g. type and fabric to refine the search the queries actually stay high but the time to process the results is reduced significantly. I don't really understand why if the queries are still high the search time is much lower unless it's to do with the number of results returned. However this does not explain why Little-Persia (which has far more products on it) takes less time as there would obviously be more results. I realize that without code this isn't easy to answer so I'm just looking to see if someone can point me in the right direction to look at just now. Are there any codes or php scripts that will calculate the user's internet speed? Since I got my websocket PHP server running nicely with my MYSQL, I can now have some fun
Attack speed is very simple, but I need your help with the unixtimestamp.
For example, There is a field name called "last_attack" and each time a user attacks a mob and a skill was performed; it will be updated with:
time();Then I disable the attack button for 2 seconds client side, but I also check that value against time() serverside as well. Now let's say the user's attack speed is 1.30% I want to make that Attack Speed check, to check it dynamically. It should now check only if the attack was less than 1.7 seconds ago instead of 2 seconds. How do I split up the unixtimestamp to work with percents? Edited by Monkuar, 03 December 2014 - 02:11 PM. I am currently using PHP 5.3.2 on IIS 7 and I am wondering how can I write a download script with resume and speed limit. After looking for a while, I found this script that works on speed limit: Code: [Select] <?php $local_file = 'file.zip'; $download_file = 'name.zip'; // set the download rate limit (=> 20,5 kb/s) $download_rate = 20.5; if(file_exists($local_file) && is_file($local_file)) { header('Cache-control: private'); header('Content-Type: application/octet-stream'); header('Content-Length: '.filesize($local_file)); header('Content-Disposition: filename='.$download_file); flush(); $file = fopen($local_file, "r"); while(!feof($file)) { // send the current file part to the browser print fread($file, round($download_rate * 1024)); // flush the content to the browser flush(); // sleep one second sleep(1); } fclose($file);} else { die('Error: The file '.$local_file.' does not exist!'); } ?> Though there is a big problem that the download file comes without extension. For example if the original file is file.ext then the client will receive it as file without any extension. I just want to use octet-stream since I have no interest in streaming or other things, but I really want the client to have the extension also. So could you please tell me what is wrong with the above script, and please show me a way to do resume download. Thank you very much for your help. Okay I have a function that stores data into an Array. The function takes about 7 seconds to run and through a number of different loops it creates one final array with about 15,000 keys. I want to recall this data a number of times in different functions, however how can I have this data easily accessible without running the function each time. EX: Code: [Select] function theFunction() { for($x=0;$x,=15000;$x++) { //Run the loop and store data. $string[$x] = //output from other loops and calculations } return $string } //Then later on if I want to recall the data the only way I know how is to do the follow: $newstring = theFunction() The only problem I have with this is that it has to re-run the function every time in order to get to the data it spits out. How can I store this data into another array outside of the function without having to re-run it? I hope this makes sense. Thanks. I am writing an OO PHP app to parse thousands of frames from a film, which will then each be manipulated and spat out using GD library funcs. Speed is of the essence. To improve speed, I have considered: - Another language!? - Use of PHP Command Line Interface? ( Working with web browser is surely a slow affair?) If PHP was a standard R6 Yamaha available to Joe Public, what would I have to do to win a GP race - you know - change gearing, alter timing, change exhaust system, rip out the air filter What does the panel suggest? Thanks B |