file_get_contents and sentinel

Posted: Wed Jul 25, 2007 8:02 am

Maybe not the correct forum, but anyway I am trying to check the content of a page using file_get_contents($url). Sentinel keeps blocking me. Why is this?

registered · Posted: Wed Jul 25, 2007 8:58 am

Could you be more specific about what you are doing? Sentinel examines $_GET and $_POST and doesn't have anything to do with file_get_contents() per se. But if you were passing the $url to a script that might do it....

Posted: Wed Jul 25, 2007 9:01 am

Simply trying to check if a link exist on another page.

Code:

function checklink($url, $mydomain){


            $url = strtolower($url);


            if (strstr(file_get_contents($url), $mydomain)) {


                $link = "linked";


            } else {


                $link = "not linked";


            }


    return $link;


  }

Posted: Wed Jul 25, 2007 9:58 am

How is checklink() being invoked? Are you calling it after passing $url or $mydomain in $_GET or $_POST?

How do you know it is file_get_contents()?

What does Sentinel say on the ban screen for the reason? What are the $_GET & $_POST values on the ban screen?

Posted: Wed Jul 25, 2007 10:14 am

No, I am not using $_GET or $_POST which leaves file_get_contents Razz

Code:

checklink($usersite, $nukeurl);

I know its file_get_contents because I stripped out everything from the function echo'd the output. BTW $usersite is just user_website from the database.

This is what I get when I did so nothing special

Quote:

You have attempted to access this site with an invalid User Agent.

If you think this is a mistake you can contact the site webmaster at admin@blah.com.

Be SURE to include the following information in any email!
User Agent: none
Remote Address: 72.249.25.225
Client IP: none
Forwarded For: none

Just curious if there are some extra steps I should take to allow that site to access the other.

registered · Posted: Wed Jul 25, 2007 10:21 am

"invalid User Agent" seems to be the key here. You may need to use the cURL functions to do a proper HTTP retreival and spoof the User Agent

Posted: Wed Jul 25, 2007 10:21 am

SGremmie, I guess my real question is how can bypass that invalid user agent bit.

Posted: Wed Jul 25, 2007 10:26 am

Oh I get it now....lol....sorry...

As evaders suggests, you might have to form up a more proper HTTP request.

Or if the site you are querying is under your control, you can modify Sentinel to remove that invalid user agent check. There was a recent thread about how to do this. I hope Bob adds a flag for that like the invalid IP check flag in a future version.

Posted: Wed Jul 25, 2007 11:22 am

Anyone a curl pro? I see lots of options to use in my function but I honestly have no clue what they mean. I am simply guessing and guessing.

Code:

function checklink($url, $mydomain){


   $ch = curl_init();


   curl_setopt ($ch, CURLOPT_URL, $url);


   curl_setopt ($ch, CURLOPT_HEADER, true);


   curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);


   curl_setopt ($ch, CURLOPT_FRESH_CONNECT, true);


   $contents = curl_exec ($ch);


   curl_close ($ch);


   $arrText = array();


   $arrText = explode("\n", $contents);


   if (strpos ($contents, $mydomain) != FALSE) {


       return true;


   } else {


       return false;


   }


}

Posted: Wed Jul 25, 2007 11:27 am

I can see now I got some learning to do.

Posted: Wed Jul 25, 2007 11:50 am

Chalk one up to the manual for some help.

This needs some refining. I still don't understand some of it. Anyway works like a champ.

Code:

function checklink($url, $mydomain){


   $agent = "CTLinks";


   $ch = curl_init();


   curl_setopt ($ch, CURLOPT_URL, $url);


   curl_setopt ($ch, CURLOPT_HEADER, 0);


   curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);


   curl_setopt ($ch, CURLOPT_FRESH_CONNECT, 1);


   curl_setopt($ch,  CURLOPT_USERAGENT, $agent);


   $contents = curl_exec ($ch);


   curl_close ($ch);


   $arrText = array();


   $arrText = explode("\n", $contents);


   if (strpos ($contents, $mydomain) != FALSE) {


       return true;


   } else {


       return false;


   }


}

Don't forget to set the agent string to something relevant. Like I said, it works but I haven't put it through a tough beta yet.

registered · Posted: Wed Jul 25, 2007 7:08 pm

floppydrivez wrote:

I can see now I got some learning to do.

Me too.

Thanks Evaders!

Posted: Wed Jul 25, 2007 7:13 pm

This has actually turned out to be very useful. Its for sure worth the read of what is capable with cURL. The limit here is a server must have it installed.

http://www.php.net/manual/en/ref.curl.php

http://www.php.net/manual/en/function.curl-exec.php

Posted: Wed Jul 25, 2007 8:16 pm

cURL is pretty fun stuff. Sadly it allows the hackers to do the same as you, HTTP requests that look pretty legitimate... fortunately I've not seen massive abuse of it yet.