If you want to scrape content from the web don’t make it so obvious.
While CURL-ing content with PHP, it’s not uncommon for people to get upset because you’re either being clever and avoiding paying for something, or you’re just flat out stealing someone’s content.
The easiest way for them to do this is by checking the user-agent and that’s your biggest enemy. If you look in your php.ini file you’re probably set to identify as ‘PHP’ which is not only obvious but it’s easy to block. If you’ve got users visiting your domain identified as PHP; someone is trying to steal your stuff.
Fortunately there are numerous ways round this, you can modify your .htaccess file, set a PHP variable or modify the agent using Curl. Below are examples of how to identify your actions as a Mozilla browser:
.htaccess
Add the following line to your .htaccess file and that should do the trick:
php_value user_agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9
PHP set
You could also use an ini_set to define the user agent too, just place this PHP line into the head of the script doing the Curl:
ini_set('user_agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9');
Set it in Curl
You can also set the parameter in the Curl script itself, meaning that only this action is identified as Mozilla. Just add the following line into the Curl script in your PHP (not forgetting to change the $curl variable to whatever you’re using:
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9');
