RewriteCond %{HTTP_REFERER} not working .htaccess

Hi All,
I am hoping someone can help me, as I am totally stumped. I wrote support and they said this was a "programming" issue and couldn't help me.

Anyway, I am trying to block referral visits from ebay to my site.

I placed the following in my .htaccess file (per the article in the wiki)
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co.uk [NC]
RewriteRule .* - [F]

And when I go to ebay and click on the link pointing to my site, it comes right up. (And yes, the referring link is cgi.ebay.com (or .co.uk).

All of the rewrites in my htaccess file are working correctly, so I have absolutely no idea what the issue is. I have cleared my cache and tried it from a completely different computer, yet my site still comes up when clicking on the referring link.

Any ideas/help/advice is greatly appreciated.

Thanks
Morty

How about changing it to (see additions in red):

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.com/.* [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co.uk.* [NC]
RewriteRule .* - [F]

The only other difference I see between yours and the ones I use on a few domains would be that I have RewriteBase / after RewriteEngine On.

Still not working :( This is really starting to drive me crazy.

I put in exactly what you had, and after clearing my cache, still able to get to the website.

Then based on your statement about RewriteBase, I tried this but I am not sure if it is what you meant.

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.com/.* [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co.uk.* [NC]
RewriteRule .* - [F]

Thanks!
Morty

Semi-educated guessing here. How about change from:

RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.com/.* [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co.uk.* [NC]
RewriteRule .* - [F]

to (red):

RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.com.* [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co.uk.* [NC]
RewriteRule ^.* - [F]

Did you check error logs for clues? What are you trying to stop, if you wouldn't mind giving a little more detail?

You're not using a referer blocker by any chance? :-)

thanks anonymous,
That didn't work either :( Ugh, this should not be that difficult.

Basically someone on ebay is pointing to my site in their listings, and I don't want to be associated with them.

Not sure what you mean by a refferral blocker, but I am not running anything, just trying to manually add it in my .htaccess file.

I have other Rewrite calls in my file, all of those work fine. So I have no idea what is going on.....

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://cgi.ebay.com/
RewriteRule /* http://www.yoursite.com/restricted_url.html [R,L]
RewriteCond %{HTTP_REFERER} ^http://cgi.ebay.co.uk/
RewriteRule /* http://www.yoursite.com/restricted_url.html [R,L]

That's a minor tweak to a copy paste from the wiki. If that doesn't work, I'm giving up for now (sorry).

You'd need to put up a restricted_url.html page or live with 404's, I suppose. A page would deliver an appropriate message if you wanted.

A referer blocker is something that removes or modifies referers being sent by your browser. For example Firefox has a plugin and some firewalls can do it. A standard setting might be to always send a referer of the homepage of the site being requested from (i.e. your site in this case). If you (or someone else) were using one, it could prevent what you're trying to accomplish.

Tried it and it is not working :(
(created the restricted_file.html and everything)

Uggghhhhh.

Thanks for your help.....

Does anyone else have this problem?
Perhaps I am missing something basic?

BTW, I stripped out everything else in my .htaccess file and tested it again (to eliminate any possible conflicts) and it still didn't work.

Morty

You might look in your access log to see if something funny is going on to make the referer different than expected. Just a thought. Now I'm giving up again. :-) Somebody will come along and be the hero before too long. Then it'll look obvious.

I'm pretty useless with .htaccess (kchrist, where are you?), but you might be able to try a different approach like this:

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://(www.)?.*(-|.)ebay(-|.).*$ [NC,OR]
RewriteRule ^(.*)$ %1 [R=301,L]

If I've worked this out correctly, I think it will block any referrer that contains "ebay" somewhere in the URL.

This brought the entire site down, giving me the following error message (even when accessing the URL directly)
The page isn't redirecting properly

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

* This problem can sometimes be caused by disabling or refusing to accept
cookies.

This is so frustrating.
This is a dyanmic php site, could that have anything to do with it? I tested blocking links from other sites that I control and I can't block those either (so this eliminates anything along the lines of the ebay referral being blank).

Help :)

Morty

PS> Thanks to everyone and their ideas so far, keep 'em coming, I am willing to try anything at this point!

Checked the logs nothing funny there.

I switched the code that scjessey recommended (taking out the OR) to
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://(www.)?.*(-|.)ebay(-|.).*$ [NC]
RewriteRule ^(.*)$ %1 [R=301,L]

this eliminated the sitewide error, but I could still link from ebay.....

I probably should have noted that I wasn't trying to add a ., but rather ".*" to catch anything -- in case there was something silly in a query string that he was getting past the check.

Give this a shot--but keep in mind that it might block more than you want it to.

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} .*ebay.(.*) [NC]
RewriteRule .* - [F]

Here's what that "should" do: Block anything with "ebay." in it, no matter where it is (I left the dot to not block way more domains than necessary). But keep in mind that if that works, you would block names like freebay.com, ebay.someotherdomain.net, etc...

If that works, then we can start adding stuff back until it stops, to see where it's going wrong.

Also, you should probably include the exact URL from where you're clicking from the browser address bar (if it's something you don't mind revealing, or don't care if we know your domain).

Thanks, but that didn't work either.

At this point I would prefer not to share the URL's.

However, the issue is not with the ebay domain itself, I can't block any domains including my own.

Am I really the only person on dreamhost having this problem?

Thanks again,
Morty

OK,
I have taken everything down to its simplest form, and tried blocking domains on a different site, hosted at the same DH server, and blocking doesn't work on these sites either.

Here's the test scenario:
From http://{snip}/sorry.htm
that link should be blocked at {snip}

I have to be missing something really really simple? right?

Thanks,
Morty

Edited by jmorty732 on 12/01/06 11:43 AM (server time).

These may seem like silly questions, but just to make sure it's not something super simple being missed:

Is your browser set to hide referrer info?

Where is your .htaccess file?

If your domain is example.com, it should be in the /username/example.com directory.

Here's a test for your browser. I just blocked discussion.dreamhost.com from seiler.dreamhosters.com.

So give this a click and see where it gets you.

I tested it and it works, so it should work for you. This is exactly what I put in the .htaccess file:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} /403test.html
RewriteCond %{HTTP_REFERER} .*discussion.dreamhost.com.* [NC]
RewriteRule .* - [F]

EDIT: Edited to change link to a specific URL so it will always work here as an example, without blocking all other links from the forum. Also updated .htaccess example to show the specific file being blocked from this site. Remove that line to block an entire domain.

Found it!!!!

It is my Norton Firewall, blocking the referrer.
UGGGHHHHHHHH.

Thank you so much! I appreciate everyone's help. I was about to lose my mind.....

Now, is there anyway to get around firewall's blocking referrers? If I find the IP and block that , would that work?

Thanks,
Morty

PS. thank you, thank you, thank you!!

Looks like I got here a little late for the party.

In reply to:
Now, is there anyway to get around firewall's blocking referrers?

Nope. Referers (sic) are strictly optional; user agents may or may not send them, as they choose, and proxy servers may strip them out at will. You could try also blocking anything with an empty referer string but this is guaranteed to block legitimate traffic as well.

What I'm curious about is why you want to prevent people from following links from Ebay to your site.

Glad it worked out! wink

Well, you can block blank referrers, but then you block type in traffic and probably a few bots that shouldn't be blocked.

RewriteCond %{HTTP_REFERER} ^$

I'd probably avoid doing that unless blocking ebay is more important than losing important traffic... and maybe even dropping from search engines.

You can block the IP like this, using 127.0.0.1 as an example:

RewriteCond %{REMOTE_ADDR} 127.0.0.1

You could shoot for either and hope the referrer isn't blocked, or that multiple IPs aren't being used:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co(.*)$ [NC,OR]
RewriteCond %{REMOTE_ADDR} 127.0.0.1
RewriteRule .* - [F]

Note that I wildcarded eBay after ".co" so it will catch com or co.uk.

Ahh, that's what I figured.

The reason I want to block the link is that someone is selling a turnkey script, and is saying that my site demo's the script. Which it sort of does and sort of does not, as I have made some serious custom modifications to it. And I never agreed to have my site used as a "demo".

Previous attempts to contact the person , have been futile, so I wanted to take the matter into my own hands.

Thanks!!!

Now, I need a nice stiff drink!!!

Morty

> So give this a click and see where it gets you.

Thanks for that. It answered another question I had, which was: If somebody follows a "blocked" link, and then hits return on the URL (navigation toolbar or whatever called), will the referer continue to be sent and blocked, or will the page display? The answer is: the page will display. OTOH, hitting reload *does* continue being blocked.

> And I never agreed to have my site used as a "demo".

You could use it as an opportunity to advertise your modified version.

Then scratch blocking an IP, since ebay users can be anywhere.

If he won't respect your wishes... Then why block, when you could just redirect to porno/beastiality sites, then report him to ebay for linking to offensive material? tongue

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} !(.*googlebot.*|.*slurp.*) [NC]
RewriteCond %{HTTP_REFERER} ^http://(cgi.)?ebay.co(.*)$ [NC]
RewriteRule .* http://INSERT-BEASTIALITY-URL-HERE [R,L]

That lets google & yahoo past ( I was too lazy to look up the exact user-agents). A link's a link... no sense in blocking the good guys. ;)

Yeah, trust me, I have thought of reporting him (or her) to Ebay, but I want to try and find a way without getting him in trouble with ebay, and without offending the customers. I figured a nice "403" would do.

I don't think the "offense" is that bad, and I try to give people the benefit of the doubt (maybe my messages are being dropped in the spam block folder, or maybe he doesn't check that email account, etc). I don't like messing with someone's revenue source, cus if someone messed with my web site revenue it would devastate me financially, you know the whole karma thing.

Thanks for your help, I truly appreciate it.

(I still can't beleive it was Norton this entire time and I didn't think of it).

> (I still can't beleive it was Norton this entire time and I didn't think of it).

Yeah, me neither, especially when I discussed referer blocking and mentioned firewalls, hours ago. :-)

You have to chuckle when you foul yourself up and spend a lot of time trying to mess with somebody else.

Correct. It's the same level of security (read: inconvenience) that prevents hotlinking.

With the 403 in this case, you have to hope that either:

1. That's enough to turn off a possible bidder, that maybe hits refresh once (and doesn't think to simple hit enter or the go button). A potential bidder could see the error as a problem with the script that they were considering bidding on and lose interest in the auction (good reason for the seller to find a new 'demo' site).

2. The seller isn't a jerk that changes the link to "Copy & paste this URL in your browser." Then you're pretty much screwed, unless you want to dedicate part of your homepage to calling the seller a jackass and linking the person to a place where he/she can get the script for free. ;)

That's why I'd take the more fun route of redirecting to porn or beastiality ... let them refresh all they want. tongue And in that case (unless he goes plain text instead of link), he'll surely get several ebay complaints about it.

If you wanted to go through more trouble, you could grab the person's IP on the first block, then block that with a script, which would block a second attempt by hitting enter. Use cookies, etc... But like blocking a blank HTTP_REFERER, you chance blocking wanted traffic as well. Personally, I wouldn't go through all that trouble unless there was a serious problem with stealing bandwidth--which probably isn't the case here with a few ebay links.

LOL

BTW, I didn't ignore your comment about the firewall. I disabled Norton, looked at the settings in my browser and tried the links and it still didn't work.

It wasn't until I saw the link that seiler provided where the blocking was confirmed, and when it didn't work for me, I decided to dig into the firewall issue further.

I had to manually set Norton to not block referrals to the domain. That's when it worked.

Thanks for your help, I do appreciate it. :)

RewriteCond %{HTTP_REFERER} not working .htaccess