Dreamhost's policy on remote file access

Can anyone tell me whether the new policy to dis-allow accessing remote files through php will be a blanket policy that halts the use of RSS feeds?

shouldn't do that at all no

I've never seen an rss reader that used that function

jason

> shouldn't do that at all no
>
> I've never seen an rss reader that used that function

Nope, definitely not - the policy change is for that very specific function only, not for the generic act of "grabbing the content of a remote file and doing something with it". RSS, Atom, etc. tools are fine, policy-wise.

Basically, we're just trying to thwart a recent onslaught of security problems.

- Jeff @ DreamHost
- DH Discussion Forum Admin

I use PHP to include some RSS feeds. Using PHP, as far as I know, requires you to read the external file in some manner, such as the fopen(), file(), or the CURL functions. According to the PHP docs, those functions require fopen wrappers to be enabled.

Indeed, my testing with my local copy of PHP confirms that. If I disable allow_url_fopen on my local copy, I can no longer access the remote RSS feeds. That said, the Dreamhost change does not, in fact, appear to have caused any problems reading external files.

I too need an alternative. My news page on my main website : http://www.infinity-zero.net links to a script on my forums forums.infinity-zero.net

<?php include("http://forums.infinity-zero.net/ssi.php?a=news&show=7"); ?>

While I might be able to move that script from my forums account to my default domain account I have a second script which takes RSS feeds from my blog hosted by google and then makes an image out of it.

What can I use to get the RSS feed?

Can I compile my own instnace of PHP, run it as a CHI, and then enable the url thing?

Nope. I just checked. Its not possible for me to move the ssi.php script as it intergrates with the rest of the IPB forums. What can I do?

Haven't looked at their source code yet, but MagpieRSS has this on their page: "Does not use fopen(), work even if allow_url_fopen is disabled."

So maybe you could use that script, or copy whichever method they're using to retrieve the RSS feed.

Just minutes ago, the change was apparently finally implemented. All the external RSS feeds i was displaying died and my currency conversion, which uses an external XML file, quit working as well. This is totally unacceptable. If there is not some other way to grab the content of an external file, I'm afraid I'll have to change hosts.

I'm using MagpieRSS to syndicate my feeds, and I'm having no problems with them.

> This is totally unacceptable.

We actually publicly announced this change over a month ago, though for some inexplicable reason not all customers were constrained. While this does regrettably inconvenience some people, I'm not sure I'd go so far as to say that it's unacceptable.

While we empathize with your situation, URL-based includes tended to entail a number of security and performance related issues that we had to head off. This is necessary in order to maintain the stability and integrity of our hosting servers and - by extension - your site(s).

As for workarounds, various people have been using cURL to pull content from outside sources. I imagine there are other implementations of this feature as well.

- Jeff @ DreamHost
- DH Discussion Forum Admin

Thanks for that. Yep. I've found that you can work around it using fsockopen instead of fopen or file_get_contents. It's just a bit more work.

I use custom call buttons for my service since you made your change they no longer work on the MANY domains I host with you. Is there a work around on this? It calls up an image. here is a piece of the code.If not I also will have to find other means of hosting :(

Thanks

function imageOutput($filename) {
//this function outputs an image when run.
Header("Content-type: image/gif");

// get contents of a file into a string
$handle = fopen ($filename, "r");
$contents = fread ($handle, filesize ($filename));
echo $contents;
fclose ($handle);
}
if ($good)
imageOutput($filename);

It is true about the announcement being over a month ago. I waited with "bated breath" till (and through March 19th), and initiated a couple of threads re the consequences to various 3rd Party scripts on this board. The 19th of March came, and went, with *no* change on any of my domains....I even posted here about why phpinfo showed allow_url_fopen still enabled and got no answer, other than one post which suggested it wasn't relevant for PHP as CGI.

Tested all my sites, all was well...no problem. *now*, with three sites under development and *no* time reserved to troubleshoot this change, it occurs. No better than no warning....actually worse, as time and effort was wasted in the days surrounding the 19th of March for no reason.

Guys, it is your host, and you should certianly do with it as you will..but common courtesy, not to mention a modicum of respect for your customers, requires that you *do what you say you will do, when you say you will do it*....If you are not going to do that, than don't bother pretending that you are keeping your customers informed...just do as you please and let the chips fall where they land.

So far:

All Mambo CMS Newsfeeds are broken...

still looking for additional collateral damage.

This sucks much worse than when I had prepared myself and clients for potential problems on March 19th...

Live and learn,

rlparker

We also are seeing a lot of damage with the change not just with my code I posted above. You know a lot of us use newsfeeds,other types of PHP scripts,codes,programs,etc and we don't ALL look at every tiny bit of code. So I had no idea this was going to cause us such a mess. But now I do have an idea of what a mess this is!

I have had around 100 or so people sign up under me for dream host accounts,with that said all the problems we have had in the last few months with data bases down,servers down and now this I'm really at a loss. Some of them are having problems now with this and are pretty upset. But like you said RL it's dreamhost servers and they can do as they wish. It's just to bad with this change it's causing such a major head ache.

Edited by webgods on 04/21/05 03:09 AM (server time).

I agree that this was handled badly. I got an annoucement this morning indicating that we should be using cURL instead of the file system functions to get remote files. Had this been indicated back when the original annoucement was made, instead of the post here telling us it wouldn't make any difference, we would have had some time to prepare. It seems to me that I looked into cURL some time ago and it was not enabled on the DH servers, so I had discounted the possibility of using it now.

At any rate, cURL is now enabled and works just fine to replace any constructs where you may have been using fopen or file_get_contents to get a remote file. I had a little problem getting cURL enabled on my Windows development machine. For any others that are having the same problem, see http://zend.com/pecl/tutorials/curl.php for some tips on enabling it. The one thing that page does NOT mention is to copy the file libeay32.dll from the PHP dlls directory to your Windows system directory. Once done, everything worked correctly.

Good luck to you all.

This has cause a big problem on my site as well. I use Magipie on my site and it has been "incapacitated." I'm basically a novice php user but I have a feeling Magpie itself is "allow_url_fopen" immune, however the "require_once" calls used to include it are not.

I have been forced to try and learn cURL in the few minutes I have between other important jobs and I am failing miserably.

I agree that there could have been a bit more information given to inexperienced users that may not have been aware that the changes would affect them. When I saw the first notice I had no idea what "allow_url_fopen" meant, so I asumed that it was an advanced feature that woul dbe recognized by people that used it. If the message had included some language indicating that "require" and "include" would be disabled I would have had a much better chance of avoiding a sudden semi-catastrophic site failure. Dreamhost has always done right, in my experience, in the past, however I feel in this instance more could have been done to explain the change and the ramifications of disabling these common php functions.

I understand the need for security, and I understand why the change was needed. However, the "we told you so" attitude seems uncharacteristic of this company.

Finally, I know many of you have a firm grasp around the testes of php and curl. Perhaps someone could explain to me how to get cURL to retrieve a php script file and then run that script. All I get is the text of the document displayed on the html page. Three hours of Googling has been fruitless.

Thanks for your time.

-Jay

If it were me (and it's not).. I would curl it to get it, save it localy, and then use inclide or include_once on the local version.. If appropriate, I might also check the age of the local version and only retrieve it (curl) once a minute, or 5, or 20.

-Jason

I40.com - Home Page
MP3Mystic - Personal Streaming Music server.
(No longer hosted with Dreamhost)

Jay-

I too am learning my way around this, so my answers should *not* be considered authoritative. Last night, however, I *did* manage to get my newsfeeds back using magpierss. All my domains in question are running PHP as CGI (DH recommended and now the DH default for new domains). User macmanx also reports using magpie without any problems.

To see my quick-and-dirty workaround in action you can visit http://www.helium-labs.com/ and select the "Science News" link or, for another example (of the code I included below) visit http://www.cgiservices.net/sciencenewsx.php.

In the hope that this could be of help to you, I suggest you:

1) Get the latest magpierss from sourceforge (magpierss-0.71.1.tar.gz as of last night)

2) Install into a web accessable directory as per instructions included in the tarball. The only trick here (if you don't want to edit the class file) is to store the 5 indicated files into a directory called magpierss.

3) edit/modify one (or more) of the example .php files included in the tarball to reflect your needs.

4) Integrate those edited .php files into your site to display content as desired.

I did not find any problems with "reguiring" the necessary files to operate magpie.

I have included a short example .php file that worked for me, showing two different methods of callilng the rss feed with the magpie rss_fetch function:

--- begin code ------

<html>
<head>
<title>News</title>
<link href="http://www.helium-labs.com/templates/rhuk_solarflare_ii/css/template_css.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<?
require_once('magpierss/rss_fetch.inc');

$url = 'http://rss.news.yahoo.com/rss/science';
$rss = fetch_rss( $url );

echo "Latest From: " . $rss->channel['title'] . " ";
echo "(Links will open in a new window)";
echo "<ul>";
foreach ($rss->items as $item) {
$href = $item['link'];
$title = $item['title'];
echo "<li><a href=$href target='is_blank'>$title</a></li>";
}
echo "</ul>";

$rss = fetch_rss('http://news.search.yahoo.com/news/rss?p=RFID+Privacy&ei=UTF-8&fl=0&x=wrt');
echo "Latest from: " . $rss->channel['title'] . " ";
echo "(Links will open in a new window)";
echo "<ul>";
foreach ($rss->items as $item) {
$href = $item['link'];
$title = $item['title'];
echo "<li><a href=$href target='is_blank'>$title</a></li>";
}
echo "</ul>";
?>
</body>
</html>

------ end sample code ---------

A file similar to this one (I used my Mambo stylesheet, and didn't particularly code it pretty!), that lives is the directory *above* your "magpierss" folder installed per the instruction in the tarball should work; it works well for me and can be included itself as needed in your code or wrapped in an iframe (which is what I did for a quick and dirty implementation in Mambo to replace the now-broken Mambo newsfeed component).

To me, this seemed easier and quicker than grappling with cURL, though i *do* intend to attack that another day.

Regards,

rlparker

I can do that, but it doesn't make any sense to me. All the atom.xml files I want to parse, as well as all of the MagpieRSS files are already stored locally.

I already tried using inlude_once to retrieve them and it failed.

Do those files need to be stored somewhere specifically for them to be accessed "legally"?

I have been using basically the same setup.

<?php

define('MAGPIE_DIR', 'http://www.gloryfish.org/blogs/magpie/');
require_once(MAGPIE_DIR.'rss_fetch.inc');

$rss = fetch_rss( 'http://www.gloryfish.org/blogs/news/atom.xml' );

//push blog content onto array:
$item = $rss->items[0];

//prepare strings from array:
foreach($rss->items as $item) {

$href = $item['link'];
$href = $item['link'];
$title = $item['title'];
$created = $item['created'];
$content = $item['atom_content'];

//display content:
echo "<h2>$title</h2>n";
echo "$contentn";
echo " n";

}

?>

The only difference I can see, essentially, is that I am using an absoulute path to the rss_fetch.inc script, as opposed to a relative path. Would that have any effect?

Also, thank you all for responding. Makes me smile. :)

-Jay

Update: Darn skippy that was it! Woo hoo!

I changed the paths to my locally stored content from absolute to relative and everything is fine. Thanks for the help rlparker!

Edited by GloryFish.org on 04/21/05 02:18 PM (server time).

Now I could be wrong about this, but i *think* the problem might be with the fact that your code that defines the "MAGPIE_DIR" references a URL ("http://etc). It would seem to me that while that would have worked before, it is the URL rather than the PATH might now be the problem.

What happens if you try the following snippet (replacing yours as appropriate):

-------- begin snippet ---------

define('MAGPIE_DIR', '/magpie/');
require_once(MAGPIE_DIR.'rss_fetch.inc');

$rss = fetch_rss( '/news/atom.xml' );

-------- end snippet -------------

using your correct PATHS instead of URLs - I'm just guessing at your actual paths, or course, and assuming that the code is being called form your http://www.gloryfish.org/blogs/ directory. I'm *not* a php coder, but I think this may be the source of the problem form other things I have read around the web on this.

If I'm nutso, please ignore, and I'll be quiet and let others more profficient than I try to help.

Good Luck!

rlparker

Yep, worked perfectly. Thank you!

In reply to:
The only difference I can see, essentially, is that I am using an absoulute path to the rss_fetch.inc script, as opposed to a relative path. Would that have any effect?

Have I missed something here? I thought the whole point of not allowing PHP allow_url_fopen was to impose security on people who wanted to open files with remote paths? It should be just fine if you use relative paths, but I'd actually suggest using absolute (relative from root) paths:

<?php
define('MAGPIE_DIR',$_SERVER['DOCUMENT_ROOT'].'/blogs/magpie/');
require_once(MAGPIE_DIR.'rss_fetch.inc');
$rss = fetch_rss($_SERVER['DOCUMENT_ROOT'].'/blogs/news/atom.xml');

...other stuff...
?>

--------------------
Simon Jessey
Keystone Websites | si-blog

Yours is "much more better" .....I *told* you I wasn't a php coder! ;-)

rlparker

That looks great, thank you.

Believe me we would have not forced you all to make these changes had the security hole presented by this PHP functionality not been so great. We have seen literally hundreds of website break-ins in the last several months alone that would have been prevented had this setting not been enabled before. I'm not exaggerating! This is probably the number one way hackers are breaking into websites currently. We had been seeing problems from it every single day.

Due to the security issues we originally announced that the change was effective immediately but then realized how many problems it would cause and rolled it back. After we realized this change would cause so many problems we did what we could to alleviate broken websites by giving everyone a lot of time to prepare in advance. There were those two announcements early on. The second announcement included a link to the PHP documentation on the PHP website with the complete list of potentially affected function calls. It is quite a long list! Unfortunately, the PHP documentation does not go into any more detail so it's not clear how each function is affected. Additionally, our announcement had a link to one of our user's websites with information about cURL and how to use it to replace url fopen.

The announcement said we would turn off allow_url_fopen 'shortly after' March 29. When deciding on the date, we had already assumed many people would put off making the change until the last minute and had already planned to give additional time beyond that date. We didn't expect people to do nothing and just wait and see what would break, but rather be proactive and make changes in advance. That was the point of giving everyone advance notice. So, March 29 came and passed and we proceeded with pushing the changes out everywhere. It took a little longer than expected but now it has gone out.

At this point, I'm not sure what else we can say but 'we told you so' since that was really the only option we had. It was a fairly major change and we knew it would break websites and we didn't have any easy way to know which websites or in which way. We included links to related information in the announcement in an attempt to help everyone figure out if it affected them or not. We are admittedly not really PHP programmers ourselves so the PHP documentation was just about all we had to work with, unfortunately.

From the comments here, it sounds like most everyone would have preferred a hard deadline with no leeway so we'll most likely make any changes of this sort in that fashion in the future. That will allow those of you who prefer to wait to put out fires as they come up to plan time to do that on the day of the deadline.

We are sorry for the extra work and frustration caused by this change. Once all the broken websites are fixed up, everyone's websites will be much more secure so we can get back to working on great new features for everyone.

For anyone else perusing this thread, these are the useful links:

blog.unitedheroes.net/archives/p/1630/
blog.unitedheroes.net/curl/
http://us4.php.net/filesystem

- Dallas
- DreamHost Head Honcho/Founder

I do NOT know everything there is to know about PHP
the script I'm running is for custom call buttons aka custom indicators. Here is a snip of the code. The question I have do I have to install this "curl" program just to get it to work again? For one line of code? =fo open ?

function imageOutput($filename) {
//this function outputs an image when run.
Header("Content-type: image/gif");

// get contents of a file into a string
$handle = fopen ($filename, "r");
$contents = fread ($handle, filesize ($filename));
echo $contents;
fclose ($handle);
}
if ($good)
imageOutput($filename);

It's not something you have to install. It's built into PHP just like the fopen you were using before. There's a simple example on this page:

http://blog.unitedheroes.net/curl/

The other people here who have figured out how to change their code can probably be of more assistance as well.

- Dallas
- DreamHost Head Honcho/Founder

Thanks for posting. I still don't understand what part of the code to replace with what code. Guess I'll have to find a PHP tech or hire someone to fix all the problems I now have on my sites.

Woohoo... now I got to figure out how to implement this in my current geeklog.net installation...

Ugh.

Good luck Dan.. It has cost us some $, down time, and total head ache. I swear since this has happen I keep finding more errors on scripts such as top-list which I paid $ for now they don't work so trying to get with someone who can help fix all the problems.

Just something I have noticed:

osCommerce installed with One-Click Install is broken. Currency Update functionality does not work any longer but is giving errors about URL file-access being disabled in the server configuration.

Maybe DreamHost could do something to make sure that One-Click installed software would pass DH security requirements? I know that you actually don't store sources but pull them from outside but maybe some note in the email received after successful installation or maybe even on-the-fly patching or something?

- miikka

Just to be clear, osCommerce isn't broken, one feature is. Outside of that one function, osCommerce works brilliantly off of one-click.

Yes, true. My fingers were a little bit too quick today :)

(I still think that something, which is installed using DH One-Click installation system should work fully in DH's environment by default including standard features included in the base install; plugins etc. which are installed by the user are another thing then of course)

- miikka

We'll look into this and see what's going on. Thanks for the information.

- Dallas
- DreamHost Head Honcho/Founder

I figured out what to do since fopen is no longer allowed on dreamhost and since I have spent A LOT of $$ on scripts and programs. I moved some of my domains to a hosting company that has no problem with fopen.(I still have a lot of domains hosted here) :) everything works again and I'm back to what I do best making $. Good luck people!

I understand your point of view but I will tell you that you are just delaying what is going to be a standard for hosting..

fopen is insecure and it is not in the best interest of shared hosting machines to have it open..

____________________

WP/Gallery2 Integration Community -> http://wpg2.ozgreg.com/

Well I'm sure that the updates with all these scripts will come out soon which in time will take care of the problems. But until then I did not have much choice since no one I know knew how to fix the problems. I don't understand it enough with out screwing up everything lol.

Dreamhost's policy on remote file access