cpu & memory limits

I'm trying to put together some scripts for manipulating jpegs with the imagemagick convert program. Unfortunately, I end up with my tasks being killed intermittently.

I realize these are relatively cpu/memory intensive operations, but they're not time critical - i.e. if it takes 10 minutes to complete the operation, I'm fine with that. I've tried using the nice command within the scripts, but that doesn't seem to make much of a difference.

I have several questions:
1. What are the cpu/memory limits that cause a task to be killed on dreamhost. This seems to be related to the load on the servers more than the task that I am running.
2. How can I avoid triggering such limits? Is the nice command the way to do this? How can I effectively use nice?

Any tips or hints on your experiences greatly appreciated.

I NEED KNOW THAT TOO.

I used a process php via ssh and it is killed after few time.

Regards

I didn't get a direct answer to my post, but by looking through various sources including old messages and the support wiki, I got some insights.

The task killer seems to trigger on process cpu time and memory usage and system load average of the machine you're on. So, I used a cron tab to kick off small tasks that pipelined into each other via temp directories. That way, if one task got killed and didn't complete the output, it would try again the next time cron kicked it off - eventually completing the task at hand. Works pretty well, though it does take 10 minutes or so for the whole pipeline to complete under good conditions.

If you can't accomplish what you're trying via small tasks, a Private Server might be the way to go.

YMMV.

Can you do any of these things on your local machine?

Use the 3DOM50 promo code for 3 extra lifetime domains and $50 off
More Dreamhost coupons here!

Thank you dear,

Can you tell-me wich command do you used in cron to kick off the smalls tasks?

Yes I had orderd yesterday the PS, but the delay is 7 to 10 days!! WOW :S so many time...

Regards
Janio

cron is a unix facility for running tasks at particular times. See "man cron" and "man crontab" for more details.

As to which commands I used, I wrote scripts which perform a sub-processing of the overall task I'm trying to accomplish.

If you're not a programmer, that will probably not make a lot of sense. PS would definitely be the best way to go.

Good luck.

I'm trying to implement ZenPhoto like functionality for my users, so performing such operations on my local machine doesn't make much sense.

I'm a little confused why ZenPhoto is sanctioned and can use the memory and cpu resources necessary to perform thumbnailing, but my processes can't. It looks to me like there are limits on user processes that are relaxed or absent for 1 click installs.

In reply to:

---

I'm a little confused why ZenPhoto is sanctioned and can use the memory and cpu resources necessary to perform thumbnailing, but my processes can't. It looks to me like there are limits on user processes that are relaxed or absent for 1 click installs.

---

It might help to remember that the ZenPhoto one-click install is an "easy" one-click implementation that *does not run in the normal user space on a general purpose shared server*.

Users cannot get at the filesystems on the servers that run this particular type of "one-click" install, and these servers are therefore not subject to the many and varied uses users normally inflict upon your standard DreamHost shared server. The software on these machines is optimized for the purpose for which is was intended - running ZenPhoto, and that alone!

That said, it is understandable that such a single purpose machine has different limits imposed on it than you will find on a machine where users have shell access and can install, and run, a wide variety of processes.

--rlparker
--DreamHost Tech Support

In reply to:

---

I'm trying to implement ZenPhoto like functionality for my users, so performing such operations on my local machine doesn't make much sense.

---

Ah, that's an interesting use. Since it's running as a cron job and wasn't time critical, and you were running into resource limitations on your Dreamhost web server, I was thinking that you might be able to utilize a server sitting at your house running a cron job where it would nightly suck the images down from your Dreamhost web server, do all the cpu intensive image-manipulation work, then reupload all the processed thumbnails.

BTW, I know there's some argument about whether it really works or not, but have you tried running your jobs niced?

BTW, another potential solution is only running your script when the load is low. This doesn't solve the problem entirely because your job may be killed anyway but is very nice to your neighbors on the server. Then again, nice may satisfy that requirement adequately. (Don't mind me, I'm just paranoid about server load from waiting processes, since that's a frequent server killer)

Use the 3DOM50 promo code for 3 extra lifetime domains and $50 off
More Dreamhost coupons here!

I have a related question:

How do I tell how much CPU time I am consuming each day? I don't see it anywhere on the dreamhost panel. I have some fairly complex matrix operations that run on my site and can last for up to 30 seconds per portfolio. It currently runs the operations sequentially but I would like to make them either multi-threaded or separate processes. Before I do that, though, I'd like to know what the current CPU usage is.

Any ideas?