Java process killed

Anyone have any suggestions on how I can run my Java process without it being killed off after < 10 seconds?

The process basically downloads ~100 rss feeds (just the xml files), analyzes them, then puts the results into a mysql database. The process needs to run every 6 hours or so. It takes ~10 minutes to complete (on a non-dreamhost server where it doesn't get killed) but most of that time is spent waiting for the HTTP responses.

Here are the things I have already done to try to make my process less offensive. But it still gets killed:

1. Run with "nice -n+20 java ......"

2. Run java w/ low memory options (e.g. -Xmx8m).

3. Make the process single threaded (i.e. it checks the rss feeds serially so as not to use too much socket or cpu resources at one time. This obviously makes the process take a lot longer to complete but that doesn't matter to me).

4. Optimize my calls to MySQL (e.g. re-using connection objects, minimizing the number of insert statements, locking tables and disabling keys when doing many inserts).

5. I even tried to put a few calls to Thread.sleep(10) around within a few loops to further reduces CPU usage.

Nothing has helped. The process still gets killed almost immediately.

Anybody have other ideas to try? All guidance and suggestions greatly appreciated!

Sincerely,

Chris Eagan

In reply to:

---

being killed off after < 10 seconds?

---

Leads me to suspect a memory limit. You could ask DreamHost directly. Something that takes 10 minutes on a stand-alone server sounds like a heavy load. You might try running your steps separately, and see if any can finish.

Thanks for the response. I will contact support if I can't resolve this otherwise. It takes 10 minutes to run but the vast majority of that time is waiting for HTTP responses so I don't think CPU load is the problem.

if your application uses too much resource, it could be killed by DH. http://wiki.dreamhost.com/index.php/Persistent_Processes

I was just thinking and you could run this process outside of your host and update the DB remotely.

It's possible to have a machine outside of DH connect to the db server. So, just set up an "at" task on a pc in your office or home do do this apparently heavy process.

It does blow out the idea of having everything in one place, but it can at least complete reliably then.

Wholly

Thanks for the responses.

I also contacted support and they confirmed my processes were killed because they were over the memory limit (interestingly they don't disclose what that limit it). I tried for several days to tune the java process with its various -XX: and -X options but I couldn't get any combination to work (see http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp) I also installed Java 6 and ran with that but that didn't work either. I am still baffled because the process runs in < 32 MB on my home computer.

So I ended up porting 1 process to ruby and running another process on my home computer, while still connecting to my db at dreamhost as Wholly suggested.

Thanks again!

Chris Eagan

i'm not sure about java. but some plugins may take too much resource. but we do not like to disable any useful plugins right.

This is the only solution. We can't do anything but to understand we are in a shared server