Cron and wget....

Posted: Mon May 29, 2006 6:21 am

I know this should be easy....but I can not find an example of it anywhere. I read through the Cron man pages but it was all advanced stuff. What I want to do is fairly simple.

My wget command works fine....
wget http://server/data/dir/text.txt

What I want is it to save to the correct location.

In this case httpdocs/cache

These are all simple text files.

While I am bothering everyone...I might as well ask how I could string these all together? Lets say I have 10 text files all in the same remote dir and save them to the same cache dir.....ie text1.txt, text2.txt etc....

If you know of a working example of this...I am happy to go read....I rooted around and could not find an example that included the change of dirs.

Thank You for your time!!

Dawg

registered · Site Admin Joined: Jun 04, 2004 Posts: 6437

Maybe it would help if you could explain what you're trying to do functionally, rather than technically.

Posted: Mon May 29, 2006 3:21 pm

There are Text files on a remote Nation Weather Service server.....I would like Cron to run every 30 mins and go get those text files and then save them to my httpdocs/cache file. If it is called 1234.txt on the remote server, I would want it called 1234.txt in the cache file.

The problem is when I start doing LONG forecasts with lots of images it really bogs down and takes a LONG time to display. So wanted to switch to using a server Cron job and do it in the background.

Kguske, If you would like to see the real life examples of what I am talking about just shoot me a Pm and I will link you to some examples.

Dawg

Posted: Mon May 29, 2006 3:39 pm

Have you tried a PHP approach vs. using wget? My guess is that the wget works differently when used by cron.

Posted: Mon May 29, 2006 4:06 pm

K,
That is how I am doing it now. One set of scripts I have runs on Nuke Cron. It runs every 30 mins....checks the time stamp......if the time stamp is older than X it goes and opens the remote text files and then saves them to my cache dir. the second set does the same thing but does not run on Nuke Cron. When you enter the MOD it does all the above. These RUN real SLOW. It is a combination of data and images (graphs) and it takes for ever for these things to load.

I figured since I own the server the would do it at the server level (both sets) and then just read the data at the script level.

I can CD into the cashe dir and then run the wget request correctly but for the life of me I can get it to do it through Cron. The CD does not work so I can't get the files into thee correct DIR.

Dawg

registered · Posted: Tue May 30, 2006 6:21 am

Quote:

I can CD into the cashe dir and then run the wget request correctly but for the life of me I can get it to do it through Cron. The CD does not work so I can't get the files into thee correct DIR.

Is cron running as "root" or some other user? I wonder if that user cannot cd into your directory. You may need to mess with the permissions. However, another thing crossed my mind also: are you able to execute a shell script from cron? Since you "own" the server, I would think so. Could you not please all the necessary commands into one shell script and then have cron call it instead?

Posted: Tue May 30, 2006 9:32 pm

M,
I had not thought about just writing a shell script for it. Hmmmm.

On the cron....I would bet it is running as user. I think your idea of a shell is the way to do it. There are over 100 of these things we pull every 30 mins or so. (According to the update times of the remote files) It is going to be a BIG project to change it all over but should be fun none the less.

Thank You for your thoughts!

Dawg

Posted: Wed May 31, 2006 2:22 pm

u r welcome!