Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

We've moved the forum!

Please use forum.silverstripe.org for any new questions (announcement).
The forum archive will stick around, but will be read only.

You can also use our Slack channel or StackOverflow to ask for help.
Check out our community overview for more options to contribute.

All other Modules /

Discuss all other Modules here.

Moderators: martimiz, Sean, Ed, biapar, Willr, Ingo, swaiba

Queued Jobs / RSS Connector Help


Go to End


17 Posts   3970 Views

Avatar
Garrett

Community Member, 245 Posts

4 May 2012 at 4:14pm

Well, on the jobs tab under queued it says the job is scheduled to run at 10:03PM, which is exactly 15 minutes after I ran the job manually by clicking on the gear icon. 10:03PM came and went and the queue still says the job is going to start at 10:03. It's like the crontab has nothing to do with it. And I don't know of any way to "reset" this job without either running it manually again or by deleting it and recreating it.

*/1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2
*/15 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=3

...is how it is now. Why is the first one 1 minute and the second one 15? I'm sorry Marcus, I'm just finding this confusing.

//Garrett

Avatar
Garrett

Community Member, 245 Posts

4 May 2012 at 4:27pm

Are you saying that the Queued Jobs UI will show the job based on what you have in your crontab file? I thought I had to create the job in the UI and that the crontab just executes it.

Avatar
Marcus

Administrator, 89 Posts

4 May 2012 at 4:50pm

Not at all. The jobs in the UI is just a list of items in the DB. The crontab entries are just there to run a script that will execute those jobs one by one. The separate crontab entries exist to process different queues (don't worry about the specifics of that, other than you should have the two crontab entries I mentioned in my previous post). Nothing you put in the crontab appears in the UI, or vice versa. Effectively, a job only gets executed when the "ProcessJobQueueTask" script is run, and this script is executed by cron every minute.

Is the "Job Type" field for the job 2 (it should be, as that's the default). That means that the job is in the queue '2'. If you then execute the following script from the command line, what output do you get?

php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2 

It should be something like

Running task 'ProcessJobQueueTask'...

[2012-05-04 14:44:37] Processing queue 2
[2012-05-04 14:44:37] No new jobs

(or it will mention processing the job if that's the case)

The other thing you can do is

php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask list=1 

which will list out all the jobs (regardless of when they're going to execute) currently in the job queues. The output will be something like

Running task 'ProcessJobQueueTask'...

[2012-05-04 14:48:05] Processing queue 2
[2012-05-04 14:48:05] Found 0 jobs for mode 1
[2012-05-04 14:48:05] Found 6 jobs for mode 2
[2012-05-04 14:48:05] Found 4 jobs for mode 3

Avatar
Garrett

Community Member, 245 Posts

5 May 2012 at 2:12am

Yes, Jobtype is 2.

Here is the output when I run it from the command line, when there are new items to import:

Running task 'ProcessJobQueueTask'...

[2012-05-04 09:40:59] Processing queue 2
[2012-05-04 09:40:59] Running Scheduled import from Horn Group Twitter Feed 

When there are no new items to import:

Running task 'ProcessJobQueueTask'...

[2012-05-04 09:51:34] Processing queue 2
[2012-05-04 09:51:34] No new jobs

...as you predicted. When I run this:

/Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask list=1

I get:

Running task 'ProcessJobQueueTask'...

[2012-05-04 09:55:54] Processing queue 2
[2012-05-04 09:55:54] Found 0 jobs for mode 1
[2012-05-04 09:55:54] Found 1 jobs for mode 2
[2012-05-04 09:55:54] Found 0 jobs for mode 3

But we're kind of going in circles here. Let me take you through exactly what I have done.

1.) On External Content tab, I created an RssContentSource using a valid RSS feed. This content source populates in the interface without issue.
2.) Went to the Import tab. Selected my target BlogHolder. Selected "Skip" for Duplicate Strategy. Selected 15 mins.for repeat interval.
3.) Clicked "Save"
4.) No new job is created in Jobs tab. This seems wrong.
5.) Clicked "Start Importing". (New items are imported successfully -- initially.)
6.) Job IS created in Jobs tab. Seems that you must perform an initial import to get the job into the database. "Start After" on the job I just created says datetime 15 minutes later than the "Added" datetime. This is obviously correct.
7.) The "Start After" time comes and goes, and nothing is ever imported again.
8.) Anytime I click the gear icon, the import fires.

My crontab file is exactly as you've prescribed. I can't figure out why this isn't running on its own.

//Garrett

Avatar
Marcus

Administrator, 89 Posts

7 May 2012 at 2:58pm

A guy at work had something come up that sounded very similar to this problem, it might be relevant?

What was happening was that the CLI version of PHP had a different timezone set to the apache module version of PHP. This meant that when the CLI script ran looking for jobs, the 'current' time as figured by PHP was several hours behind what it should have been (ie if local time was 11am, the cli php thought it was 3am), meaning the jobs weren't ever being picked up until 8 hours after they should have been.

A couple things to do to check (from the command line)

1) php -i | grep timezone (to see what timezone is set)
2) double check that `php -r "echo date('Y-m-d H:i:s');"` matches what `date` returns on the commandline
3) Make sure that `php -r "echo date('e');"` matches what the timezone set in (1) is

Separate to that, in your crontab, you can add >> ~/cron.log to log all the cron executions, to see what is actually happening each time the crontab is executed.

So you end up with

*/1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2 >> ~/silverstripe-cron.log

This will log the result of the job queue processor each time it is called by cron (or show you any errors that might be preventing it executing...).

Lastly, check your syslog file (/var/log/syslog or similar) to see if cron's reporting any errors

Avatar
Garrett

Community Member, 245 Posts

8 May 2012 at 3:08am

Edited: 08/05/2012 3:08am

Hi @Marcus, thanks again for your replies.

AUTOMATED IMPORT IS NOW WORKING!

Feel like a simpleton instead of a singleton. I took a closer look at my code, and I had some OnAfterWrite() logic in the ExternalBlogEntry class which, in combination with my duplicate strategy set to "Skip," was simply causing nothing to be imported (and therefore published) because there was nothing new to import! Haha. As soon as somebody tweeted again, a half an hour later, voila it was there on the site.

A note: the _ss_environment.php file is hugely important. Discovered after the above problem was fixed that, if you don't set the $_FILE_TO_URL_MAPPING, everything that gets published, on a site using the file system publisher anyway, will be a 404. Probably should add this to the documentation.

Regarding this config file, where should it live? I couldn't decide so I put it in BOTH my apache user's home directory AND in the webroot. The links are working now so I know one of them is working, but.... which one is it? Which _ss_environment.php can I remove? Don't want to confuse any future maintaining developers.

I can't thank you enough for your ongoing attention in this matter. I learned a lot about this very powerful suite of modules.

//Garrett

Avatar
Marcus

Administrator, 89 Posts

8 May 2012 at 10:48am

Edited: 08/05/2012 10:59am

Awesome! I should have actually had the info about logging the cron stuff earlier - that would have indicated that the import was going earlier :)

For the _ss_environment.php file, I normally put mine in the webroot with something like

<?php

global $_FILE_TO_URL_MAPPING;
$_FILE_TO_URL_MAPPING[dirname(__FILE__)] = 'http://site.url.com';

While you can put it in parent folders, if you're like me and have multiple projects in a subfolder, this ends up causing conflicts, so I always put the _ss_env file in the webroot.

Avatar
Garrett

Community Member, 245 Posts

9 May 2012 at 2:13am

Ok so SS will look in both places then....?