Jump to:

10993 Posts in 2727 Topics by 1818 members

All other Modules

SilverStripe Forums » All other Modules » Queued Jobs / RSS Connector Help

Discuss all other Modules here.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1 2 3
Go to End
Author Topic: 2022 Views
  • Garrett
    Avatar
    Community Member
    245 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Well, on the jobs tab under queued it says the job is scheduled to run at 10:03PM, which is exactly 15 minutes after I ran the job manually by clicking on the gear icon. 10:03PM came and went and the queue still says the job is going to start at 10:03. It's like the crontab has nothing to do with it. And I don't know of any way to "reset" this job without either running it manually again or by deleting it and recreating it.

    */1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2
    */15 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=3

    ...is how it is now. Why is the first one 1 minute and the second one 15? I'm sorry Marcus, I'm just finding this confusing.

    //Garrett

  • Garrett
    Avatar
    Community Member
    245 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Are you saying that the Queued Jobs UI will show the job based on what you have in your crontab file? I thought I had to create the job in the UI and that the crontab just executes it.

  • Marcus
    Avatar
    Administrator
    86 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Not at all. The jobs in the UI is just a list of items in the DB. The crontab entries are just there to run a script that will execute those jobs one by one. The separate crontab entries exist to process different queues (don't worry about the specifics of that, other than you should have the two crontab entries I mentioned in my previous post). Nothing you put in the crontab appears in the UI, or vice versa. Effectively, a job only gets executed when the "ProcessJobQueueTask" script is run, and this script is executed by cron every minute.

    Is the "Job Type" field for the job 2 (it should be, as that's the default). That means that the job is in the queue '2'. If you then execute the following script from the command line, what output do you get?

    php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2

    It should be something like

    Running task 'ProcessJobQueueTask'...

    [2012-05-04 14:44:37] Processing queue 2
    [2012-05-04 14:44:37] No new jobs

    (or it will mention processing the job if that's the case)

    The other thing you can do is

    php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask list=1

    which will list out all the jobs (regardless of when they're going to execute) currently in the job queues. The output will be something like

    Running task 'ProcessJobQueueTask'...

    [2012-05-04 14:48:05] Processing queue 2
    [2012-05-04 14:48:05] Found 0 jobs for mode 1
    [2012-05-04 14:48:05] Found 6 jobs for mode 2
    [2012-05-04 14:48:05] Found 4 jobs for mode 3

  • Garrett
    Avatar
    Community Member
    245 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Yes, Jobtype is 2.

    Here is the output when I run it from the command line, when there are new items to import:

    Running task 'ProcessJobQueueTask'...

    [2012-05-04 09:40:59] Processing queue 2
    [2012-05-04 09:40:59] Running Scheduled import from Horn Group Twitter Feed

    When there are no new items to import:

    Running task 'ProcessJobQueueTask'...

    [2012-05-04 09:51:34] Processing queue 2
    [2012-05-04 09:51:34] No new jobs

    ...as you predicted. When I run this:

    /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask list=1

    I get:

    Running task 'ProcessJobQueueTask'...

    [2012-05-04 09:55:54] Processing queue 2
    [2012-05-04 09:55:54] Found 0 jobs for mode 1
    [2012-05-04 09:55:54] Found 1 jobs for mode 2
    [2012-05-04 09:55:54] Found 0 jobs for mode 3

    But we're kind of going in circles here. Let me take you through exactly what I have done.

    1.) On External Content tab, I created an RssContentSource using a valid RSS feed. This content source populates in the interface without issue.
    2.) Went to the Import tab. Selected my target BlogHolder. Selected "Skip" for Duplicate Strategy. Selected 15 mins.for repeat interval.
    3.) Clicked "Save"
    4.) No new job is created in Jobs tab. This seems wrong.
    5.) Clicked "Start Importing". (New items are imported successfully -- initially.)
    6.) Job IS created in Jobs tab. Seems that you must perform an initial import to get the job into the database. "Start After" on the job I just created says datetime 15 minutes later than the "Added" datetime. This is obviously correct.
    7.) The "Start After" time comes and goes, and nothing is ever imported again.
    8.) Anytime I click the gear icon, the import fires.

    My crontab file is exactly as you've prescribed. I can't figure out why this isn't running on its own.

    //Garrett

  • Marcus
    Avatar
    Administrator
    86 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    A guy at work had something come up that sounded very similar to this problem, it might be relevant?

    What was happening was that the CLI version of PHP had a different timezone set to the apache module version of PHP. This meant that when the CLI script ran looking for jobs, the 'current' time as figured by PHP was several hours behind what it should have been (ie if local time was 11am, the cli php thought it was 3am), meaning the jobs weren't ever being picked up until 8 hours after they should have been.

    A couple things to do to check (from the command line)

    1) php -i | grep timezone (to see what timezone is set)
    2) double check that `php -r "echo date('Y-m-d H:i:s');"` matches what `date` returns on the commandline
    3) Make sure that `php -r "echo date('e');"` matches what the timezone set in (1) is

    Separate to that, in your crontab, you can add >> ~/cron.log to log all the cron executions, to see what is actually happening each time the crontab is executed.

    So you end up with

    */1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2 >> ~/silverstripe-cron.log

    This will log the result of the job queue processor each time it is called by cron (or show you any errors that might be preventing it executing...).

    Lastly, check your syslog file (/var/log/syslog or similar) to see if cron's reporting any errors

  • Garrett
    Avatar
    Community Member
    245 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Hi @Marcus, thanks again for your replies.

    AUTOMATED IMPORT IS NOW WORKING!

    Feel like a simpleton instead of a singleton. I took a closer look at my code, and I had some OnAfterWrite() logic in the ExternalBlogEntry class which, in combination with my duplicate strategy set to "Skip," was simply causing nothing to be imported (and therefore published) because there was nothing new to import! Haha. As soon as somebody tweeted again, a half an hour later, voila it was there on the site.

    A note: the _ss_environment.php file is hugely important. Discovered after the above problem was fixed that, if you don't set the $_FILE_TO_URL_MAPPING, everything that gets published, on a site using the file system publisher anyway, will be a 404. Probably should add this to the documentation.

    Regarding this config file, where should it live? I couldn't decide so I put it in BOTH my apache user's home directory AND in the webroot. The links are working now so I know one of them is working, but.... which one is it? Which _ss_environment.php can I remove? Don't want to confuse any future maintaining developers.

    I can't thank you enough for your ongoing attention in this matter. I learned a lot about this very powerful suite of modules.

    //Garrett

  • Marcus
    Avatar
    Administrator
    86 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Awesome! I should have actually had the info about logging the cron stuff earlier - that would have indicated that the import was going earlier

    For the _ss_environment.php file, I normally put mine in the webroot with something like

    <?php

    global $_FILE_TO_URL_MAPPING;
    $_FILE_TO_URL_MAPPING[dirname(__FILE__)] = 'http://site.url.com';

    While you can put it in parent folders, if you're like me and have multiple projects in a subfolder, this ends up causing conflicts, so I always put the _ss_env file in the webroot.

  • Garrett
    Avatar
    Community Member
    245 Posts

    Re: Queued Jobs / RSS Connector Help Link to this post

    Ok so SS will look in both places then....?

    2022 Views
Page: 1 2 3
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.