Shortcircuit workflows

[linkyndy]

Let's say we have the following, simple, workflow:

A -> B -> C

And Job A checks for a value, which if false, the whole workflow should be stopped immediately. Is this possible at the moment with Gush?

As an example, let's say we process a new user signup with the above workflow. Job A retrieves the newest user from the database. If the user has been already processed, we'd like to call the workflow complete; the subsequent steps are not necessary anymore, since the user has been already processed.

[pokonski]

Hey @linkyndy! Currently it's not possible but there was a suggestion about branching workflows in run time depending on their result, but I haven't had much time tinker with a solution for it.

Right now you could spawn a new Workflow inside A if conditions are met, but it's not a nice solution IMO

[linkyndy]

Thanks for your fast answer! Curious to see how this will develop.

[ace-subido]

EDIT: I just re-read OP's shortcircuit scneario, my original comment is irrelevant and doesn't help at all 😄

[ace-subido]

As an example, let's say we process a new user signup with the above workflow. Job A retrieves the newest user from the database. If the user has been already processed, we'd like to call the workflow complete; the subsequent steps are not necessary anymore, since the user has been already processed.

Hi @linkyndy I've just reimplemented something along those lines, this is how I went about it:

# example_workflow.rb
class ExampleWorkflow < Gush::Workflow

  def configure(user_id)
    run(FirstJob, params: { user_id: user_id })
    run(SecondJob, {
      params: { user_id: user_id },
      after: FirstJob,
    })
  end

end

# first_job.rb
class FirstJob < Gush::Job

  def perform
    user = User.find params[:user_id]

    if user.processed?
      self.fail!
      return
    end

    # continue code ...
  end

end

However this marks the flow.status as :failed. But it doesn't run the SecondJob once I call self.fail (I found this after snooping around the source).

@pokonski is this how .fail! could be used? I could put this in the README if you like, would be glad to contribute a PR! 😄

[pokonski]

I don't think this can be used, because fail! will stop the whole workflow 🤔

[ace-subido]

Yep! That's what I was aiming to achieve 😄the subsequent actions would be skipped, I think based on @linkyndy said:

If the user has been already processed, we'd like to call the workflow complete; the subsequent steps are not necessary anymore, since the user has been already processed.

I figured, that's what he/she is also aiming for.

there was a suggestion about branching workflows in run time depending on their result, but I haven't had much time tinker with a solution for it.

What would this look like? Could I take a stab at this, just pseudo-code and how the usage would look like? I'll think up of something later today, some sort of proposal for this.

[pokonski]

One of the good examples had an idea to allow providing two paths, like so:

    run SomeJob,
        before_success: MainJob,
        before_failure: AlternativeJob,

Though the naming is rather unfortunate because it suggest the MainJob will ran before SomeJob is succeesful, so needs better naming 💃

[linkyndy]

@ace-subido Marking the whole workflow as failed may be too harsh; @pokonski Indeed, it's not really clear from the DSL 😊

[pokonski]

But coming back to the problem from the original issue by @linkyndy, one way would be to introduce a skipped state in jobs. Similar to how Gitlab CI works. I would be fine with accepting such resolution

[ace-subido]

@pokonski so something like this:

class SkippedJob < Gush::Job
  def perform
    # marks the job as 'skipped', this would also do a 'return'
    self.skip! 
  end
end

We could also do something like skip_remaining! which skips all of the other jobs too, marking the entire workflow as :skipped.

class ExampleWorkflow < Gush::Workflow
  def configure(user_id)
    run SkipRemainingJob
    run SecondJob, after: SkipRemainingJob
  end
end

class SkipRemainingJob < Gush::Job
  def perform
    # marks the job as 'skipped' and all other jobs after, marks the workflow itself as 'skipped' too
    self.skip_remaining! 
  end
end

@pokonski What do you think?

[pokonski]

Yeah this sounds good! One clarification here:

which skips all of the other jobs too

I assume you mean skip all the jobs that were supposed to run after the job skip_remaining! is called from, right?

[ace-subido]

Yeah, probably cascade throughout job.outgoing in Gush::Worker#enqueue_outgoing_jobs and all it's descendants.

[linkyndy]

I don't really see any value in this job.skip!. If I want to "skip" the current job and go to the next one, I can simply return.

My initial question was related to cancelling the entire workflow from that moment on, and dealing somehow with the remaining jobs.

[ace-subido]

If that's the case, I'll add more to the PR. Something like: skip_remaining!, which would tag everything from that point as skipped

[rajaravivarma-r]

This particular feature looks very useful for an application we are developing. I would rather be very explicit and name the methods as skip_workflow! for skipping the whole workflow and skip_descendants! for skipping the jobs which should run after the current job.