Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

Audit Cache/Store (ie Replay and Recovery)

#194

Been considering how to do error recovery. Unwinding a workflow is now possible, in many cases, since Joins can store state as full-fledged objects (although if joins are blocks this isn't true... procs don't serialize). However, when you unwind a partially-run workflow you can lose information that is within some types of joins, like a theoretical counter, or in the current Join running in no-stack mode.

Once you unwind back to app.run, you will have audits of all the results. That suggests a replay is possible but you need something that a given task can query for cached results. Then, in replay mode, you load a clean workflow, load the audits into the cache, and run the tasks so they use the cached results when possible.

This should quickly re-create the state at the time of crash, without re-running upstream tasks. Notes:

  • Tasks cannot have side-effect in this case. Result is the only meaningful output of executing with inputs.
  • A clean, new workflow must be made. This allows joins to have 'side effects' like counters and such.

Note after unwinding you will also have (via join.inputs/join.outputs) a full representation of the workflow, but this may be 'polluted' in the sense that there may be state changes. This is useful for inspection during debugging, and potentially for restart.

For restart:

  • Joins would have to track their state explicitly, ie with @_results and @index so that a dump has a handle on results and where to restart.

Reported by bahuvrihi · March 22nd, 2009 @ 09:47 AM

State: new
Milestone: none
Assigned to: nobody

Activity

  1. bahuvrihi
    bahuvrihi

    When an error occurs:

    • set termination/raise termination error
    • mark the failing task(s)
    • as you unwind, cache the results for each audit
    • requeue the lead tasks

    Then during debuggin'

    • the failed task(s) located and manipulated
    • when the app is restarted the tasks run with cached results, allowing the join state to be recreated

    A similar thing can be accomplished if results are cached along the way, for example to an external data store. How caching/auditing occurs is a separate issue with speed vs memory issues.

    March 23rd, 2009 @ 10:35 AM

  2. bahuvrihi
    bahuvrihi

    One issue is that this may run into a duplication problem if a join has enqued tasks. If:

      a -- b -- c --[0][1,2]q

    And task c fails, I guess this is actually ok because by the time c runs, b will have already been taken off the stack.

    Not ok, however if:

      a -- b --:q c -- d --[0][1,3]

    Because c will be enqued after b, then d fails. Replay will enque d c again, leading to duplication. If b doesn't run at all, however, then this model is ok.

    So, this would have to be a dispatch-level cache/replay and not a middleware thing. Don't dispatch if completed.

    June 1st, 2009 @ 12:14 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.