App Serialization
Reported by bahuvrihi | March 21st, 2009 @ 11:02 AM
Serialization of App is essential for a dump/recover cycle. YAML is the obvious serialization format. Issues:
- not all objects serialize (ex Thread, Proc, IO)
- apps are heavily nested and self-referential
- sometimes complex objects are hash keys, and this can lead to YAML failing on load
A prime example of the last issue is wrt Aggregator. Currently aggregators store tasks as keys (most commonly) and this breaks the dump/load.
A number of monitors exist (Aggregator, Logger, ExecutableQueue, App) and these won't dump properly if they have mon_current thread set (and likely if they have any in the waiting queue). Special behavior is needed to circumvent this.
A number of potential blocks like on_complete exist. These need to raise some sort of warning, and they need circumvention.
== Plan of Attack ==
Currently (75d383388334f55bfa968ec3df834216ad94369d) I'm using a combination of to_yaml and yaml_initialize overrides. Perhaps there's a way to do it by extending the YAML emitter.
Comments and changes to this ticket
-
bahuvrihi March 21st, 2009 @ 11:36 AM
Notes wrt hacking YAML.
require 'yaml' require 'test/unit' class A attr_reader :key, :hash, :array def initialize(value='value') @key = value @hash = {:key => value} @array = [:key, value] end end class B attr_reader :a def initialize(a=A.new) @a = a end end class CustomEmitter < YAML::Syck::Emitter module OutExtension def map( taguri, to_yaml_style) super do |out| out.extend MapExtension yield(out) end end def seq( taguri, to_yaml_style) super do |out| out.extend SeqExtension yield(out) end end end module MapExtension def add( k, v) puts [k,v].inspect super end end module SeqExtension def add(v) v.inspect super end end def initialize super @out.extend OutExtension end end class TestYAML < Test::Unit::TestCase def test_yaml_emitter_can_be_extended a = A.new b = B.new(a) emitter = CustomEmitter.new.set_resolver( YAML.resolver ) dump = b.to_yaml(emitter) d = YAML.load(dump) assert_equal 'value', d.a.key assert_equal({:key => 'value'}, d.a.hash) assert_equal [:key, 'value'], d.a.array end end
-
bahuvrihi March 21st, 2009 @ 01:23 PM
Made a nasty hack using the above system before discovering the formatting error that syck produces.
-
bahuvrihi March 22nd, 2009 @ 09:24 AM
- State changed from new to resolved
- Assigned user set to bahuvrihi
The current solution is acceptable. Pure-YAML serialization that removes any Proc or Threads. This allows the YAML to be re-loaded and inspected, even reconstructed for re-use.
It will have issues because the dump/reload is not totally faithful, but as an 80% I think it goes.
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A framework for making configurable, file-based tasks and workflows.