#193 ✓resolved
bahuvrihi

App Serialization

Reported by bahuvrihi | March 21st, 2009 @ 11:02 AM

Serialization of App is essential for a dump/recover cycle. YAML is the obvious serialization format. Issues:

  • not all objects serialize (ex Thread, Proc, IO)
  • apps are heavily nested and self-referential
  • sometimes complex objects are hash keys, and this can lead to YAML failing on load

A prime example of the last issue is wrt Aggregator. Currently aggregators store tasks as keys (most commonly) and this breaks the dump/load.

A number of monitors exist (Aggregator, Logger, ExecutableQueue, App) and these won't dump properly if they have mon_current thread set (and likely if they have any in the waiting queue). Special behavior is needed to circumvent this.

A number of potential blocks like on_complete exist. These need to raise some sort of warning, and they need circumvention.

== Plan of Attack ==

Currently (75d383388334f55bfa968ec3df834216ad94369d) I'm using a combination of to_yaml and yaml_initialize overrides. Perhaps there's a way to do it by extending the YAML emitter.

Comments and changes to this ticket

  • bahuvrihi

    bahuvrihi March 21st, 2009 @ 11:36 AM

    Notes wrt hacking YAML.

    
    require 'yaml'
    require 'test/unit'
    
    class A
      attr_reader :key, :hash, :array
      def initialize(value='value')
        @key = value
        @hash = {:key => value}
        @array = [:key, value]
      end
    end
    
    class B
      attr_reader :a
      def initialize(a=A.new)
        @a = a
      end
    end
    
    
    class CustomEmitter < YAML::Syck::Emitter
      module OutExtension
        def map( taguri, to_yaml_style)
          super do |out|
            out.extend MapExtension
            yield(out)
          end
        end
        
        def seq( taguri, to_yaml_style)
          super do |out|
            out.extend SeqExtension
            yield(out)
          end
        end
      end
      
      module MapExtension
        def add( k, v)
          puts [k,v].inspect
          super
        end
      end
      
      module SeqExtension
        def add(v)
          v.inspect
          super
        end
      end
      
      def initialize
        super
        @out.extend OutExtension
      end
    end
    
    class TestYAML < Test::Unit::TestCase
      
      def test_yaml_emitter_can_be_extended
        a = A.new
        b = B.new(a)
        
        emitter = CustomEmitter.new.set_resolver( YAML.resolver )
        dump = b.to_yaml(emitter)
        
        d = YAML.load(dump)
        assert_equal 'value', d.a.key
        assert_equal({:key => 'value'}, d.a.hash)
        assert_equal [:key, 'value'], d.a.array
      end
    end
    
  • bahuvrihi

    bahuvrihi March 21st, 2009 @ 01:23 PM

    Made a nasty hack using the above system before discovering the formatting error that syck produces.

  • bahuvrihi

    bahuvrihi March 22nd, 2009 @ 09:24 AM

    • State changed from “new” to “resolved”
    • Assigned user set to “bahuvrihi”

    The current solution is acceptable. Pure-YAML serialization that removes any Proc or Threads. This allows the YAML to be re-loaded and inspected, even reconstructed for re-use.

    It will have issues because the dump/reload is not totally faithful, but as an 80% I think it goes.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

A framework for making configurable, file-based tasks and workflows.

People watching this ticket

Attachments

Pages