Topic: ActiveRecord Serialize and Performance

I am trying to serialize a relatively large array (about 400 items, with each item being a hash) to a database record.  This operation never completes - I tried replacing the the object with a dummy small hash and it goes through fine.  Is there a size limit to the size of an objec that can be serialized?
This whole problem arose because I need to persist a large list over multiple requests (the list is the result of a very expensive operation on multiple XML files).  This object is way too big for session, and I though it would be easy to retrieve from an ActiveRecord serialized property.
As far as I've been digging, there is no other space (i.e. ServletContext or ApplicationContext in java world) to store an object like this?
Am I approaching the problem wrong?  Are there some best practices to follow here?

Thanks in advance,
Arun

Re: ActiveRecord Serialize and Performance

Are you serlializing 400 items to a single database record? How about setting it up so each item in the array is a single row in a new database table, and the hash is the columns. This sounds a little more organized. You'll probably still have a problem trying to save the 400 items though.

Do you know at what point the script hangs? Does it hang when saving to the database or during the serialization? Try splitting the array into groups of 50 or so and serialize/save those individually. You also might want to try saving to a file instead of the database.

Railscasts - Free Ruby on Rails Screencasts

Re: ActiveRecord Serialize and Performance

Ryan,

Thanks for the quick reply.  Actually I'm relatively new (couple of months) to Rails and have already built a couple of apps. > 5x productivity than when I was in Java (last 8 years with J2EE and Spring).
I've gotten around the problem by reducing the size of the list to some bare elements - probably not a permanent solution, but works for now.  That, however makes me believe its the way ruby/rails is serializing the object that makes the process slow.  I was also tinkering with Marshal.dump, but I kept getting problems with 'singleton errors.'  Would I be able to store the binary in a column and Marshal.load it back for the next request?  Show I create a new class to store this instead of using a Hash?

I only need keep these around for the duration of a session, so I thought creating a whole table hierarchy was overkill - plus as you said inserting 400+ records is not going to go quickly for each of these types of requests.

As a side - the one thing that is much slower is XML processing of large files in Ruby with REXML.  I'm at almost 4 minutes building a doc from an XML file ~ 1MB.  Any suggestions on how I can cut this down?

Thanks,
Arun