|Every time I almost get around to working on an arc based web service, one of the challenges I end up facing is, 'how do I store my data in an arc-friendly, scalable fashion?' What have you guys been using for your services, and what would you like in a database system if I was to work on one?|
So far, most of my data models have been simple obj's with an id field, indexed in multiple ways by adding them to the relevant index tables. Like "users" and "items", with subsets of items and users as necessary, using the push-on-update semantics. This seems fairly scalable, and relatively directly mappable to some of the newer nosql key-value storage systems, but I'm not sure what would be best for arc.
Most of what I want feature wise is just the ability to save and load items by key, generate the indexes, and be able to persist the links between objects. Ideally, a user could have references to items, and in the live instance they would just be pointers in a list, but when persisting have them convert to the guid and back again. Has anyone made anything like that, or been using a similar model?
I also wanted something where I could version the objects, though that feature is proving more challenging and may be unnecessary. Ideally, this would allow for an immutable data store, and I could load the state of the entire system by date, or a single object by incremental version. That may end up being best implemented by making a Cassandra (or similar) interface and storing each object as a key in a CF and each column be the date of the change.
Thoughts on the data model and versioning concept? Should this be implemented directly in arc or just as a mapper over an existing nosql system? I had started out by using git as the database, but supposedly the index system that git uses doesn't scale very well to large numbers of objects, which I'm expecting to have.