AW: Big, BIG databases.

[Bauer, Georg]

Hi!

Every now and again I get events database corruption. Deleting it
fixes everything up, but it's a worry.

Yep. Metakit - especially in non-standard environments - sometimes behaves
quite worrying. I think it would be time to start refactoring that throws
out Metakit and pulls in something more stable. I'd gladly accept patches
;-)

My weblog.data file is currentyly 119,029,070 bytes in size. That
strikes me as just crazy: there's only a couple thousand posts, and
they're not 64K each. I've tried the alleged Metakit database
verification tool, and can't get it to visibly do anything. So, I'm
kinda worried.

Metakit manages the free space itself. It should do the right thing, but it
does produce weird database sizes sometimes. Did I say that it would be nice
to throw out metakit? ;-)

bye, Georg

[Jeremy Bowers]

Post by Bauer, Georg
Yep. Metakit - especially in non-standard environments - sometimes behaves
quite worrying. I think it would be time to start refactoring that throws
out Metakit and pulls in something more stable. I'd gladly accept patches
;-)

I've started hacking on this on a couple of occasions, but it is pretty
hard for me as an "outsider".

I really find myself itching for a two part split, where, taking the
Weblog tool as a for-instance, there is a Post class, which can be
loaded in several ways, and some sort of storage backing abstraction
that would do most of what metakit does now, but it's too large a
project for me to undertake on my own. (Particularly as it is really the
WeblogTool I want to hack on and that's really tightly integrated.)

I keep going to do things and running up against the fact that the
rendering and storage are so interleaved, and so far I haven't been
successful in disentangling it yet, because I haven't come up with a
need that justifies the time.

I've got some time coming up over the next few weeks. Maybe I'll take
another crack at it, especially as my "don't complain unless you're
willing to do something about it" sense is nagging me. It occurs to me
as I type this that it might be easier to start out with a Post object
and then work on abstracting the storage, rather than the other way
around as I tried before.

I've got some things to attend to fir$t, but I'll hit you up for a
subversion account sometime over the next couple of weeks, Georg. I'll
start in a branch, of course, but it'll be easier for me if it's in
source control the whole time. And I'll try to remember to stay 2.2 only
:-)

[Georg Bauer]

Hi!

| I've got some things to attend to fir$t, but I'll hit you up for a
| subversion account sometime over the next couple of weeks, Georg. I'll
| start in a branch, of course, but it'll be easier for me if it's in
| source control the whole time.

Sure, no problem, just drop me a note.

| And I'll try to remember to stay 2.2 only
| :-)

Oh, with the imminent release of Debian Sarge (that carries 2.3), I
think we can just use 2.3 as a starting point for PyDS 0.8 (and I think
it should at least become 0.8 if we change the underlying database).

Ok, "imminent release" with Debian can still mean 1-2 years ;-)

bye, Georg

[Georg Bauer]

Hi!

| as I type this that it might be easier to start out with a Post object
| and then work on abstracting the storage, rather than the other way
| around as I tried before.

My plans on this up to now were to use SQLObject and SQLite as a
replacement of Metakit. Of course it would require to change the overall
structure of PyDS, as Metakit isn't abstracted away currently. Maybe
some base ideas on PyDS:

- - tools should be mostly independend on one another, at least highlevel
tools like wiki, weblog, pictures etc. - this includes their databases,
they should have their own "tablespace" to work with.

- - tools should use stuff from other tools via the getTool() call and
then accessing member functions of the returned instance. They should
never directly access database stuff from other tools but allways go
through member functions.

- - tools should provide hooks for other tools to "hook into" processing
(like some tools hooking into the rendertool or some others hooking into
the weblog). This should include database stuff: don't try to do one big
database schema to model the post class, just do a rather abstract and
barebones schema and add stuff by using the current way of tools hooking
into other tools and managing their own tables.

That's the way that PyDS is currently built and I think sticking to this
way of doing it would make the database layer switch easier. Maybe it
really would just be a good idea to use SQLObject exactly where
currently Metakit is used and look what needs to be changed. I didn't
use the subtable stuff from Metakit too often, so it's mostly simple
tables managed as a list of objects in Metakit - and that should match
quite fine to a list of objects managed by SQLObject.

We can abstract away the SQL stuff mostly by using SQLObject - what
would give us the nice sideeffect of not having to concentrate on one
database too much and would allow to start with SQLite but allow people
to "upgrade" to PostgreSQL if they have needs that go beyond SQLite.

bye, Georg