testing cpan with a local tarball:
  TARBALL=/media/edgy/home/ski/mysrc/XML-Comma-1.993.tgz
  mkdir /root/.cpan/build
  tar xfz $TARBALL
  cd XML-Comma-*
  cpan .

when installing, set DBI timeout lower?

calling ->erase() when you don't have perms to delete the store doesn't 
seem to do the right thing... (no error - try with a TravelPictureLocal) 
- but perhaps that is a reflection of what dave mentions below:

dave reports index hooks die silently without throwing an error:
(01:57:13 PM) davethemadx: Hmm, btw, if an index hook dies... it 
apparently does so silently.
(01:58:10 PM) xbrianskix: really.....
(01:58:30 PM) xbrianskix: that actually might be related to the problem 
we were having with pg and textsearches

a terser, and hopefully quicker to parse data format not based on XML?
  http://en.wikipedia.org/wiki/YAML
  http://en.wikipedia.org/wiki/JSON
  - or perhaps use XML with attributes, as
    <element name="foo" />
  is a hell of a lot terser and easier to read than
    <element><name>foo</name></element>
  the selection of storage format would come in a layer right next
  to storage/location... or perhaps look at how the gzip compression
  is implemented and do something like that?

bah! sync_deferred_textsearches is broken with pg! it's worked-around 
for now, but that stinks...

#TODO: allow nice way to use of $dbh->quote() on collections and
#	textsearches (or is this done behind the scenes already? if so add 
#	mention of that to the docs)

#TODO: access to convenience methods like AllAfrica::Travel::Post::get_uri
#  and blurb_text... for now we have to call
#	AllAfrica::Travel::Post::get_uri($vdoc), instead of $vdoc->get_uri;

#it'd also be nice to say things like the below, instead of
#having a method with the same name as:
#     <textsearch>
#         <name>full_text</name>
#         <code><![CDATA[ sub {
#            return $_[0]->title."\n".
#                   join("\n", $_[0]->tags())."\n".
#                   $_[0]->body_text;
#         } ]]></code>
#      </textsearch>

shortcut methods skip set_hook (and probably read_hook) !
	(at least with a blob_element - try $doc->original("foo")
	 on a travel...Picture::Local doc)

<location> for BlobElements
	this deals with elements on a different FS in a "smart" way,
		and lets you move blobs out from the same hierarchy
		as the xml docs, so you can grep -r there easily
	another way to think of this is as the inverse of
		the <extension> field (ie, <prefix>, akin to
		PREFIX=/some/path make install)

integrate Versionable.pm from travel code into base?

parameterized where_clauses and such. single quotes are always a pain in 
the rear here...

default mode of destination dir should be $owner:{www-data|web}
	and mode 664 on comma.log

introspection cleanup:
  add to docs that we deprecate from top-level of DocDef:
    <required>, <plural>, <is_i*_from_hash>, these should  be in
    a properties block now:
      <properties>
        <required>qw ( foo bar )</required>
        <plural>qw ( foo )</plural>
        <doc_key>qw ( bar )</doc_key>
      </properties>
  add support for this syntax:
    <element><name>foo</name>
      <properties>plural,required</properties></element>
    <element><name>bar</name>
      <properties>doc_key,required</properties></element>
  allow support for persistent setting of this in a UI system
    or be somewhere else in a UI-settable system
    where can we store this adjunct data?
    perhaps this should wait until a file can be both a def and doc?
  also clean up the "4 different ways" problem, standardize on the
      last (while retaining compatibility):
    $def->is_{required|plural|is_ignore_for_hash}
    $el->def->is_{nested|blob}
    $macros{enum|boolean|range|timestamp|timestamp_created|timestamp_last_modified}
    $def->has_property($required_plural_blob_nested_enum_or_etc, $el_name)
  also rewrite legacy functions like is_required as calls to has_property

insights from introspection:
1 - a bug: you can load a <TravelPost>...</TravelPost> with doc on disk
    that has <tpt>...</tpt> - add a sanity check there.
2 - a possible bug: the below dies if we don't have a listing_fields (?)
  <nested_element>
      <name>listing_fields</name>
      <element><name>foo</name><macro>something_that_installs_validate_hook</macro></element>
      <required>qw ( foo )</required>
  </nested_element>

comma/postgres bug:
  need to use a superuser for now.
  later do what it suggests and change COPY to \copy ?

#complete test:
dropdb -U postgres comma ; createdb -U postgres -O comma comma; \
make distclean ; perl Makefile.PL && make && make test && sudo make install

TODO: make sure all new API stuff is in doc/guide.html and Bootstrap.pm
TODO: test-case for $blob_el->append...

things to add to guide.html:
	everything in comma-2.html, plus probably some other stuff...

add to comma-2.html/Changelog:
	some sort of smart dump of svn commits to chronicle

2.1:
- apply & fix the stuff in weak/ for ->parent, ->top, etc.
- get more info on copyright bounding start dates from comma 1.21
- there is no way to get the size of the store from the index... perhaps
	this is a feature? ... but it would be nice to have an 
	$index->total_count or so
- change default version of read_doc() to return a
  virtualdoc when called off of an index iterator.
- THINK: is it possible & safe to do the above optimization
  when one calls retrieve_doc?
- make an Iteratable.pm or so which Storage::Iterator and
  Index::Iterator inherit from ( we probably can't use it for much
  but it will look nicer that way ;) )

$sit->length() vs. $idx->count() ... resolve this inconsisency if poss.
consider iterator_refresh for storage iterators?
$sit->length(max => $n) would be nice for gigantic stores...

dave ideas:
  2 - (not sure if this is something we should do via comma, but to 
    fight the "SQL resistance" at MMFA):
  this common construct is reaaaaaly slow:
        for ( @{ $doc->issue_ptr } ) {
           push @issues, $index->single( where_clause => "doc_id = $_" )
        }
        foreach my $issue (@issues) {
          ...
        }
  you hit the DB N times. you can hit it just once by building a gigantic
  where clause, e.g.
        my $wc = join ("OR", map { "doc_id = $_" } @{ $doc->issue_ptr });
        my $it = $index->iterator(where_clause => $wc);
        while(++$it) { 
          ...
        }
  or even better, using an IN statement:
        my $wc = "doc_id IN (".join (",", @{ $doc->issue_ptr }).")";
        my $it = $index->iterator(where_clause => $wc);
        while(++$it) { 
          ...
        }


Reported by tim, 2/15/2007

This doesn't work... Probably to fix this after we get the weakref
stuff in and right.
$ perl -MXML::Comma -e 'use strict ;
	my $def = XML::Comma::Def->read( name=>"AllAfrica_Publisher" ); 
	foreach my $el ( $def->def_sub_elements() ) {
		my $tag = $el->name(); 
		print $tag . "\n" if $el->element_is_plural($tag);
	}'

This does:
$ perl -MXML::Comma -e 'use strict ;
	my $def = XML::Comma::Def->read( name=>"AllAfrica_Publisher" );
	foreach my $el ( $def->def_sub_elements() ) {
		my $tag = $el->name();
		print $tag . "\n" if $def->is_plural($tag);
	}'

This is an API bug. THE other is_* have this problem as well.


Test::More stuff:
- standardize/clean up the order of "use" statements, etc.
- tests that only nominally use Test::More (numbered tests via ok): 
    order.t, indexing.t, parser.t, storage.t, validation.t


with postgres, we get lots of spurious warnings starting "NOTICE"
	these can be ignored (i think/hope)
	the Pg code in general needs a general cleanup too.

pg for dummies:
  To get postgres up and running on my machine, for use with comma, I
  just did an ubuntu install, then:
    apt-get install postgresql-8.1 postgresql-server-dev-8.1
    sudo cpan -i DBD::Pg
    sudo bash
     $EDITOR /etc/postgresql/8.1/main/pg_hba.conf
     ## change:
        local  all  postgres  ident sameuser
        local   all         all                               ident sameuser
     ## to:
        local   all         postgres                          ident sameuser
        local all all password
        #local   all         all                               ident sameuser
     /etc/init.d/postgresql-8.1 restart
     su postgres
      createuser --pwpompt comma #for now make a superuser
        #there is a bug we need to fix with COPY commands with Pg
        #relevant tests are in t/indexing.t
      createdb -O comma comma
      exit #shell as postgres user
     psql -U comma comma #make sure you can connect
     psql -U postgres #make sure you can connect
     #edit Configuration.pm accordingly

- element-specific locking for when whole body lock is unnec. and too
  heavy weight?
- maybe use ExtUtils::MakeMaker::prompt for automated testers
- it'd be nice to have "iterators" on nested plural elements
- test cases for $blob_el->{append|set}() combinations
- make $blob_el->set() more efficient by nuking the fcopy and
  instead keep the appendage and the original data seperate
  use _set_was_called for this
- audit for SQL queries => parameterized queries, e.g.:
  my $sth = $dbh->prepare_cached ( "SELECT bla FROM foo WHERE key = \"$key\"");
  $sth->execute();
becomes:
  my $sth = $dbh->prepare_cached ( "SELECT bla FROM foo WHERE key = ? ");
  $sth->execute( $key );
this is MUCH more secure and less error prone. it appears most of the
SQL code we have (if not all) already does this, but since I just 
discovered it, I want to make sure it ALL does...

- t/indexing.t sometimes fails because of an incomplete previous run, or
	cruft in the database. make it more robust.
	### step 5. - STORE_ERROR/POST_STORE_ERROR/DOC_INDEX_ERROR (0001) 
	think: table to drop, delete entryy in index_tables, delete any comma_locks (dangerous)

- it'd be nice if we could do some SQL-ish things with storage iterators... but best to
  read up on the progress (or is that lack of progress) with SQLite first...

