- make install hack:
  in misc/install-extras.pl, exec perl -e 'use XML::Comma' to
  force compilation of Inline::C stuff for Parser?

- document select_count in guide.html, etc.

- no doc_id on new (unstored) docs makes creating docs that point at one 
  another hard

- an optional argument to to_array() indicating you want the whole
  enchilada as if there was an iterator_refresh first. also an optional
  argument indicating you need the index pos reset after your call.

- do a basic SQL connect check, and if it fails, skip all the rest of
  the indexing tests, marking them failed. see other cpan modules for
  how to do this (i remember seeing things like "skipped: various reasons")

doc_is_new is confusing now.
since a new doc can be set an arbitrary number of times w/o need of 
get_lock or keep_open, in some sense a new doc is always new until you 
load it from disk. currently doc_is_new ceases to be new as soon as you 
associate the doc with some storage, which is what brian wants, but you 
can write code like this (from multi_store_doctype.t)
  my $doc = XML::Comma::Doc->new( type => $def_one_name );
  $doc->element( "foo" )->set( "foo_$_" );
  $doc->bar( "bar_$_" );
  $doc->store( store => "one" ); # updates indices 'one' and 'all'
  $doc->store( store => "two" ); # updates indices 'two' and 'all'
resolve this ambiguity.

fix Index->erase() to clear cached data as necessary
then implement rebuild(drop_first => 1);

alter table statements fail with postgres for when we change a def's 
sql_type or so:
  DBD::Pg::st execute failed: ERROR:  syntax error at or near "MODIFY"
  LINE 3:               MODIFY type VARCHAR(255)
                        ^
   [for Statement "
            ALTER TABLE TravelPo_0001
                MODIFY type VARCHAR(255)
        "] at /usr/local/share/perl/5.8.8/XML/Comma/SQL/Base.pm line 293.

would it be worth having some SQL-backend-neutral names for sql_types?
e.g. pg:
  varchar text name bytea bool date time - etc.
vs mysql:
  VARCHAR CHAR INT TINYINT - etc.
more here: 
  http://www.ibm.com/developerworks/db2/library/techarticle/dm-0606khatri/

trying to rebuild an index_only store shouldn't work, but it should give 
a better error message than this:
tperl -e 'XML::Comma::Def->Shorten->get_index("main")->rebuild(verbose => 1)'
  beginning re-index from store main...
  error while rebuilding: UNKNOWN_ACTION -- no method or element 
  'extension' found in 'DocumentDefinition:store' at -e line 1

in travel::webuser::get_user_by_email, change single_read to single
and you uncover what appears to be a bug. I think this is related to the
++$it problem?

if i try to do <field><name>foo</name></field>, and foo is a plural
  element, i don't get indexed - nor do i get any error output!
  when is the right time to abort?
  also, is it possible that we should auto-detect by plurality
  if we want a field or collection, and have both mean the
  same thing? (perhaps not, the programmer needs to know the
  difference to create sql to actually query the thing)

shoot, we have to rejigger te install process to have a seperate 
dbconfig step sooner than later, as when we return from cpan currently,
we make and make test, and die slowly w/o that info

testing cpan with a local tarball:
  TARBALL=/media/edgy/home/ski/mysrc/XML-Comma-1.993.tgz
  mkdir /root/.cpan/build
  tar xfz $TARBALL
  cd XML-Comma-*
  cpan .

when installing, set DBI timeout lower?

convert all uses of bareword file handles to proper my $foo style
  (that is, if this is compat with 5.6)

calling ->erase() when you don't have perms to delete the store doesn't 
seem to do the right thing... (no error - try with a TravelPictureLocal)
  (just a reflection of the index errors not propagated problem we had?)

a terser, and hopefully quicker to parse data format not based on XML?
  http://en.wikipedia.org/wiki/YAML
  http://en.wikipedia.org/wiki/JSON
  - or perhaps use XML with attributes, as
    <element name="foo" />
  is a hell of a lot terser and easier to read than
    <element><name>foo</name></element>
  the selection of storage format would come in a layer right next
  to storage/location... or perhaps look at how the gzip compression
  is implemented and do something like that?

bah! sync_deferred_textsearches is broken with pg! it's worked-around 
for now, but that stinks...

#TODO: allow nice way to use of $dbh->quote() on collections and
#	textsearches (or is this done behind the scenes already? if so add 
#	mention of that to the docs)

#TODO: access to convenience methods like AllAfrica::Travel::Post::get_uri
#  and blurb_text... for now we have to call
#	AllAfrica::Travel::Post::get_uri($vdoc), instead of $vdoc->get_uri;

#it'd also be nice to say things like the below, instead of
#having a method with the same name as:
#     <textsearch>
#         <name>full_text</name>
#         <code><![CDATA[ sub {
#            return $_[0]->title."\n".
#                   join("\n", $_[0]->tags())."\n".
#                   $_[0]->body_text;
#         } ]]></code>
#      </textsearch>

shortcut methods skip set_hook (and probably read_hook) !
  (at least with a blob_element - try $doc->original("foo")
   on a travel...Picture::Local doc)

<location> for BlobElements
  this deals with elements on a different FS in a "smart" way,
    and lets you move blobs out from the same hierarchy
    as the xml docs, so you can grep -r there easily
  another way to think of this is as the inverse of
    the <extension> field (ie, <prefix>, akin to
    PREFIX=/some/path make install)

integrate Versionable.pm from travel code into base?

parameterized where_clauses and such. single quotes are always a pain in 
the rear here...

default mode of destination dir should be $owner:{www-data|web}
  and mode 664 on comma.log

introspection cleanup:
  add to docs that we deprecate from top-level of DocDef:
    <required>, <plural>, <is_i*_from_hash>, these should  be in
    a properties block now:
      <properties>
        <required>qw ( foo bar )</required>
        <plural>qw ( foo )</plural>
        <doc_key>qw ( bar )</doc_key>
      </properties>
  add support for this syntax:
    <element><name>foo</name>
      <properties>plural,required</properties></element>
    <element><name>bar</name>
      <properties>doc_key,required</properties></element>
  allow support for persistent setting of this in a UI system
    or be somewhere else in a UI-settable system
    where can we store this adjunct data?
    perhaps this should wait until a file can be both a def and doc?
  also clean up the "4 different ways" problem, standardize on the
      last (while retaining compatibility):
    $def->is_{required|plural|is_ignore_for_hash}
    $el->def->is_{nested|blob}
    $macros{enum|boolean|range|timestamp|timestamp_created|timestamp_last_modified}
    $def->has_property($required_plural_blob_nested_enum_or_etc, $el_name)
  also rewrite legacy functions like is_required as calls to has_property

insights from introspection:
1 - a bug: you can load a <TravelPost>...</TravelPost> with doc on disk
    that has <tpt>...</tpt> - add a sanity check there.
2 - a possible bug: the below dies if we don't have a listing_fields (?)
  <nested_element>
      <name>listing_fields</name>
      <element><name>foo</name><macro>something_that_installs_validate_hook</macro></element>
      <required>qw ( foo )</required>
  </nested_element>

comma/postgres bug:
  need to use a superuser for now.
  later do what it suggests and change COPY to \copy ?

#complete test:
dropdb -U postgres comma ; createdb -U postgres -O comma comma; \
make distclean ; perl Makefile.PL && make && make test && sudo make install

TODO: make sure all new API stuff is in doc/guide.html and Bootstrap.pm
TODO: test-case for $blob_el->append...

things to add to guide.html:
  everything in comma-2.html, plus probably some other stuff...

add to comma-2.html/Changelog:
  some sort of smart dump of svn commits to chronicle

2.1:
- apply & fix the stuff in weak/ for ->parent, ->top, etc.
- get more info on copyright bounding start dates from comma 1.21
- there is no way to get the size of the store from the index... perhaps
  this is a feature? ... but it would be nice to have an 
  $index->total_count or so
- make an Iteratable.pm or so which Storage::Iterator and
  Index::Iterator inherit from ( we probably can't use it for much
  but it will look nicer that way at least ... )

$sit->length() vs. $idx->count() ... resolve this inconsisency if poss.
consider iterator_refresh for storage iterators?
$sit->length(max => $n) would be nice for gigantic stores...

dave ideas:
  2 - (not sure if this is something we should do via comma, but to 
    fight the "SQL resistance" at MMFA):
  this common construct is reaaaaaly slow:
        for ( @{ $doc->issue_ptr } ) {
           push @issues, $index->single( where_clause => "doc_id = $_" )
        }
        foreach my $issue (@issues) {
          ...
        }
  you hit the DB N times. you can hit it just once by building a gigantic
  where clause, e.g.
        my $wc = join ("OR", map { "doc_id = $_" } @{ $doc->issue_ptr });
        my $it = $index->iterator(where_clause => $wc);
        while(++$it) { 
          ...
        }
  or even better, using an IN statement:
        my $wc = "doc_id IN (".join (",", @{ $doc->issue_ptr }).")";
        my $it = $index->iterator(where_clause => $wc);
        while(++$it) { 
          ...
        }


Reported by tim, 2/15/2007

This doesn't work... Probably to fix this after we get the weakref
stuff in and right.
$ perl -MXML::Comma -e 'use strict ;
  my $def = XML::Comma::Def->read( name=>"AllAfrica_Publisher" ); 
  foreach my $el ( $def->def_sub_elements() ) {
    my $tag = $el->name(); 
    print $tag . "\n" if $el->element_is_plural($tag);
  }'

This does:
$ perl -MXML::Comma -e 'use strict ;
  my $def = XML::Comma::Def->read( name=>"AllAfrica_Publisher" );
  foreach my $el ( $def->def_sub_elements() ) {
    my $tag = $el->name();
    print $tag . "\n" if $def->is_plural($tag);
  }'

This is an API bug. THE other is_* have this problem as well.


- standardize/clean up the order of "use" statements, etc. in t/*.t


with postgres, we get lots of spurious warnings starting "NOTICE"
  these can be ignored (i think/hope)
  the Pg code in general needs a general cleanup too.

pg for dummies:
  To get postgres up and running on my machine, for use with comma, I
  just did an ubuntu install, then:
    apt-get install postgresql-8.1 postgresql-server-dev-8.1
    sudo cpan -i DBD::Pg
    sudo bash
     $EDITOR /etc/postgresql/8.1/main/pg_hba.conf
     ## change:
        local  all  postgres  ident sameuser
        local   all         all                               ident sameuser
     ## to:
        local   all         postgres                          ident sameuser
        local all all password
        #local   all         all                               ident sameuser
     /etc/init.d/postgresql-8.1 restart
     su postgres
      createuser --pwpompt comma #for now make a superuser
        #there is a bug we need to fix with COPY commands with Pg
        #relevant tests are in t/indexing.t
      createdb -O comma comma
      exit #shell as postgres user
     psql -U comma comma #make sure you can connect
     psql -U postgres #make sure you can connect
     #edit Configuration.pm accordingly

- element-specific locking for when whole body lock is unnec. and too
  heavy weight?
- maybe use ExtUtils::MakeMaker::prompt for automated testers
- it'd be nice to have "iterators" on nested plural elements
- more test cases for $blob_el->{append|set}() combinations
- make $blob_el->set() more efficient by nuking the fcopy and
  instead keep the appendage and the original data seperate
  use _set_was_called for this
- audit for SQL queries => parameterized queries, e.g.:
  my $sth = $dbh->prepare_cached ( "SELECT bla FROM foo WHERE key = \"$key\"");
  $sth->execute();
becomes:
  my $sth = $dbh->prepare_cached ( "SELECT bla FROM foo WHERE key = ? ");
  $sth->execute( $key );
this is MUCH more secure and less error prone. it appears most of the
SQL code we have (if not all) already does this, but since I just 
discovered it, I want to make sure it ALL does...

- t/indexing.t sometimes fails because of an incomplete previous run, or
  cruft in the database. make it more robust.
  ### step 5. - STORE_ERROR/POST_STORE_ERROR/DOC_INDEX_ERROR (0001) 
  think: table to drop, delete entryy in index_tables, delete any comma_locks (dangerous)

- it'd be nice if we could do some SQL-ish things with storage iterators... but best to
  read up on the progress (or is that lack of progress) with SQLite first...

