perldocs for modules
   other than Record
more aggregators
   linreg
perf test
   recs-collate lru size 1 v. optimized adjacent recs collate
   JSON v. old recs v. other serialization (Data::Dumper, Storable, ?)
recs-join - Join in sqlite for larger db streams
recs-sort, recs-todb - need value options for keyspec groups
recs-fromhtmltable - parse an html file, extract data from a table
recs-collate - use Getopt::Long's multiple option values supports to provide easier aggregators, e.g.  something like --sum x

Aribritation Collation clumps
  - suportt week over week
  - add new aggreators that expect 2 or some number of recs

recs-autojoin - join stream against itself

Key groups:
  rerun field group for each record - may have an issue with recs-collate
  'Value attribute' ideas:
      !REGEX!v=numeric
      '!perc-(\d+)!v=Percentile: $1'
      '!perc-!v-eval=Percentile {{field}}'

RecordShell idea:
  14:34 <ben_bernard> my idea is basically you have like status files in a history tree
  14:34 <ben_bernard> and you have some sort of interactive way of selecting where you are in history
  14:34 <ben_bernard> and also merging up your history into a preivous branch
  14:34 <ben_bernard> and you could move back and forth between commands...
  14:34 <ben_bernard> maybe this doesn't make sense
  14:35 <ben_bernard> its definitely the sort of script you would never use
  14:35 <amling> :)
  14:35 <amling> sounds like you might have to do the brunt work of the design then
  14:35 <ben_bernard> but I think it would be cool
  14:35 <amling> with my only contribution being in the form of mockery :P
  14:35 <amling> since I could obviously never help build something I wouldn't use
  14:35 <amling> anyway, the history tree idea is cool but it definitely increases the depth of the learning curve
  14:36 <ben_bernard> ?
  14:36 <amling> the idea of having a "history" of your progress on changing your data into what you want
  14:36 <ben_bernard> I mean the only reason to use this shell is the history / stream management, I thin
  14:36 <amling> it's cool, but I wonder if it will make the shell more difficult to sapproach
  14:36 <amling> well
  14:36 <amling> the tree part anyway
  14:37 <amling> I guess if you never do anything you just get a boring linear history
  14:37 <amling> maybe you don't even have to know it could have been a tree
  14:37 <ben_bernard> yeah, I'm thinking like explicit branching
  14:37 <ben_bernard> sorta like git
  14:37 <ben_bernard> like you jump back in time
  14:37 <ben_bernard> add a command, it inserts into the linear history
  14:38 <ben_bernard> only if you 'branch' it there do you get a sub tree that doesn't apply to the current sets...
  14:38 <ben_bernard> or this is one thought for approaching the complexity
  14:39 <ben_bernard> I was also thinking of maybe instead just having a recs-chain command that captures the intermediate output...
  14:39 <ben_bernard> do you think that would be interesting?
  14:39 <ben_bernard> I think the shell is more interesting
  14:39 <amling> I think the shell is better
  14:39 <amling> I actually sort of liked the idea of having the current data pager'd into your face
  14:40 <amling> and then having options like 'step back', 'apply another command', etc.
  14:40 <amling> 'inspect previous steps'
  14:40 <ben_bernard> the issue is that it isn't the shell, right
  14:40 <ben_bernard> I'm concerned that that will make it useless
  14:40 <ben_bernard> perhaps have a ! prefix that will operate on the current set of files or something
  14:41 <ben_bernard> also I was thinking that you could just run commands
  14:41 <ben_bernard> and it would auto-pipe the current stream and create an output file
  14:41 <ben_bernard> doesn't have to be recs
  14:42 <ben_bernard> hmmm
  14:42 <ben_bernard> now I'm thinking: you have a set of streams you're working with
  14:42 <ben_bernard> you have a history tree for each stream
  14:43 <ben_bernard> you can use '!' or a command like 'shell' to dump into a shell with all the streams available, the RecordShell detects new files or changes
  14:43 <ben_bernard> (new files are new streams, changes mean an un-editable history jump, though you can still jump to before it, it just can't update after it...)
  14:44 <amling> it could have the shell your dumped into log history somewhere special and try to make sense of it
  14:44 <ben_bernard> so my thought
  14:44 <amling> in very limited cases it might mean ability to understand what happened (not just treat the shell session as a black box)
  14:44 <ben_bernard> is that in general you can run shell commands (with |s) and it will work just like recs commands
  14:45 <ben_bernard> so you only use ! or shell when you have to do the black magic
  14:49 <ben_bernard> what I would really like is sort of a programmatic shell
  14:49 <ben_bernard> like if I could get an interface that was like run this, but tell me the command name and what files were written to, oh and also here is your input stream

Keith's personal section
    decollate / deaggregators
    fix die w/in getopt
    --incr collate by time and by record count (--incr is by record count 1)
        maybe even windowed incremental?
            this would be expensive without allowing aggregators to remove points...
            some aggregators are vaguely order dependent (last) so what removal even means for them is unclear
            maybe we promised we'll only remove from the back?
        threshold start for increment as well (start after 10 records, start after 10 seconds)
    omfg, hierarchy stuff...
    field group
        for e.g.  recs-collate --default-agg sum=tp*
    moving average aggregators
        need a field and some need a "time" field
    domain language short term
        > *) operator overloading for DL::Value
        > *) consider name('...', <valuation/agg>) alternative to current --dlk and --dla naming mechanism
        >
        > Also possible: --dlnamedaggregator (to separate named from unnamed) or
        > inject_agg, etc. could take an optional name.
    recs-collate
        option shorthand for mr/ii adhoc aggregators, like
            --mr-agg sum_x '{{x}}' '$a+$b'[ '$a']
            --ii-agg sum_x 0 '$a+{{x}}'[ '$a']
        note to implementor: how the fuck are we taking so many arguments?
    refactor keys nightmare once and for all, this will include domain language keys (and help on them!)
    domain language
        brainstorming
            map_agg(map_ex('"sum_$f" => sum($f)', grep_re('/^t/', fields())))
                fields() is a VALUATION, returns an arrayref of fields
                grep_re(SCALAR, VALUATION) is a valuation
                grep_ex(SNIPPET, VALUATION) is a valuation
                map_ex(SNIPPET, VALUATION) is a valuation
                map_agg(VALUATION) is an aggregator, builds a map based on (per-record, sort of) aggregators specified by valuation
                this is definitely more powerful shit, it's just very verbose and that short example is now huge
            nested cubing aggregator
                like a full collate w/in its slice
                ugh, hash-taking functions (should ease the madness but at collision this is gonna be fucking tricky)
                    maybe they register their OK keys to avoid matching too much?
                    also, jesus, how do they define the "type" their arguments are?
                    ARRAY[VALUATION]?
                    HASH[AGG]?
