----------------------------------------------------------------
TOP OF LIST:

! 5.10.1 !
  sort-within-records
    ??
  truncate
    myfound
  count -n
    https://github.com/johnkerl/miller/issues/370
  seqgen nr/fnr
    myfound
  unsparsify -f
    https://github.com/johnkerl/miller/issues/387
  pprint headerless
  golang ask .......
    ...
  JSON-parse bug
    https://github.com/johnkerl/miller/issues/394

! gating after: 'int x = 1; x = "abc"'

small:
* justWroteEmptyLine -> pprint too ...
* nidx @ 10-min
* text-put @ 10-min
* xls note (issue #396) @ file-format docs

medium:
? records as mlrmap -> mlrval?
  o reduce $* copy ...
  o opens the door to some (verb-subset) truly arbitrary-JSON processing ...

big:
* emitp/emitf
* json-null as type; & all dispo matrices/vectors ...
* smart fmtnum

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* parser-fu:
  o iterative LALR grok
- jackson notes
- gocc .txt/.go for simple grammars
o find/bookmark/grok rob's lexer slides
o iterate on a parser-generator with JSON config file
no need to bootstrap a parser for the parser-generator language
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

? reg-test: diff the *.err files?

? allow [[...]] / [[[...]]] at assignment LHS

! reconsider OFMT -- %.6f is not really OK when inputs have far more digits.

* new columns-to-arrays and arrays-to-columns for stan format

* single-line JSON for DKVP/CSV/etc ...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! JSON null needs to be passed through as-is ... somehow ... new mlrval type??
  o https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions
  o "null can be added to any value, and returns the other value unchanged."
  o "the length of null is 0"
  o note json types: string, number, boolean, null, object, array.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! smart fmtnum w/ state-machine on chars, and casts, etc -- decouple from raw sprintf
  o use:
    - %
    - 0 - " " + "'"
    - number for min field width
    - "." and precision width
    - s   d i o u x X   e E f F g G
    - "%%"
  o need a parser for validity, and a mapper to Go-builtin formats
  o need to pull out the printed type & use that for casting/ignoring. examples:
    - %d int: ok
    - %d float: int(x) then ok
    - %d string: convert to %s
    - %f int: float(x) thenok
    - %f float: as is
    - %f string: convert to %s
    - %s int: convert to %d
    - %s float: convert to %f
    - %s string: ok
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! smarter head-stop etc https://github.com/johnkerl/miller/issues/291
  o done-reading back-channels in stream.go
    https://gobyexample.com/non-blocking-channel-operations
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! fixed-width pprint input-scanner
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

* UDF nexts:
  o more functions (see below)
  o function metadata, w/ sort-by, for online help
  o json_parse / json_stringify functions?
  !! strlen for UTF-8; wat
  ! regex captures
  o strmatch https://github.com/johnkerl/miller/issues/77#issuecomment-538790927
  o DSL sort function https://github.com/johnkerl/miller/issues/77#issuecomment-321916921

* bash completion script https://github.com/johnkerl/miller/issues/77#issuecomment-308247402
  https://iridakos.com/programming/2018/03/01/bash-programmable-completion-tutorial#:~:text=Bash%20completion%20is%20a%20functionality,key%20while%20typing%20a%20command.

* Check for "Options:" in all mappers
* check for `f.Value` reassigned in all mappers -- reflects true default?

* more copy-on-retain for concurrent pointer-mods !
  o make a thorough audit, and warn everywhere
  o either do copy for all retainers, or treat inrecs as immutable ...

* sliding-window averages into mapper step (C + Go)
* stats1 rank

* double-check rand-seeding
  o all rand invocations should go through the seeder for UT/other determinism

* comment-handling
  - delegator for CSV ...

! quoted NIDX
  - how with whitespace regex -- ?
! quoted DKVP
  - what about csvlite-style -- ? needs a --dkvplite ?

* clean up 'mlr --help' to not advertise things it doesn't do yet

* ASV? golang csv package parameterizes comma but not crlf :(

* mlr -f / -F
  o others in mlrcli_parse.go
  o copy over all operator help-strings from ../c

* $map.attr1.attr2 syntax for DSL -- ? needs override of "." operator w/ symbols ... ?

================================================================
REMAINING VERBS:

seqgen
having-fields
repeat

bar
clean-whitespace
count-distinct
format-values
fraction
histogram
least-frequent
most-frequent
sec2gmtdate
tee
top
uniq

nest
reshape
step
join
merge-fields
stats1
stats2

================================================================
REMAINING FUNCTIONS:

* new columns-to-arrays and arrays-to-columns for stan format

* regextract regextract_or_else

* system

* sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime sec2localtime
* dhms2fsec dhms2sec fsec2dhms fsec2hms hms2fsec hms2sec localtime2sec gmt2sec
* strftime strftime_local strptime strptime_local

https://github.com/lestrrat-go/strftime
https://github.com/pbnjay/strptime

================================================================
CATEGORIES:

* widen verb coverage

* widen CLI coverage
  o --c2x et al. -- fold in markdown
  o implement mlrrc

* I/O
  o widen reader coverage
    - TSV/ASV
    - XTAB
  o widen writer coverage
    - TSV/ASV
    - markdown
  !!! golang encoding/csv ignores blank lines -- i need those for csvlite schema-change
    detection -- needs an input-stream delegator -- ?
  o --allow-ragged-csv-input|--ragged
  o non-lite DKVP reader/writer
  o comment-filtering-compaosed io.Reader
  o make a pprint reader w/ width-detect

* widen DSL coverage
  ! print/dump/emit w/ redirect options
  o c/reg_test/run reorgs
  o err-return for array/map get/put if incorrect types ... currently go-void ...
    ! the DSL needs a full, written-down-and-published spell-out of error-eval semantics
  o typed localvars
  o for-knv
  o profile mand.mlr & check for need for idx-assign
    -> most definitely needed
  o multiple-valued return/assign -- ?
    - array destructure at LHS for multi-retval assign (maps too?)

* beyond:
  o support 'x[1]["a"]' etc notation in various verbs?
  o sort within nested data structures?
  o array-sort, map-key sort, map-value sort in the DSL?
  o closures for sorting and more -- ?!?

----------------------------------------------------------------
MAYBE:

* string/array slices on assignment LHS -- ?
* push/pop/shift/unshift subroutines

----------------------------------------------------------------
GOCC UPSTREAMS:

* support "abc" (not just 'a' 'b' 'c') in the lexer part

----------------------------------------------------------------
DEV NOTES:

* build-dsl
* build
* fmter
* check
* git diff to see what changed
* commit

----------------------------------------------------------------
PARSER/LEXER PLAN:

* grok ch27
* grok pike-lexer slides
* dragon at ... what level ?

----------------------------------------------------------------
NITS/NON-IMMEDIATE:

* "Miller: " prefixes on all errors.New
* libify errors.New callsites for DSL/CST
* record-readers are fully in-channel/loop; record-writers are multi with in-channel/loop being
  done by ChannelWriter, which is very small. opportunity to refactor.
* address all manner of xxx and TODO comments
* throughout -- 'map' used too much -- 'mapping/mapper' -> 'transform' or something?
* AST insertions: make a simple NodeFromToken & have all interface{} be *ASTNode, not *token.Token
* mlr --help-for w/ stdout redirect for manpage -- ?
* mlr verb -h -> stdout & exit 0
* cst printer with reflect.TypeOf -- ?
* godoc ...
* mlrdoc false && 4, true || 4 because of short-circuiting requirement
* update whyc.html with efficiency notes from go/README.md
? makefile for build-dsl: if $bnf newer than productionstable.go
* I/O perf delta between C & Go is smaller for CSV, middle for DKVP, large for JSON -- debug
* neaten/error-proof:
  o mlrmapEntry -> own keys/mlrvals -- keep the kcopy/vcopy & be very clear,
    or remove. (keeping pointers allows nil-check which is good.)
  o inrec *types.Mlrmap is good for default no-copy across channels ... needs
    a big red flag though for things like the repeat verb (maybe *only* that one ...)
! clean up the AST API. ish! :^/
* defer reg_test/run until there is quite a bit more verb/dsl/io expansion
* doc: no boolish truthy falsy
* json:
  d thorough UT for json mlrval-parser including various expect-fail error cases
  d doc re no jlistwrap on input if they want get streaming input
  d UT JSON-to-JSON cat-mapping should be identical
  d JSON-like accessor syntax in the grammar: $field[3]["bar"]
  d flatten/unflatten for non-JSON I/O formats -- maybe just double-quoted JSON strings -- ?
    - make a force-single-line writer
    - make a jsonparse DSL function -- ?
  d other formats: use JSON marshaler for collection types, maybe double-quoted
  o research gocc support
  o maybe a case for hand-roll
* ICE on scan of 0xffffffffffffffff
* support 0b0101 et al.? mlrdoc if so
? dsl/ast.go -> parsing/ast.go? then, put new-ast ctor -> parsing package
  o if so, update r.mds
* keep or get rid of $[[[ ]]] ... ?
* 1-up note from mlrval_collections -> doc6
  o substr -- not backward-compatible
  o example of key-stringify to get truly awkish associative-array behavior
  o also $1 vs $[1] -- the former 'sticks' with input-data ordering.
  o moral: miller is a name-indexed tool as its primary purpose. positional
    indexing is supported but it can be confusing.
* mlrdoc: there should have always been a >>> and >>>=, and now there are
! systime printing out in scientific notation :(
! head early-out -- cite the relevant issue & fix the right way ...
* relnotes: label b,i,x vs x,i,b change
* put -- multiple -f? as in C impl?
* emit { int : string } results in absent-something :^/
* doc note 'array' is now a keyword
* dump is now redundant? print @*
* double-check dump CR-terminators depending on expression type
* doc nullish-coalesece, along with its precedence
* ordering of builtin functions in mlr -f output ........
  ? primarily by type, secondarily by name?
* CST validators: remove references to bare-boolean last statement in main block
  - this is a feature
* good example of wording for why/when to make a breaking release:
  https://webpack.js.org/blog/2020-10-10-webpack-5-release/
* unset, unassign, remove -- too many different names. also assign/put ... maybe stick w/ 2?
* make truncate() UTF8-friendly
* (absent) print on dump -- should simply be empty
* document array-unset as shiftback
! error if UDF has same name as built-in
* huge commenting pass
* profile mlr sort
* go exe 17MB, wut. try to discover. (gocc presumably but verify.)
* fill-down make columns required. also, --all.
* check triple-dash at mlr fill-down-h ; check others
* doc that multi-arg print interleaves spaces; they can strcat to avoid
* clean up unused exitCode arg in sort/put usage.
  o also document pre/post conditions for flag and non-flag usages x all mappers
* need to fix up ;; and start-with-; handling in the grammar so multiple put -f/-e work right :^/
? emit @x or emit x -- should make k/v pairs w/ "x" & value -- ? check against C impl
* doc one-variable for is over keys for maps, elements for arrays
* asserting_{type}: os.Exit(1) -> return nil, err flow?
* test put/filter w/ various combinations of -s/-e/-f
* math operators on map? array? e.g. a<b for contains? (prefer functions ...)
* more text examples in mlr-put doc
* mt_void keep-or-not .......
  o check dispo matrices
  o if keep, need careful MT_VOID at from-string constructor -- ? or not ?
  o comment clearly regardless
* window.mlr, window2.mlr -> doc somewhere
* bitwise_and_dispositions et al should not have _absn for collections -- _erro instead
* fmtnum: auto-replace %l_, %ll_ -> no l/ll for backwards compatibility, and comment accordingly
* mlrdoc for new go-only verbs
* ast-parex separate mlr auxents entrypoint?
* port u/window*.mlr from mlrc to mlrgo (actually, fix mlrgo of course)
* doc: substr in inferred-numeric fields: https://github.com/johnkerl/miller/issues/290.
  o xref to 1-up note.
* why error? mlr put '$y=true; $y &= $x<0.5' s
* line/column caret at parse-error messages -- would require some GOCC refactoring
  in order to get the full DSL string and the line/number info into the same method
* csvlite rd/wr: comment for USV/ASV too. no need for escaping then.
* comment schema-change supported only in csvlite reader, not csv reader
* for-multi: C semantics 'k1: data k2: a:x v:1', sigh ...
* neaten mlr gap -g (default) print
* case-c-* / case-go-* -> case-* eventually
* implement min/max for array/map ... using cmp rules
! write out thorough min/max/cmp cases for all orderings by type
* silent zero-pass for-loops on non-collections:
  o intended as a heterogenity feature ...
  o consider a --errors or --strict mode; something
* note about non-determinism for DSL print/dump vs record output-stream now ...
* mlrdoc the ??? operator
* document --jvstack is now a back-compat no-op
* doc about put -S/-F cannot make sense anymore
* put/filter updates:
  o filter statement needn't come last anymore: '$u = 1; NR > 5; $v = 2'
  o put 'NR > 3' and put 'filter NR > 3' are the same.
    more generally, bare-boolean and filter bare-boolean are synonymous -- 'filter' DSL keyword
    is legacy.
  o mlr put and mlr filter are identical except:
    - mlr put accepts zero statements like 'mlr put ""'; mlr filter requires at least one.
    - mlr filter 'filter NR > 2' is a syntax error <---- rethink even that .....
* [[...]] / [[[...]]]:
  o ....
  o ....
  o ....
  o put '$array = [1,2,3,[4,5]]' is a syntax error unfortunately; need '$array = [1,2,3,[4,5] ]'
* doc: string index/slice access on RHS
* substr(7,5) -- "" or error?
* write up re auto-deepen logic -- foo[NR] -- should create a map not an array unless already array
  o idea is arrays are created with explicit array literals, then extended as such
  o otherwise preserve map semantics for backward/awkish compatbility
  o arrays for JSON
  o doc clearly that these are different:
    begin {
      foo = []
    }
    foo[NR] = $foo # auto-extends a array with indices 1,2,3,...
    --versus--
    foo[NR] = $foo # auto-creates and auto-extends a map with keys "1","2","3",...
* reorder locations of get/put/remove methods in mlrval/mlrmap
* grep out all error message from reg-test outputs & doc them all & make sure index-searchable at readthedocs
* short 'asserting' functions (absent/error); and/or put --strict or somesuch
