                                 Muldis::D
---------------------------------------------------------------------------

Following is a summary of things that still need doing.  It is specific to
the Muldis D specification distribution only, and doesn't talk about things
that would go in other distributions, including implementations.  (But,
look at lib/Muldis/D/SeeAlso.pod for a list of actual or possible
implementations.)

Alternately, this list deals with possible ideas to explore, which may or
may not be good ideas to pursue.

The following list is loosely ordered by priority, but list items may
actually be addressed in a different order.  There is no specific time
table for these items; they are simply to be done "as soon as possible".

* Generally speaking, make a new release to CPAN once every week, assuming
the progress is non-trivial, so there are regular public snapshots with
nicely rendered documentation.

* Add concept of shallowly homogeneous / sh- relation types to complement
the deeply version, and named maximal types like SHRelation, SHSet,
SHArray, etc to complement the DH/etc, and sh_set_of/etc to complement
dh_set_of/etc; but not sh-scalar or sh-tuple as the concept doesn't make
sense there.  Then update functions like Relation.union/etc to take
sh_set_of.Relation rather than set_of.Relation, which more formally defines
some of their input constraints.

* Add ext/TAP.pod, which is a partial port of Perl 5's Test::More / Perl
6's Test.pm / David Wheeler's pgTAP to Muldis D; assist users in testing
their Muldis D code using TAP protocol.  The TAP messages have type Text.

* Create another distribution, maybe called Muldis::D::Validator, which
consists essentially of just a t/ directory holding a large number of files
that are straight PTMD_STD code, and that emit the TAP protocol when
executed.  The structure and purpose of this collection is essentially
identical to the official Perl 6 test suite.  A valid Muldis D
implementation could conceivably be defined as any interpreter which runs
this test suite correctly.  This new distro would be a "testing requires"
external dependency of both Muldis::Rosetta and any Parrot-hosted language
or other implementation, though conceivably either could bundle a clone of
Muldis::D::Validator rather than having an actual external dependency.
This test suite would be LGPL licensed.  This new distribution would have a
version number that is of X.Y.Z format like Muldis::D itself, where the X.Y
part always matches that of the Muldis D spec that it is testing compliance
with, while the .Z always starts at zero and increments independently of
the Muldis D spec, as often there may be multiple updates to ::Validator
for awhile between releases of the language spec, and also since .Z updates
in the language spec only indicate bug fixes and shouldn't constitute a
change to the spec from the point of view of ::Validator.

* Create another distribution, maybe Muldis::D::Cookbook or ::Tutorial or
::Examples or what-have-you, which collects miscellaneous and large
documentation associated with the Muldis D language but isn't part of the
spec itself, with the main focus on showing reams of Muldis D example code
or how-tos or comparisons with SQL etc.  A main reason to be separate is so
that examples can be updated and expanded at a natural pace without having
to increment the language spec version number in the process, and to keep
the actual spec distro slimmer.  When this ::Cookbook distro is started,
its initial content can include updated versions of what is in archives/ of
the current distro, along with copies of static historical versions and
announcements etc.  Early Muldis D examples to write should focus on tasks
that are difficult to do in SQL that are easier in Muldis D.  For example
recursive operations (note, SQL has the term "Common Table Expressions" to
mean the WITH clause), such as something genealogy or a nested hierarchy of
locations with points of interest in them.  Also the examples in David
Fetter's FOSDEM 2009 talk (see
http://wiki.postgresql.org/images/c/c1/Trees_and_More_FOSDEM_2009.pdf )
would be good examples to translate to Muldis D.  Also implement the
common standby that Perl ORM docs like to use, about artist/song/album/etc,
and maybe look at the Songbird database format.  Or perhaps among the top
of the list for new examples should be a Muldis D implementation of Hugh
Darwen's course assignment "CS252 - Fundamentals of Relational Databases -
Module Assignment 2007" at the University of Warwick, described as "Your
task is to design a relational database to record information concerning
personal bank accounts: details about the customers holding these accounts
and details of individual payments of money into and out of these
accounts."  Maybe that actually will be the best first non-trivial example,
since it was designed with some strengths of D languages vs SQL in mind.
In general just the PTMD_STD dialect will be used for all examples, though
in some cases parallel examples in a HDMD_PerlN_STD dialect may be used.
Side note: Postgres 8.4 is apparently the first version of Pg to have CTE.
See also the entire database constraint set from the example database in
the book "Applied Mathematics for Database Professionals" (which SIRA_PRISE
is stated to enforce efficiently). Including stuff like "Every trainer
should be giving trainings in the location of his own department for at
least half of the total time that he is giving trainings." and "You can
only be assigned as a trainer for some course if either you have been
employed for over a year or else you have yourself attended another session
of that same course, and the trainer who gave that session that you
attended attends as a participant the training you give." (which the book
labeled the "pièce de résistance").

* IN PROGRESS ...
Update the STD dialects with special syntax for declaring whole depots
in a single expression, catalog+data.  This can be useful for example with
a simple Muldis D implementation that wants to represent a persisted depot
on disk as a single file that is Muldis D source code.  This would mostly
be like a bootloader definition except that details related to the name
that a depot is mounted with aren't given as they would with an
application-side bootloader.

* IN PROGRESS ...
Update the STD dialects with special syntax for some nonscalar catalog
types, particularly definitions of routines and types, so that both doing
this can be made even more like a normal language, and also the missing
pieces to support more appropriate syntax coloring and routine locating in
text editors looking at Muldis D source, eg a 'function' keyword at the
start of each function.  Here are some keywords that can then be added:
function, updater, procedure, type, param, ro_param, upd_param, expr, stmt,
subtype, key, possrep, etc; they apply to both 'outer' and 'inner'
types/routines; also keywords for declaring whole packages or something.

* Update the STD dialects to support inline definition of basic inner
routines (and types?) right in the expressions/etc where they are used,
such as filter functions in restriction() invocations, so many common cases
look much more like their SQL or Perl counterparts, or for that matter, a
functional language's anonymous higher order functions.  This syntax would
be sugar over an explicit inner definition plus a FooRef value selection,
which means the inner def effectively is an expression node, and users can
choose to name or not name the FooRef selecting node as normal with value
expressions.  It is expected that the inners could be declared anonymously
and names for them (the inn.foo, not the FooRef's lex.foo) would be
generated as per inline expression nodes etc.  Presumably also add some new
keyword for a routine to invoke itself without using a name like normal;
this would be sugar for normal functional call syntax, using gen inn name.

* Further to the previous item, add some special syntax, similar to how one
references a parameter to get its argument's value, which can see into the
caller's lexical scope.  This would be sugar over declaring parameters with
the same name and having the caller explicitly pass arguments to it,
without having to explicitly write that.  Generally this syntax would only
be used with inline-declared routines.  But similarly, add some special
syntax allowing one to essentially just write the body of a routine without
having to explicitly write its heading / parameter list, which is useful
for routines invoked directly from a host language, where said parameters
are attached to host bind variables.  Now one still has to say what the
expected data type is for these bind variables, but then the explicit
syntax for such Muldis D routines is more like that of a SQL statement you
plug into DBI or whatever, without the explicit framing.  May not work
anywhere, but should help where it does.  Maybe use $$foo rather than $foo
to indicate that the 'foo' wasn't explicitly declared in the current
lexical scope and we are referring to the caller or a bind variable.  Or
rather than $$foo, have something like "(param Bar $foo)" for an
expression-inline parameter definition and use, where the part after the
"param" has all the same syntax as an actual param list; this is the one
for host language bind parameters.  Actually that might be useful by
itself.  Similarly "(caller $foo)" would be the look to parent Muldis D
lexical scope, or $$foo would just do that maybe, unless this should have
an explicit type declaration still.  Note, if same inline-declared host
param used more than once, you just need "(param Bar $foo)" form once, and
other uses can just say $foo as per usual; in fact, it must be this way.

* Add one or more files to the distro that are straight PTMD_STD code like
for defining a whole depot (as per the above) but instead these files
define all the system entities.  Or more specifically they define just the
interfaces/heads of all the system-defined routines, and they have the
complete definitions of all system-defined types, and they declare all the
system catalog dbvars/dbcons.  In other words these files contain
everything that is in the sys.cat dbcon; anything that users can introspect
from sys.cat can also be read from these files in the form of PTMD_STD
code, more or less.  The function of these files is analagous to the Perl 6
Setting files described in the Perl 6 Synopsis 32, except that the Muldis D
analogy explicitly does not define the bodies of any built-in routines.  An
idea is that Muldis D implementations could take these files as is and
parse them to populate their sys.cat that users see; of course, the
implementations can actually implement the routines/types as they want.
Note that although this Muldis D code would be bundled with the spec, it is
most likely that the PTMD_STD-written standard impl test suite will not.
Note that these files will not go in lib/ but in some other new dir.  Note
that it is likely any implementation will bundle a clone of these files
(suitably integrated as desired) rather than having an actual external
dependency on the Muldis::D distro.  Note that some explicit comment might
be necessary to say there are no licensing restrictions on copying this
builtins-interfaces-defining code into Muldis D implementations, or maybe
no comment is necessary.  Probably a good precedent is to look at what
legalities concern existing tutorial/etc books that have sample code.

* Further to the prior item, add to Routines_Catalog.pod and other files
definitions of any remaining routines, eg String routines, that would be
needed so that for all system-defined types all the necessary
system-defined routines would exist that are necessary for defining said
types, especially their constraint or mapping etc definitions.  So in
String.pod, besides the usual suspects like [catenation, repeat, length,
has_substr] etc, we also need: 1. "has_single_elem" tests if its single
param String $topic has exactly one element or not, and its invocation
comprises the whole "constraint" of the sole possrep of the SEString type
by way of which the Int type is defined; a more generic length test won't
work since that would take an Int argument, so we can't define Int with it.
2. "all_elems_in_range" tests if every element of its param String
$topic is an integer in the range given by the other arguments, which can
be Int typed; this is used in the "constraint" def of the [B|O|UCP] types.
Also add "is_coprime" or GCD or LCM or etc which are used either in the
constraint definition of Rat or in a normalization function for Rat; see
also "the Euclidean algorithm" as an efficient way to do the calculations.

* Consider adding type introspection routines like: is_incomplete() or
is_dh() or is_primitive|structure|reference|enumerated etc.  Or don't
since one could look that up in the system catalog.  But more tests on
individual values might be useful, or maybe we have enough already.

* Consider adding a direct analogy to Perl 6 "roles", whose primary purpose
is to help with managing software reuse, mainly when multiple types have a
number of attributes in common, a role can define these and then the
multiple types can compose that role.  This concept should be workable with
both InnerScalarType and InnerNonscalarType.  A role can compose other
roles.  A role or type that composes a role can both add additional
attributes of its own to what the role defines, and the composer can add
extra constraints over the composed attributes like forcing a subtype.  Now
on one hand we could make a role a distinct concept, which by itself can't
be instantiated as a type, or we could reuse the above 2 (IST/INT) type
definition facilities and say an existing such type can be composed into
another.  Now that latter case would in a manner of speaking give Muldis D
the advantages of other OO languages or SQL that work using "specialization
by extension" except that composing an existing type in this way will
actually, for a scalar base, create a disjoint type not a subtype, so we
get the code reuse but Muldis D is still specialization by constraint.  On
the other hand, if we just declare roles as roles that can't be
instantiated themselves, then declaring a role could also serve to declare
a non-dh union type name that can be referred to in type constraint
definitions, that union type implicitly ranging over all actual types that
compose it, and so in this case the composers actually are subtypes; the
problem with this is, what to do about a default value for said union type
whose role isn't composed into anything ... maybe each one needs an
implicit DVPT subtype?  But maybe by TTM terms, what a role actually is is
a dummy type of sorts, where you declare the union name before what it
unions, and the regular types it unions declare themselves part of the
union rather than the other way around.  In any event, a composed role
would never appear in a value's MST or root type list, same as any type
declared as a union.  I should look at TTM and see if they intended for
dummy types to fulfil a role such as this.  Except all directly declared
subtypes of a role union are always disjoint, for scalar ones anyway; the
role is defin shorthand.

* IN PROGRESS ...
A Muldis D host|peer language can probably hold lexical-to-them variables
whose Muldis D value object is Reference typed, and so they could
effectively pass around an anonymous Muldis D FuncRef etc like a closure of
their own language.  Such a value object would be a black box to the host
and can't be dumped to Muldis D source code.

* IN PROGRESS ...
Fully support direct interaction with other languages, mainly either peer
Parrot-hosted languages or each host language of the Muldis D
implementation.  Expand the definition of the "reference" main type
category (or if we need to, create a 5th similarly themed main category) so
that it is home to all foreign-managed values, which to Muldis D are simply
black boxes that Muldis D can pass around routines, store in transient
variables, and use as attributes of tuples or relations.  These
of course can not be stored in a Muldis D depot/database, but they can be
kept in transiant values of Muldis D collection types which are held in
lexical variables by the peer or host language; that language is then
really just using Muldis D as a library of relational functions to organize
and transform its own data.  We also need to add a top level namespace by
which we can reference or invoke the opaque-to-us data types and routines
of the peer or host language.  This can not go under sys.imp or
sys.anything because these are supposed to represent user-defined types and
routines, which in a dynamic peer language can appear or disappear or
change at a moment's notice, same as in Muldis D; on the other hand, types
or routines built-in to the peer/host language that we can assume are as
static as sys.std, could go under sys.imp or something.  This also doesn't
go under fed etc since fed is reserved for data under Muldis D control and
only ever contains pure s/t/r types.  Presumably this namespace will be
subdivided by language analagously to sys.imp or whatever syntax Perl 6
provides for calling out into foreign languages co-hosted on Parrot.  Since
all foreign values are treated as black boxes by Muldis D, it is assumed
that the Muldis D implementation's bindings to the peer/host language will
be providing something akin to a simple pointer value, and that it would
provide the means to know what foreign values are mutually distinct or
implement is_identical for them.  One thing for certain is that every
foreign value is disjoint from every Muldis D value, and by default every
foreign value is mutually distinct from every other foreign too, unless
identity is overloaded by the foreign, like how Perl 6's .WHICH works.
The foreign-access namespace may have a simple catalog variable
representing what types and routines it is exposing, but to Muldis D this
would be we_may_update=false.  It stands to reason that Muldis D's standard
FuncRef etc can point to the name of a foreign routine as much as a native.
TODO:  Figure out how to declare a Muldis D non-dh type whose attributes
are declared of some foreign type ... or maybe we can't and all such
attributes would simply be declared Universal all the time, might be enuf.

* IN PROGRESS ...
First round to above:  Add system-defined subtype of Reference called
sys.std.Core.Type.Foreign (itself 'special' like FuncRef etc), the new
one being a root type, and then we simply say that all values passed in
from a host or peer language are of that root type, period; this new one is
declared in Types.pod probably maybe I guess.  Then update Perl5_STD and
Perl6_STD to add a new selector node kind 'Foreign' which takes any Perl
value or object as its payload; this is treated completely as a black box
in general within the Muldis D implementation.  For matters of identity
within the Muldis D envirnment, it works as follows:  Under Perl 6, the
Perl value's .WHICH result determines its identity.  Under Perl 5, if the
value is a Perl ref ('ref $obj' returns true) then its memory address is
used, and this applies to all objects also (since all refs are mutable,
this seems to be the safest bet); otherwise ('ref $obj' is false) then the
value's result in a string context, "$obj", is used as the identity; the
mem addr and stringification would both be prefixed with some constant to
distinguish the 2 that might stringify the same.  By default, a
ForeignValue supports no operators but is/not_identical.

* Add catalog types that represent thrown exceptions.  Probably scalar.

* Also study SQL concept of conditions and handlers, looks sort of like
something between exception handling, signals; or it is their exceptions.

* Update the virtual attributes maps so there is a way to manually specify
a reverse function, as meanwhile all the virtuals don't have to be either
read-only or updatable due to an automatically generated reverse function,
which might vary by implementation, which may be considered broken.  Note
that the reverse functions might have to be defined as per-tuple
operations, separately for insert/substitute/delete.

* Add transition constraints.  The initial version would simply take a
'before' and an 'after' state argument.  But it may be good to provide an
alternative where arguments show the delta since often that is what
constraint writers want to know anyway.

* Add other kinds of triggered routines.

* Update Text.pod (or add other extensions) to add general search and
replace or transliteration or regular expression et al functionality,
besides the bare minimum SQL-LIKE-alike currently provided.  Also add
functions for splitting strings on delimiters or catenating them with such.

* Add functions for comparing or processing continuous ranges or intervals
as per is_inside_range ... the tests are for eg if 2 ranges do or don't
overlap, or whether an overlap is specifically between the start of one and
the end of another, or whether ranges simply touch, or whether a range is
entirely within another or not, etc ... some are analagous to set
operations, but not so much as they talk about continous intervals, or as
continuous such as the type in question supports ... useful particularly
for use with date data types.  as for supporting non-continuous ranges,
which are probably represented by sets of continuous ranges, that might be
simple enough to support here too ... or some range stuff might go off to
another extension perhaps or perhaps not.

* Add 'Period' type etc or some such with 'from' and 'to' or 'start' and
'end' etc attributes, that represents a period in time; the attrs would be
Instant values and this might be a core type in support of temporal or
multiversioned databases.  See also the PGTemporal project.

* Consider supplementing the current generic infinite-loop-until-leave
statement with alternatives that use boolean control variables, such as
'while' or 'repeat-while'.

* Consider adding an imperative for-each looping statement; the main
question here is whether it should work on any (unordered) relation or just
on an Array (in which case it iterates through the tuples in sequence by
index); the question is what tasks the for-each would be used for; perhaps
both versions are useful; presumably the main reason to have for-each at
all is when I/O is involved and some derivative needs to be output either
where order matters or where order does not matter; but perhaps only a
routine is needed here such as a catenate function plus normal I/O output.

* Consider converting the try-catch routine into a statement node, an
optional part of which is to make the insides of the try/catch blocks part
of the same lexical scope as the invoking routine.  But there is still a
lot to like about having each of the try/catch blocks being a separate
(inner) routine with its own lexical scope, such as in predicting what a
rollback would affect, and it might be easier to implement as-is.

* Add sequence generator updaters|procedures in Integer.pod.

* Add Bool-resulting relational operators EXISTS and FORALL, that provide
"existential quantification" and "universal quantification" respectively,
these being useful in constraint definitions.  See TTM book p168, pp394-5
for some info on those.  Also add analogies to Perl 5's List::MoreUtils
operators any(), all(), notall(), none(), true(), false(); some of those
may be the same as EXISTS/FORALL.  Also add an EXACTLY operator like the
Tutorial D language has, and a one() op that is between any() and none().
Maybe some pure boolean ops can be added analagous to the above also; eg
any() an alias for or() and all() an alias for and().

* Consider adding random value generators for data types other than integer
and rational numerics, such as for character strings or binary strings.

* Consider updating the STD dialects to to provide alternate special
syntax for certain values, for example in PTMD_STD allowing value literals
like these to be valid: Tuple:d0, Relation:d0c0, Relation:d0c1.

* Make further updates to Temporal.pod to improve correctness or
normalization, particularly with durations, maybe add some UTCDuration
subtypes.  Also expand the selection of routines to allow more control over
semantics.  Seek advice from the Perl DateTime people about this stuff.

* Consider updates to Temporal.pod that let one record time zones etc
attached to instant values; this would be a non-central feature.

* Provide operators for the spatial data types, maybe overhaul the types.

* Update the system catalog concerning managing shared|exclusive locks or
looking for consistent reads between statements, etc.

* Update the system catalog to deal with database users and privileges etc.

* Flesh out matters related to starting or communicating between multiple
autonomous in-DBMS processes, in general, besides the special case about
sequence generators.

* Consider some more elegant solution about matters of closures et al.

* Whatever else needs doing, such as, fixing bugs.
