$Id: TODO,v 1.16 2001/07/15 19:46:38 rcaputo Exp $

===============================================================================
                             Introduction
===============================================================================

------------
What This Is
------------

This file is part roadmap and part wishlist.

The roadmap section is relatively small and lists confirmed changes.
Most of these changes will break existing code, which is why they're
listed here.  The changes which don't break programs usually don't
need to be documented this far ahead of time. :)

The larger wishlist section is composed of changes being considered
but which may never come to pass.

Everything is open to discussion, even the confirmed changes.  If you
spot something you'd like to comment on, please feel free to mail me
directly.  The POE mailing list might be a better place to discuss
changes since several people can discuss what you're commenting on.

Hopefully issues can be worked out before changes are implemented.

-------------------------
Typographical Conventions
-------------------------

Sections marked with "(!!!)" may break existing programs.

Various forms of "Xyz" are used throughout to mean "for every".  For
example, "POE::Filter::Xyz" means "for every POE::Filter subclass".

Bits marked with question marks (like "????.??.??") are blanks to be
filled in once their values are determined.  They are usually version
numbers and release dates.  Almost everywhere, though, their context
gives some idea of the time frame involved.  For example, an unknown
date 28 days after some release means that there is at least a month
to fix something after the release.  If the release version is also
blank, then you're doubly safe, and so on.

===============================================================================
                 Depreciations & Changes IN PROGRESS
===============================================================================

These changes have begun or will begin shortly.  If they plan to break
existing programs, there will be at least a month (usually two) of
advance warning.

--------------------------
XyzState to XyzEvent (!!!)
--------------------------

In the wheels, rename XyzState to XyzEvent everywhere.  Wheels don't
define states, they emit events.  Callivg Wheels' parameters XyzState
has been inconsistent and confusing.  It's bitten a lot of people.
Call them what they are instead.

Schedule:

1. Release a version with XyzEvent and XyzState working side by side.
   XyzState becomes undocumented in favor of XyzEvent.

   Done: 2001.07.15.

2. Wait at least 28 days after the release.  The next stage of
   breakage will fall on or after 2001.08.12.

3. Release a version that generates warnings when XyzState is used.

4. Wait at least 28 days after the warning release.  The next stage of
   breakage will fall on or after 2001.09.09.

5. Release a version that does not support XyzState at all.

-----------------------
Improve Alarm Interface
-----------------------

This change adds new features without breaking the way old ones work.

Schedule:

1. Release a version with the new functions in place.

   Done: 2001.07.15.

2. Optimize the new functions.  Currently there are chunks of common
   code which have been copied, pasted, and altered for their new
   environments.  Some of this can become macros.

Notes:

+ alarm_set(), alarm_adjust(), and alarm_remove() are done
+ delay_set() is done
+ alarm_remove_all() is done

- decided against alarm_remove_by_event(); use alarm() instead
- decided against alarm_remove_by time(); it's not very useful

----------------------------------
Depreciate queue_peek_alarms (!!!)
----------------------------------

The new alarm functions let programs track their alarms by ID, so they
no longer need to peek into POE's alarm queue.

Schedule:

1. Release a version with queue_peek_alarms() depreciated and
   undocumented.

   Done: 2001.07.15.

2. Wait at least 28 days after the release.  The next stage of
   breakage will fall on or after 2001.08.12.

3. Release a version with a depreciation warning in
   queue_peek_alarms().

4. Wait at least 28 days after the release.  The next stage of
   breakage will fall on or after 2001.09.09.

5. Release a version with queue_peek_alarms() removed altogether.

===============================================================================
               Depreciations & Changes BEING CONSIDERED
===============================================================================

These changes are not confirmed yet.  They are still under heavy
consideration, and they may never actually happen.  Even if they
include depreciation schedules, they are only tentative.  I can't
overly stress that this is the wishlist section of the TODO file, but
I'll certainly try. :)

----------------------------------------
Signal Handling Flawed As Designed (!!!)
----------------------------------------

This is a high priority task.

Signal propagation (and the _signal event) are vestigial design from a
time when POE was to be the core for a multi-user object environment.
They provide some extra features that make writing MUDs easier, but
they hamper many general programming tasks that POE is now being used
for.

Signal handling needs a basic redesign to make it more conducive to
generic programming, especially for components.  The problem is that
handling a signal in one place does not prevent it from being
delivered elsewhere.

The example that brought this to my attention is an IRC client being
written with POE::Component::IRC.  The main program traps SIGINT to
prevent the user from accidentally killing the client.  Perhaps it
will ask the user to confirm the kill, to prevent errant keystrokes
from doing damage.

However, because SIGINT is delivered everywhere, POE::Component::IRC
dies regardless whether the main program catches that signal.  This is
bad because the back-end IRC connection goes away when the user
interface doesn't necessarily want it to.

While the current signal propagation scheme is useful in some cases,
it's bad for the general case.  Every component would need to agree
whether it should or shouldn't catch signals in a particular program.
That's madness.

So this needs a redesign and a depreciation schedule.

--------------------------------
Make Things Inheritance-Friendly
--------------------------------

There was a time in POE's development where just using @ISA would make
method calls three times slower.  Whatever perl was doing got fixed,
and now it's safe to use inheritance again.

    b01_solo: 13 wallclock secs ( ... ) @ 924187.73/s (n=10000000)
  b02_parent: 12 wallclock secs ( ... ) @ 901408.45/s (n=10000000)
   b03_child: 12 wallclock secs ( ... ) @ 919540.23/s (n=10000000)

Session should be made inheritable, if it's not already.  The base
Wheel, Filter, and Driver classes maybe.  Component could provide some
useful common things; perhaps Philip Gwyn can add interoperability
stuff there and make components do the right things.

Kernel?  Should it be inheritable?

----------------------------------------------------
The POE Community Wants a Bigger Hand in Development
----------------------------------------------------

The YAPC 2001 POE BOF agreed that its members want a bigger part in
the project's development.  This is great!  It should reduce my
workload and generally expand POE knowledge across a larger group of
people.  If I get hit by a passing blimp, life goes on without me.

For such a group to remain organized, however, I feel the need to
codify the goals and heuristics that guide my own development.
Certainly these would just be guidelines.  Evolution happens, and they
may change over time as the project continues to grow up, but it's a
good idea to know going into it what's expected to come out of it.

So, what am I trying to do here?  What are the decision priorities
when making trade-offs?  Where do I want to go today?

----------------------------------------
Filter Changing Flawed As Designed (!!!)
----------------------------------------

POE::Filter::Xyz's get() method is optimized to reduce the number of
times it must be called.  Unfortunately, this means it parses raw
input in advance of the next input event.

Consider parsing HTTP requests.  Requests start with some headers
which must be parsed as lines, followed by a blank line, and then raw
streamed input which may be almost anything.

The Filter::Xyz optimization has it accepting raw data and returning
at once as many parsed things as it can.

This causes problems when switching filters.  When a filter changes,
there quite likely is some data beyond that which was already parsed
with the old filter.  When switching from lines to raw data when
moving from HTTP headers to the body itself, part of the body may have
been interpreted as lines.

The solution with the least amount of public breakage would be to redo
the input/dispatch loop within POE::Wheel::ReadWrite.  Currently it's:

  if (defined(my $raw_input = $driver->get($handle))) {
    foreach my $cooked_input (@{$input_filter->get($raw_input)}) {
      $k->call($me, $$event_input, $cooked_input, $unique_id);
    }
  }

The Filter would have to be pessimized to return one thing at a time
instead of many, and the Wheel would need to loop, calling the filter
as many times as necessary to return everything:

  if (defined(my $raw_input = $driver->get($handle))) {
    while (my $cooked_input = $input_filter->get_one($raw_input)) {
      $k->call($me, $$event_input, $cooked_input, $unique_id);

      # Undefining $raw_input flags get_one() to just return whatever
      # it has in subsequent calls.
      undef $raw_input;
    }
  }

In the new design, rapid bursts of data are parsed one at a time.
This leaves as much data as possible in the filter's raw buffer, so it
can be transferred to the next filter in a mid-stream change.

This does what people expect at the expense of speed in most cases.
Artur is vaguely against this, and rightfully so: why should most code
slow down for the relatively rare instances where people change
filters?  However, requiring someone set a flag ahead of time
requesting this feature seems to me like a setup for failure.  People
will forget, wonder why it doesn't work, and eventually grow to
despise having to set that flag.  Mark my words, copy them, and paste
them somewhere safe.

Part of the design requires a new get_one() method for all the
existing filters.  get_one() gets one thing, leaving the raw buffer
alone.  Wheel::Xyz can use can() to determine if get_one() exists and
support that if it's available.  If not-- for example, in legacy or
third-party filters which haven't been updated-- it can fall back to
the less reliable behavior.  Furthermore, set_filter() and its
variants can carp if used with unreliable filters.

---------------------------------------
Give Exported Constants Some Prefix(es)
---------------------------------------

<Micksa> eep, poe uses yet another calling convention
<Micksa> like:
<Micksa>    my ($kernel, $heap, $session) = @_[KERNEL, HEAP, SESSION];
<Fletch> that's the normal convention, it just doesn't hardcode what
         order particular arguments will be passed
<Micksa> it's, like, the 4th way I've seen so far for passing named
         parameters.
<Fletch> it's so that if calling order ever changes programs that use
         the symbolic constants don't break
<Fletch> it's really no different than say
         `my($kernel,$heap)=@_[0,3]', just that the 0 and 3 come from
         constant subs rather than magic numbers
<Micksa> problem is, the symbolic constants are rather likely to
         clash.
<Fletch> maybe there could be an option to prefix them, like
         poe_KERNEL or P_KERNEL?
<Micksa> *shrug*

That sounds like a good idea.

-------------------------------------------
Change Internal Helpers To Static Functions
-------------------------------------------

Despite recent optimizations in perl 5, object methods are still a wee
bit slower than function calls.  Furthermore, those changes aren't in
older versions of perl where POE still runs fine.

Because of Kernel restructuring in 0.05_02 (1999.01.12) prohibiting
multiple Kernel instances, we can finally make POE::Kernel's private
methods static functions.

What do we get?  Better performance (by a factor of 3 or so) in older
versions of perl.  Less argument passing internally (no $self; the
lexical $poe_kernel instead).  Greater encapsulation, but who cares
about that?

This is a relatively easy change, and it would make a good starter
project for someone who wants to get familiar with Kernel.pm and
Kernel/*.pm.

------------------------------------------
Signal Handler Return Values Must Go (!!!)
------------------------------------------

Signal handlers' return values are significant, determining whether
the handler has handled a signal, which in turn determines whether the
session will survive a terminal signal dispatch.  This is the only
time where a state/event handler's return value is significant within
POE itself, and it's incongruous with the rest of the library.  It's
also really easy to accidentally handle (or not) a signal by mistake:
simply implicitly return some random value.

To do:

* Add a POE::Kernel method: handle_signal().

  This will indicate that the currently dispatched signal has been
  handled for the session.  Should it also affect all sessions?  For
  example, if one session handles SIGINT, would they all remain alive?

* Change the documentation to list handle_signal() as the proper way
  to handle terminal signals.

* Run a complete test/coverage suite to make sure the current signal
  handling code is not broken in the process of adding the new code.

* Add new tests for handle_signal() to make sure it works, and run a
  complete test/coverage suit to verify that.

* Port the samples to use handle_signal() instead of return values.

* Release version ?.??, and announce the depreciation.

* Wait at least 28 days after the formal depreciation announcement
  (until ????.??.??) before continuing this change schedule.  This
  gives people time to adapt their production code to the new signal
  handling method before the warnings start.

* Add depreciation warnings whenever a terminal signal is handled by
  its return value instead of the new handle_signal() method.

* Release version ?.??, and announce the next phase of the signal
  handler depreciation.

* Wait at least 28 days after the next phase announcement (until
  ????.??.??) before continuing this change schedule.  This gives
  people time to adapt their production code to avoid the depreciation
  warnings and subsequent breakage.

* Remove the code to recognize implicit signal handling due to
  handlers' return values, making the handle_signal() method the only
  way to do this.

* Release version ?.??, and announce that signal handlers' return
  values have been completely depreciated.

----------------------
Batch Remove Resources
----------------------

Peter J. Braam would like a way to flush all pending events for a
session.  This would be FIFO events, since there would already be a
method for alarms.

Kirill would like a way to batch-remove all selects when it's time for
a session to shut down.

+ alarm_remove_all() is done

To do:

* Add select_remove_all()

* Add event_remove_all()

Notes:

Other resources which keep a session alive: extra reference counts,
and aliases.  Should there be batch remove functions for these as
well?

I'm starting to worry about the side effects of stopping sessions
willy-nilly.  Is there a good example of stopping a session while it's
busy processing events?  "Good" means the session won't corrupt
anything by not handling whatever is in its queue.

What of everything_remove_all() ?

-------------------------------------------------------
POE::Kernel's yield() is Nearly Universally Hated (!!!)
-------------------------------------------------------

Especially by people who know threads.  It implies blocking in one
session and resumption in some other, and it freaks people out when it
returns immediately.

These names have been suggested instead:

  post_self    (2 votes)
  post_me      (1 vote)
  post_myself  (1 vote)
  postal :)    (0 votes)
  enqueue      (1 vote)
  postpend     (1 vote)
  continuation (1 vote)
  continue     (0 votes)

This is a pretty basic function, and it's one a lot of people are
probably using.  This depreciation should have a longer than usual
schedule, to avoid unnecessary breakage.

----------------------------------
SIGTSTP Does the Wrong Thing (???)
----------------------------------

Gordon Matzigkeit writes (16 Feb 2001):

| If you haven't already, would you please add TSTP as a signal that
| should not be messed with in POE::Kernel:
|
|         if (POE_USES_EVENT) { # include
|           # If Event is available, register a signal watcher with it.
|           # Don't register a SIGKILL handler, though, because Event
|           # doesn't like that.
|           if ($signal ne 'KILL' and $signal ne 'STOP'
|               and $signal ne 'TSTP' # <-- HERE
|              ) {
|             Event->signal( signal => $signal,
|                            cb     => \&_event_signal_handler_generic
|                          );
|           }

I responded:

| I think a more correct implementation might set up TSTP and CONT
| handlers like in _APUE_'s program 10.22.  That way, POE can
| gracefully suspend and resume itself while also propagating TSTP and
| CONT signals to sessions so they too can perform pause/resume tasks.

The code hasn't been designed yet, but here's the code from program
10.22 so I don't have to keep looking in the book.

int main (void) {
  int n;
  char buf[BUFSIZE];

  /* only catch SIGTSTP if we're running in a job-control shell */
  if (signal(SIGTSTP, SIG_IGN) == SIG_DFL)
      signal(SIGTSTP, sig_tstp);

  /* ... code to do stuff ... */

  exit(0);
}

static void sig_tstp(int signo) { /* signal handler for SIGTSTP */
  sigset_t mask;

  /* ... code to set up us the suspend ... */

  /* unblock SIGTSTP, since it's blocked while we're handling it */
  sigemptyset(&mask);
  sigaddset(&mask, SIGTSTP);
  sigprocmask(SIG_UNBLOCK, &mask, null);

  signal(SIGTSTP, SIG_DFL);  /* reset disposition to default */
  kill(getpid(), SIGTSTP);   /* and send the signal to ourself */

  /* kill won't return until we receive SIGCONT or something */

  signal(SIGTSTP, sig_tstp); /* reestablish signal handler */

  /* ... code to set up us the resume ... */

  return;
}

-----------------------
ARG0..ARG9 May Go (!!!)
-----------------------

HEAP, KERNEL, SESSION, and the other event handler parameter constants
were introduced to eliminate a dependence on their positions in @_.
However, the ARG0..ARG9 parameters aren't descriptive and still
contain position dependence.  This robs event parameters of many of
the benefits of using symbolic constants in the first place.

This is the current plan to depreciate them:

* Introduce new constants for the different built-in events.  Leave
  ARG0..ARG9 for user parameters.

* Document the new constants instead of ARG0..ARG9.

* Document that parameters may move around in the future, so their
  positions in ARG0..ARG9 will not be guaranteed.

* Release version ?.??.

* Wait at least 28 days after releasing the documentation change
  (until ????.??.??) before continuing this depreciation schedule, to
  give people time to react to the initial round of changes.

* Generate warnings when ARG0..ARG9 are used for built-in events.
  This will require some logic within those subs, removing their
  constant nature and slowing things down in general.

* Release version ?.??, and announce that the ARG0..ARG9 built-in
  parameter depreciation is proceding apace.

* Wait at least 28 days after releasing the warnings (until
  ????.??.??) before continuing this depreciation schedule, to give
  people time to react to the warnings and adapt their code to the new
  symbolic constants.

* Remove the warnings from ARG0..ARG9, making them constants again and
  speeding things up once more.

* Release version ?.??, and announce that the ARG0..ARG9 built-in
  parameter depreciation has completed.

----------------------------
Spin Off Useful Technologies
----------------------------

POE only really needs a few modules.  The rest are options which may
not always be needed, or they're stand-alone modules that someone else
may find useful.  For example, POE::Filter::*; POE::Preprocessor; and
POE::Pipe::* are useful by themselves.  Wheels are pretty useful, but
they're not always necessary.  It may be both useful and convenient to
split POE into smaller distributions and present one or more Bundles
to selectively load just the parts that are needed.

This is the current plan to spin off useful modules from POE:

* Develop a real plan for this.  Most of this is tentative.

* Organize subsets of POE into useful bundles, and document them.
  Make Bundle::POE the default.

* Split out wheels?

* Split out the Preprocessor?  Should it remain a POE::* module?

* Split out filters?  They're useful by themselves; should they remain
  POE::* modules?

* The samples are huge and obscure.  Split them into a separate
  distribution which doesn't install itself.

* Most of the documentation is theory and usage, and it doesn't really
  fit in manpages for modules themselves.  Split it into separate POD
  files, and maybe split them into Bundle::POE::Docs.

-----------------------------------------
Make Driver::SysRW the Default for Wheels
-----------------------------------------

This is a quick, inexpensive change.  Driver::SysRW is the only driver
to be developed/used over the course of like three years.  Make it a
default so people can stop typing it.

This should be an easy fix, but it involves tweaking several wheels.
It might be a nice introduction to wheels for someone who wants to
know more about them.

----------------------------------------
Make Filter::Line the Default for Wheels
----------------------------------------

This is an inexpensive change, but it has dubious value.  Think about
it some more first.

This should be an easy fix, but it involves tweaking several wheels.
It might be a nice introduction to wheels for someone who wants to
know more about them.

----------------------------------------------------
Split Dual-Mode Functions into xyz_set and xyz_clear
----------------------------------------------------

This would make the functions' purposes more clear and eliminate some
branches that are currently tested all the time.  The existing
dual-mode functions could remain as thunks for the single-purpose
ones.

The "June 2001" alarms functions already do this.  What about
select_xyz and the others?

-----------------------------------------
Wheel::ReadWrite Needs Input Flow Control
-----------------------------------------

Torvald Riegel has patched POE::Wheel::ReadWrite to perform throttled
reads, but I'm not satisfied with the code's design.

Torvald wants to do two things:

1. He wants to add pause and resume methods to Wheel::ReadWrite so
   that users can implement their own buffering code.

2. He wants to add two more watermarks for input flow control and have
   ReadWrite buffer data itself.

His version of ReadWrite does this:

1. He added pause_input() and resume_input() methods for convenience.
   They do $poe_kernel->select_(pause|resume)_read() calls on the
   Wheel's filehandle so a second copy of the handle isn't needed.

2. He added an input buffer and high watermark check to ReadWrite's
   "select read" state.  It automatically pauses the wheel's input if
   the high water mark is reached.

3. He added a get() method that returns (in list context) everything
   in ReadWrite's input buffer, or (in scalar context) the next thing
   from the buffer.  It also watches the low water mark and resumes
   the wheel's input if the number of buffered records drops below
   that mark.

Design notes:

Torvald's design requires sessions to actively poll for new records
once the input high water mark is reached.  I think this is too much
work.  Instead, ReadWrite should do its own polling.  FollowTail does
this, for example.

[Torvald replied that he thinks the reading session should definitely
poll for data.  Polling tells the wheel how fast a session wants data
to arrive.  Moreover, the consumer's pace may vary over time.]

At the very least, pause_input() and resume_input() would let people
write their own buffering and flow control.  This requires the most
work on the Session programmer's part, but it gives developers a
chance to start experimenting with flow control.

Observations:

Requests for input throttling in the past have been concerned about
reading from data sources which are almost always ready to be read.
Large plain files, for example, are quickly slurped, preventing queued
events from being dispatched until the files are completely read.
That tends to be bad.  Rather, these sorts of programs want their
events to be spread out over time, giving the event dispatcher a
chance to do other things.

Wheel::ReadLine does input throttling.  Its get() method tells the
wheel that a session is ready to get another line of input.  This is
not a general purpose wheel, though: it only deals with console input.

If there are other uses for input flow control, please discuss them on
the mailing list.

[Torvald says there are a lot of cases where input flow control is
needed.  Any program that needs to transfer data from fast sources to
slower consumers needs input flow control.  He says that ReadWrite
should not make an assumption about line speeds or how much time a
data consumer needs to process data.  He lists: network gateways,
programs reading files, and any socket client/server that wants to
avoid denial of service attacks (floods).  He brings up a good point:
input flow control should almost always be used in important
applications, to avoid acts of malice.]

I'm worried about adding pause_input() and resume_input() if they're
intended as temporary methods for experimenting with flow control.
Once they are implemented, it will be very difficult to remove them.
Should these be considered permanent additions, left undocumented, or
be documented as experimental?

[Torvald reminded me of a filehandle accessor as an alternative to
pause_input() and resume_input().  This might be a better, more
generic solution, but it brings up the possibility that a program make
put a filehandle into a state that isn't compatible with the wheel.

Regardless of what happens on that issue, he would like to see flow
control added to ReadWrite on the first try.]

To do (maybe):

* Add pause_input() and resume_input() from Torvald's version of
  ReadWrite.

* Document pause_input() and resume_input().

* Test pause_input() and resume_input() in [t/???.t]

* Release version ?.??.

------------------------------------------------------
Evaluate Torvald Riegel's Synchronous NFA Design (???)
------------------------------------------------------

Torvald wants POE::NFA to run entirely synchronously like a proper
nondeterministic finite state machine ought to.  I'm against that
because I haven't seen a proper need for it and it puts what I see as
an artificial limit on the class.

To do:

* Distill Torvald's e-mail on the subject and include a summary here.

* Determine whether his design can be implemented in a way that
  doesn't limit NFA to a strictly correct NFA design.  Asynchronous
  NFA events are useful and should also be supported.

------------------------------
Delay Garbage Collection (???)
------------------------------

POE's garbage collection can be slow.  It would be very cool to
schedule this in the "dead" time between events.  Especially in
instances where the next event is a timer and we have a few seconds of
nothing to do.

A lot of things rely on timely garbage collection, though.  _parent
and _child events.  Referential integrity.  Things would be a mess
between _stop and the delayed garbage collection.

Delaying alarm GC might be doable.  We can flag alarms as "dead" in
the ID->time hash and discard them as they come to the front of the
queue.  In idle times when there are no FIFO events and the next alarm
is more than 1/10 second away, we can sweep the alarm queue and pull
out the dead ones.

It will take a fair amount of planning to pull this off correctly.
I'm just documenting it for now so I don't forget.

-----------------------------------
Clean up ASSERT_* and TRACE_* Flags
-----------------------------------

Especially ASSERT_USAGE, ASSERT_RETURNS and TRACE_RETURNS which are
getting silly.  Make this stuff macros, at the very least.

-------------------------------------
Batch or Standalone POE::Preprocessor
-------------------------------------

ActiveState's perlapp (PDK) does not recognize source filters and so
cannot "compile" POE programs into stand-alone applications.  Source
filters may not be available elsewhere, so it would be useful to let
developers statically preprocess applications into stand-alone files.

Let Ilya Melamed know when this is done.  It's his request.

===
EOT
===
