<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">

<article id="deps" lang="en">
  <articleinfo>
    <title>DEPS</title>
    <subtitle>Dependency Extraction and Processing System</subtitle>
    <subtitle>Formerly <emphasis>graph-includes
	toolkit</emphasis></subtitle>

    <author>
      <firstname>Yann</firstname>
      <surname>Dirson</surname>
      <email>ydirson@altern.org</email>
    </author>

    <legalnotice>
      <simpara>This program is free software; you can redistribute it
      and/or modify it under the terms of the GNU General Public
      License, version 2, as published by the Free Software
      Foundation.</simpara>

      <simpara>This program is distributed in the hope that it will be
      useful, but WITHOUT ANY WARRANTY; without even the implied
      warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
      See the GNU General Public License for more details.</simpara>
    </legalnotice>

    <copyright>
      <year>2005</year>
      <year>2006</year>
      <holder>Yann Dirson</holder>
    </copyright>
  </articleinfo>

  <section>
    <title>In short</title>

    <simpara>DEPS is a set of perl libraries which allows to extract
    dependency information from arbitrary material (eg. program source
    files), apply various transformations on this graph to make it
    more readable or put emphasis on one aspect of it, and draw
    it.</simpara>

    <simpara>The <command>graph-includes</command> tool is the current
    command-line interface to DEPS.  It is quite limited, in that it
    only extracts dependency information from files, and applies
    predefined transformations, on which the user has limited
    influence through flags.  A more generic tool is planned, but will
    probably wait until various aspects of the DEPS design
    gets finalized.</simpara>

    <para>Currently available graph transformations are:
      <itemizedlist>
	<listitem><simpara>multi-level grouping of source
	files</simpara></listitem>

	<listitem><simpara>consolidation of groups of different
	levels, effectively allowing to show eg. files not in any
	group</simpara></listitem>

	<listitem><simpara>transitive reduction, drastically reducing
	the number of edges to be drawn</simpara></listitem>

      </itemizedlist>
    </para>

    <simpara>It currently supports graphing the C/C++ #include
    relationship, and to a certain extent perl inter-module
    dependencies, using graphviz or tulip.</simpara>

  </section>

  <section id="notice">
    <title>Important notice</title>

    <simpara>This tool has evolved from a 50-line script written for a
    particular project (<ulink url="http://wesnoth.org/">Battle for
    Wesnoth</ulink>).  Although it has been generalized much, there
    are still somewhat ad-hoc heuristics harcoded here and there,
    especially in the default project class (see class descriptions
    below).</simpara>

    <simpara>Although work is under way to make this tool as generic
    as possible, work still has to be done at all levels.  It is still
    under development, and may not suit your needs (at least, not
    yet).</simpara>

  </section>

  <section id="install">
    <title>Installation instructions</title>

    <simpara>Be sure you have a recent version of Perl installed.  At
    least List::Util is missing from versions earlier than 5.8.  You
    can also just fetch this additional package from CPAN if you
    cannot upgrade.  If you notice that another package is missing
    from your installation, please report it, so it can be listed
    here.</simpara>

    <simpara>Other required perl modules:</simpara>
    <itemizedlist>
      <listitem>
	<simpara><ulink
	url="http://search.cpan.org/~samv/Set-Object/">Set::Object</ulink></simpara>
      </listitem>
    </itemizedlist>

    <para>Installation is like many other perl packages:</para>
    <example>
      <title>Sample install session</title>
      <literallayout><prompt>$</prompt> perl Makefile.PL prefix=/usr/local
<prompt>$</prompt> make
<prompt>$</prompt> su
<prompt>#</prompt> make install</literallayout>
    </example>

    <simpara>Be sure that the directory in which the library modules
    got installed is in your perl library path.  Eg, if
    <command>graph-includes --version</command> does not give the
    expected result, try setting the <envar>PERL5LIB</envar>
    environment variable to (in the above example)
    <filename>/usr/local/share/perl/5.8.4/</filename>.</simpara>

    <simpara>New versions can be found at
    <uri>https://alioth.debian.org/projects/deps/</uri>.</simpara>

    <simpara>A darcs repository is available at <uri
    type="darcs">http://deps.alioth.debian.org/darcs/deps/</uri
    >.</simpara>

    <simpara>To be able to format the produced graphs, you will need
    one of <ulink url="http://www.graphviz.org/">graphviz</ulink> and
    <ulink url="http://www.tulip-software.org/"
    >tulip</ulink></simpara>

  </section>

  <section id="howto">
    <title>How to take advantage of this tool to improve your code</title>

    <simpara>Graph-includes is only a supporting tool for a
    refactoring effort.  It can be useful in helping a developper to
    see where he should put its efforts in order to get cleaner and
    saner dependencies in a project.</simpara>

    <simpara>In this respect, it is quite similar to a microscope: if
    you don't look at the right place, you won't see anything
    interesting.  But if you start with a small magnifying factor, you
    can locate regions of interest, and then zoom on those to get to
    the interesting stuff.</simpara>

    <section>
      <title>On the spirit of dependency cleanup</title>

      <section>
	<title>First look at a dependency graph</title>

	<simpara>When developping a project of medium size (we'll talk
	mostly C/C++ here, but that will apply to most languages),
	expecially with many people writing code, it is quite easy to
	get to a point where each file (out of several tens of
	hundreds of files) depends on too many other files.</simpara>

	<simpara>The most obvious relation is the #include one.  The
	more #includes a file has, the more time it takes to build -
	especially when those included files #include themselves a
	bunch of other files.  For a project of about 100 files, just
	producing a graph of all those files, with arrows representing
	the #include dependencies, will usually give an unreadable
	graph, and will show very little about possible improvements.
	This is why this tool has been written: to make it possible to
	get to the useful information hidden in this unusable
	dependency graph.</simpara>
      </section>

      <section>
	<title>Looking further</title>

	<simpara>A less obvious relation appears more clearly when you
	consider not files by themselves, but the set of files made of
	an interface and the matching implementation.  Let's consider
	two such sets, made of the files a.h, a.c, b.h, b.c.  a.c
	includes b.h, and b.c includes a.h, and each implementation,
	following good practice, includes its own interface.  A simple
	dependency graph as described above would show such a
	graph:</simpara>

	<literallayout class="monospaced"
>a.c -&gt; b.h
   \  /|
    \/
    /   
   /  \|
b.c -&gt; a.h</literallayout>

	<para>If OTOH we represent those sets of files instead of
	the files themselves, we now have something like:</para>

	<literallayout class="monospaced"
	  >a &lt;--&gt; b</literallayout>

	<para>This shows much more clearly that those two modules are
	intrinsicately related.  In many cases, this will express that
	whenever you use the a.o file resulting from the build of a.c,
	you'll need to link b.o as well, and vice versa.  This will be
	the case when each file uses the headers to get function
	prototypes.  Then hunting for abusive dependencies will allow,
	for example, to select with finer grain which of those modules
	of code will need to go into which executable, thus producing
	lighter executables.</para>

	<simpara>Note that such a reciprocal dependency may not be
	pathological.  Many projects tend to split a large module into
	several files for clarity, even when those files are
	inter-dependant.  It is much often in cycles of unidirectional
	dependencies that we find dependencies that should not be
	there.</simpara>

	<simpara>In other cases, headers would just have been used to
	access a type definition from b.h, and the associated b.o
	would not be needed.  In such cases, you may want to consider
	splitting such "low-level" declarations into their own
	headers.  Not only this would simplify the graph, allowing you
	to get a better grasp on your source code, but it can also
	lead to faster compilations, since each file will be able
	include less unrelated definitions.</simpara>

      </section>
    </section>

    <section>
      <title>Tuning the "files" and "includes" parameters</title>

      <simpara>Your first run will surely looks somewhat
      like:</simpara>

<literallayout class="monospaced">graph-includes -o project.ps src/ lib/</literallayout>

      <simpara>You will take care of specifying all directories or
      individual source files that make up your project.</simpara>

      <simpara>In addition to an initial graph in the project.ps file,
      which is quite likely to be incomplete by far, you will find a
      file named project.ps.graph-includes.report.  It is a text file,
      which will help us to finetune ou command-line.  Its first
      section will look something like:</simpara>

      <literallayout class="monospaced"
>General statistics:
-------------------

412 files, 353 nodes (14% dropped)
245 dependencies, 137 edges (44% dropped)
225 leaf node(s)

280 dependencies not found
0 dependencies identified as system headers</literallayout>

      <simpara>As you can see, many dependencies are declared as "not
      found".  What happens is quite similar to running a C compiler
      without any -I flags: most header files are not
      located.</simpara>

      <simpara>We have in graph-includes two different flags to
      specify paths where to look for the dependencies.  -I (aka
      -Include) specifies directories that are part of the project,
      and will allow to find all of our include-style dependencies.
      OTOH, -sysI (aka -sysInclude) specifies system directories;
      included files found in such a directory will of course not
      result in an intra-project dependency, and will add no edge to
      our graph, but will stop being displayed as part of the
      "dependencies not found" count.  Thus, they will help us to see
      how far we are from specifying all the -I flags.</simpara>

      <simpara>Now you will most likely require several iterations of
      adding -I/-sysI flags and checking the results.  But that alone
      may not be sufficient to reach the ultimate "0 dependencies not
      found":</simpara>

      <itemizedlist>
	<listitem>
	  <simpara>multi-platform source often have conditional
	  #include directives, and eg. win32 headers will probably not
	  be located on a Un*x box.</simpara>
	</listitem>
	<listitem>
	  <simpara>some generated files will require the source tree
	  to be configured in some way, or even to be partly or
	  completely built (eg. config.h generated by a "configure"
	  script, or Qt source generated by the meta-object
	  compiler)</simpara>
	</listitem>
      </itemizedlist>

      <simpara>When you are confident that those remaining missing
      dependencies are system headers for other platforms, you can go
      on and look at the graph.</simpara>

    </section>

    <section>
      <title>Possible strategies to help locating abusive
      dependencies</title>

      <simpara>Keeping in mind that we are essentially looking for
      dependency loops, we expect to obtain in then end a graph that
      will be wihout cycles, that is, with all (or, at least, most of)
      arrows pointing from left to right in our graph.</simpara>

      <simpara>Then we will look for those arrows pointing backwards,
      as a sure sign for a cycle.  Remember that if the cycle is not a
      long one, it may be legitimate; only if you judge that some of
      the modules in this cycle are really unrelated, should your
      consider it pathological.  Those backward arrows are not
      necessarily directly pointing to the abusive dependency, but
      they can surely be used to locate the culprit: by finding the
      various cycles of which our backward arrows are part of, and
      checking one by one all the dependencies in those cycles, you
      can bet at least one that, with some work, could be
      cleared.</simpara>

      <simpara>Then, the way to modifications to do are really
      dependant on your code.  Some possibilities include:</simpara>

      <itemizedlist>
	<listitem>
	  <simpara>removing an #include which is not necessary,
	  perhaps remaining from a revious code
	  reorganization</simpara>
	</listitem>
	<listitem>
	  <simpara>splitting a file in two parts, when you can easily
	  split the components of the file into distinct sets.  One
	  productive distinction to look for is to find a couple of
	  really high-level parts, that not all the parts depending on
	  this module would need.  This will most probably be related
	  to the dependency that you found abusive when looking at the
	  cycle in the graph.</simpara>
	</listitem>
      </itemizedlist>
    </section>
  </section>

  <section id="arch">
    <title>Tool architecture</title>

    <section>
      <title>Overview:</title>

      <simpara>Graph-includes was initially developped with only a
      handful of ideas, and then started to grow as I noticed where
      useful things were missing.  That initial phase was useful for
      me to get a grasp on the domain of dependency graphing, and
      provided the ground for a (hopefully) decent design, which still
      has to be completely implemented.</simpara>

      <simpara>Together with blocking issues marked
      <emphasis>+</emphasis> in the TODO list, the implementation of
      this design shall be the goal for a 1.0 release.</simpara>

      <simpara>The planned design is architectured as successive
      layers, all of which should be pluggable to allow a high degree
      of customization.</simpara>

      <literallayout class="monospaced"
>   source locator
         |
         v
  language selector
         |
         v
dependency extractor
         |
         v
graph transformations
         |
         v
      styling
         |
         v
   layout engine
         |
         v
     rendering</literallayout>

      <simpara>We will then be able to consider DEPS as being made of
      a number of parts:</simpara>

      <itemizedlist>
	<listitem>
	  <simpara>core classes and glue, implementing the above
	  design</simpara>
	</listitem>
	<listitem>
	  <simpara>standard or third-party classes, doing the real
	  work</simpara>
	</listitem>
	<listitem>
	  <simpara>command-line and gui tools to allow easy use of the
	  whole</simpara>
	</listitem>
      </itemizedlist>

      <simpara>This will hopefully make it easy for anyone to plug
      their own work at any place in the architecture.</simpara>

    </section>

    <section>
      <title>State of things</title>

      <simpara>Currently, transformations, styling and the renderer
      are properly customizable.  A couple of transformations and
      styles are already available.  The extractors are customizable
      as well, but the currently implemented mechanism is far from
      being generic enough.</simpara>

      <simpara>Language selectors are intermixed with extractors.  The
      current source locator is a local-tree one (alternatives would
      include SCM-aware locators), is hardcoded in the graph-includes
      script, and takes parameters from per-language extractors to
      find files.</simpara>

      <simpara>There is no distinction (yet ?) between the layout
      engine and the renderer.  In fact, it may not be easy to do
      this, since most layout engines are tied to a particular
      renderer.</simpara>

    </section>
  </section>

  <section id="usage">
    <title>Command-line usage for <command>graph-includes</command></title>

    <simpara>See "graph-includes --help".</simpara>

    <simpara>Please note that this tool does not allow do access the
    full power and flexibility of DEPS.  A new one, to be called
    <command>deps-grapher</command> is planned for subsequent
    releases, but not enough of the new design has been
    implemented yet to start working on this new tool.</simpara>

    <section>
      <title>output type</title>

      <simpara>The default output is a .dot file on standard output,
      suitable for formatting by dot (from the graphviz toolkit), or
      interactive editing by dotty (also from graphviz).
      Alternatively, a graph file for the Tulip graph visualizer can
      be generated instead using "--renderer=tulip".</simpara>

      <simpara>You can ask graph-includes to do the formatting for
      you, eg. using "--output=&lt;file&gt;.&lt;suffix&gt;".  It will
      run "dot -T&lt;suffix&gt;", so that "--output=mydeps.ps" or
      "--output=mydeps.jpg" will have the expected behaviour.  If your
      suffix is not known to dot, it will complain itself, so asking
      for --output=foo.bar will cause a message like:</simpara>

      <simpara>Warning: language bar not recognized, use one of: canon
      cmap cmapx dia dot fig gd gd2 gif hpgl imap ismap jpeg jpg mif
      mp pcl pic plain plain-ext png ps ps2 svg svgz vrml vtx wbmp
      xdot</simpara>

      <simpara>If you intend to print the result on paper, the default
      layout will likely be too large.  You can use --paper=a4 to
      select parameters that will produce a smaller graph and spilt it
      into pages.  This flag also changes the default output format to
      postscript.  Be warned that dot may not honor the page-splitting
      parameter for all output formats.</simpara>

      <simpara>Since the transitive reduction can take time, you may
      like the --verbose switch, which will show a progress
      bar.</simpara>

    </section>

    <section>
      <title>what to draw</title>

      <simpara>The files to be analyzed are given as non-option
      arguments, and can be explicitely specified, or found by
      recursing in directories.  Eg, to analyse foo.c in current
      directory, as well as all C/C++ files in the src/ directory,
      use:</simpara>

<literallayout class="monospaced">$ graph-includes foo.c src/</literallayout>

      <simpara>When an directory argument is specified, it is searched
      for files whose name matches a specific regexp pattern, whose
      default value depends on the specified language (see --language
      below).  This pattern can be overriden using the --fileregexp
      option.  Eg, to match in addition to .c and .h files, those with
      an additional .tmpl suffix, you could write:</simpara>

<literallayout class="monospaced">$ graph-includes -I src -fileregexp '\.[ch](\.tmpl)?$' src/</literallayout>

      <simpara>How dependencies get extracted from the source files
      depend on the language used in those files.  You can specify it
      with the --language flag.  Default value is C (which should also
      be used for other languages based on the C preprocessor, like
      C++).  There is also some partial support for perl - see
      comments in lib/graphincludes/extractor/perl.pm for more
      details.</simpara>

      <simpara>In order to tell the #include resolver where to look
      for included files, you can use the cpp-like -I (aka. --Include)
      flag.  Eg:</simpara>

<literallayout class="monospaced">$ graph-includes -I src src/</literallayout>

      <simpara>Dependencies not found in the project (ie. files
      appearing in #include but not given on command-line) are listed
      as "not found" in the graph-includes.report file for diagnostics
      purposes, unless they are found in a system directory.  System
      directories are declared in a similar fashion, with the
      --sysInclude option.  Eg:</simpara>

<literallayout class="monospaced">$ graph-includes -I src -sysI /opt/foo/include src/</literallayout>

      <simpara>Language extractor have some knowledge about default
      system include dirs: the C extractor knows about /usr/include,
      and the Perl extractor asks perl itself.</simpara>

      <simpara>To avoid having useless information on the graph,
      --prefixstrip=&lt;prefix&gt; can be used to avoid repeating a
      given prefix in all node labels.  Typically:</simpara>

<literallayout class="monospaced">$ graph-includes --prefixstrip=src/ src/</literallayout>

    </section>

    <section>
      <title>how to draw</title>

      <simpara>Files and their inter-dependency build up the first of
      all graphs, named <emphasis>files</emphasis>.  Transformations
      can then be applied to such graphs, producing new graphs.
      Production of those new graphs define a <emphasis>transformation
      graph</emphasis>, whose nodes are our graphs, and whose edges
      denotes which graphs were used in producing which other
      graphs.</simpara>

      <simpara>Typically, files will be grouped in a hierarchy of
      groups.  "Level 0 groups" typically containing just one file,
      are the nodes of the <emphasis>files</emphasis> graph.  Groups
      hierarchies are defined by the selected project class, selected
      by the --class=&lt;class&gt; option.  See below for descriptions
      of the project classes available by default, and for
      instructions to write customized project classes.</simpara>

      <note>
	<simpara>In graph-includes 0.11 and earlier, when a given node
	of level <emphasis>n</emphasis> is not part of a group at
	level <emphasis>n+1</emphasis>, it still appears at that
	level, as if it was the only member of a group.  This is not
	the case any more.  To achieve a similar result in DEPS 0.12,
	the relevant graphs must be merged using the
	<emphasis>consolidate</emphasis> transformation.</simpara>
      </note>

      <simpara>The range of group levels to be drawn is selected with
      --consolidate=&lt;min&gt;-&lt;max&gt;, which defaults to 1-1.
      Eg, for class "default", whose group levels are defined
      as:</simpara>

      <glosslist>
	<glossentry>
	  <glossterm>0</glossterm>
	  <glossdef>
	    <simpara>one file per group</simpara>
	  </glossdef>
	</glossentry>
	<glossentry>
	  <glossterm>1</glossterm>
	  <glossdef>
	    <simpara><filename>what/ever.*</filename> go into a
	    <filename>what/ever</filename> group (usually interface +
	    implementation)</simpara>
	  </glossdef>
	</glossentry>
	<glossentry>
	  <glossterm>2</glossterm>
	  <glossdef>
	    <simpara><filename>what/*</filename> go into a
	    <filename>what</filename> group, supposing top-level
	    directories denote modules of some sort</simpara>
	  </glossdef>
	</glossentry>
      </glosslist>

      <simpara>Group levels below "min" or above "max" are not
      displayed as nodes.  If a file is not a member of any group
      between "min" and "max" levels, it will simply not be
      represented.</simpara>

      <simpara>Another way of using the grouping feature is to color
      nodes according to the group(s) they belong to, using a
      class-defined color scheme, possibly modified by
      <userinput>--color
      &lt;n&gt;:&lt;label&gt;=&lt;color&gt;[,&lt;label&gt;=&lt;color&gt;&#8230;]</userinput>
      options, where &lt;n&gt; is the group level in which the group
      name &lt;label&gt; will receive a background of the specified
      color, which can be defined either by a named X11 color (like
      "blue" or "palegreen"), or by a RGB color using the standard X11
      "#RRGGBB" syntax.</simpara>

      <simpara>The number of grouping levels to be colored is limited
      by the renderer to be used.  As of 0.11, the dot renderer only
      supports coloring 2 group levels.  Groups of a lower level than
      the minimal level requested to --consolidate cannot be colored, for
      obvious reasons.</simpara>

      <simpara>For those wanting to see what edges the transitive
      reduction dropped, the --showdropped will add them to the graph
      in a different color.  Be prepared for your computer room to get
      a noticeable temperature increase for anything else than a small
      set of files with only few dependencies.</simpara>

      <simpara>OTOH, <option
      >--focus=<replaceable>node-label</replaceable ></option> will do
      the same, but only for the dependencies of a specified node.
      That should prevent the nasty effects described above, and will
      be useful for various purposes, including debugging the
      transitive reducer.</simpara>

      <simpara>People still getting cold may also like to circumvent
      the transitive-reduction engine completely, using --alldeps.
      The author assumes no responsibility for losses of mental health
      induced by trying to make any serious use of the resulting
      graph.</simpara>

    </section>
  </section>

  <section id="classes">
    <title>Existing plugins</title>

    <section>
      <title>Transformations</title>

      <section>
	<title>CompatGroup</title>

	<simpara>This transformation is a temporary one, allowing to
	use existing project classes (defining groups through a
	filelabel() method), to be used in the new design.</simpara>
      </section>

      <section>
	<title>Consolidate</title>

	<simpara>Standard group transformations may not include all
	nodes from a graph into group nodes.  Thus, a group graph does
	not as such describe all the files fed into the tool.</simpara>

	<simpara>This transformation will merge graphs describing
	different grouping levels, so that each node from the
	lowest-level graph requested will be represented by a node in
	the result graph: if it is part of a group, the highest-level
	group of which it is a member will be appear, if not the node
	itself will appear instead.</simpara>
      </section>

      <section>
	<title>TransitiveReduction</title>

	<simpara>Dependency graphs are often hard to read because of
	the large number of edges.  Since the depndency relation can
	be seen as being transitive (if A depends on B and B on C,
	then A depends on C), we can make the graph more readable by
	not drawing those edges which the transitivity property makes
	redundant.  This is what this transformation achieves.</simpara>
      </section>
      
    </section>

    <section>
      <title>Styles</title>

      <section>
	<title>Node styles</title>

	<section>
	  <title>PerGroup</title>

	  <simpara>Allows to set style attributes (eg. color) based on
	  group membership.</simpara>
	</section>

	<section>
	  <title>GroupStats</title>

	  <simpara>Adds in group nodes some statistics: the number of
	  nodes that make up this group, and the number of edges
	  between those member nodes (ie. dependencies internal to the
	  group).  Note that this is relative to the group-level
	  immediately below in the group hierarchy, NOT to the files
	  themselves - parameters will be added later to makes this
	  more customizable.</simpara>
	</section>
      </section>

      <section>
	<title>Edge styles</title>

	<section>
	  <title>WeightLabel</title>

	  <simpara>Adds a label on edges, showing how many
	  dependencies between the lowest-level nodes (eg. effective
	  number of include relationship between files) are
	  represented by each edge.  Contrast this with the
	  intra-group counts currently reported by GroupStats on
	  nodes.</simpara>
	</section>
      </section>
    </section>

    <section>
      <title>Project classes</title>

      <note>
	<simpara>The concept of <emphasis>project classes</emphasis>
	will most probably be replaced by <emphasis>project
	files</emphasis> in the near future, as functionality
	currently located there will move into more adequate
	areas.</simpara>
      </note>

      <section>
	<title>class "default"</title>

	<simpara>As implied by its name, it is the one which will be
	  used unless you use the --class option.  Although it is the
	  default one, it may still be quite rough at the moment, still
	  using some ad-hoc heuristics, and will be improved in the near
	  future.  Here are its main characteristics:</simpara>

	<itemizedlist>
	  <listitem>
	    <simpara>looks at C-style #include lines</simpara>
	  </listitem>
	  <listitem>
	    <simpara>creates level-1 groups for all files sharing the same
	      path and (disregarding the suffix) filename.  Eg, files
	      "foo/bar.c" and "foo/bar.h" would be grouped in a "foo/bar"
	      level-1 group.  In clear, it won't connect include files if
	      they are all located in an include/ directory.</simpara>
	  </listitem>
	  <listitem>
	    <simpara>creates by-directory level-2 groups.  Eg. in the
	      above example, a group "foo" would exist at level-2.</simpara>
	  </listitem>
	</itemizedlist>
      </section>

      <section>
	<title>class "uniqueincludes"</title>

	<simpara>Built on top of the default class, it is meant for
	  projects where file names are kept unique across all directories.
	  If the ad-hoc #include processing of the default class does not
	  suit your project, it is the only out-of-the-box alternative
	  available today.  Here are its main characteristics:</simpara>

	<itemizedlist>
	  <listitem>
	    <simpara>provides a single grouping level based on filenames,
	      disregarding all the directory hierarchy.</simpara>
	  </listitem>
	</itemizedlist>

	<simpara>Note that it is not meant for general use, as:</simpara>

	<itemizedlist>
	  <listitem>
	    <simpara>it will group any files with the same name in the
	      same level-0 group, possibly causing confusion.</simpara>
	  </listitem>
	  <listitem>
	    <simpara>it does not make any directory name appear in the
	      node names</simpara>
	  </listitem>
	</itemizedlist>

      </section>
    </section>

  </section>

  <section id="examples">
    <title>Examples of use</title>

    <section>
      <title>Pure command-line examples</title>

      <section>
	<title>Graphing graph-includes itself:</title>

	<literallayout class="monospaced">$ ./graph-includes -lang perl -I lib -prefixstrip lib/ -o deps.ps graph-includes lib/</literallayout>

	<simpara>graph-includes does not know in advance which classes
	  it will use</simpara>
      </section>

      <section>
	<title>Rather clean ones:</title>

	<simpara>a rather clean dependency graph</simpara>

	<literallayout class="monospaced">Maelstrom-3.0.6$ graph-includes -v -sysI /usr/include/SDL -I . -I ./netlogic -I ./maclib -I ./screenlib -o deps.ps .</literallayout>

	<simpara>more work has to be put in the wesnoth example class:</simpara>

	<literallayout class="monospaced">wesnoth-0.9.1$ graph-includes -v --class wesnoth --consolidate 1-1 -sysI /usr/include/c++/3.3 -sysI /usr/include/SDL --prefixstrip src/ -I src -o deps.ps src/</literallayout>

      </section>

      <section>
	<title>Examples only here as a reminder to write proper project
	  classes for them</title>

	<para>needs supporting features for multi-arch source trees:</para>

	<literallayout class="monospaced">qemu-0.7.0$ graph-includes -v -sysI /usr/include/SDL $(find -name CVS -prune -o -type d -printf <emphasis>-I %p\n</emphasis>) -o deps.ps .</literallayout>

	<para>needs proper file-grouping:</para>

	<literallayout class="monospaced">mesag-6.2.1$ graph-includes -o -I ./include -I ./include/GL -I ./src/mesa -I ./src/mesa/main -I ./src/glu/sgi/include -I ./src/glu/sgi/libnurbs/internals -I ./src/mesa/glapi -o deps.ps .</literallayout>

      </section>
    </section>

    <section>
      <title>Customization examples</title>

      <simpara>See graphincludes::project::wesnoth in the examples/
      dir as an example of a custom project class.</simpara>

      <simpara>Keep in mind that the API is not frozen yet, and will
      probably be overhauled more than once before an official API
      gets blessed.</simpara>

    </section>
  </section>

  <section id="caveats">
    <title>Caveats</title>

    <itemizedlist>
      <listitem>
	<simpara>This toolkit only handles explicitely-declared
	dependencies, it won't detect it if eg. a prototype was
	cut'n'pasted instead of using the correct #include, but you
	shouldn't do that anyway :)</simpara>
      </listitem>

      <listitem>
	<simpara>Dynamic dependencies computed at runtime cannot be
	detected by current plugins.  This explains why the graph for
	DEPS itself is so particular.</simpara>
      </listitem>
    </itemizedlist>
  </section>

  <section id="related">
    <title>Related tools</title>

    <simpara>A couple of tools may prove useful to be integrated with
    DEPS.  Here are some ideas.</simpara>

    <itemizedlist>
      <listitem>
	<itemizedlist>
	  <title>Through DEPS plugins</title>
	  <listitem>
	    <simpara>openc++ (http://opencxx.sourceforge.net/), which
	      provides a C++ parser library (still bad support for templates
	      in 2.8, IIRC)</simpara>
	  </listitem>
	  <listitem>
	    <simpara>synopsys (http://synopsis.fresco.org/), a
	      multi-language source code introspection tool</simpara>
	  </listitem>
	  <listitem>
	    <simpara>sparse
	      (http://kernel.org/git/?p=devel/sparse/sparse.git;a=summary),
	      Linux Torvald's C semantic parser</simpara>
	  </listitem>
	</itemizedlist>
      </listitem>
      <listitem>
	<simpara>OpenDX (http://www.opendx.org/), IBM's Visualization
	Data Explorer</simpara>
      </listitem>
    </itemizedlist>

    <simpara>I finally found a couple of tools out there, from which I
    may borrow ideas some day.  I'd be happy to hear about more of
    them.</simpara>

    <itemizedlist>
      <listitem>
	<simpara>cinclude2dot, originally from Darxus
	(http://www.chaosreigns.com/code/cinclude2dot/), then taken
	over by F. Flourish (http://www.flourish.org/cinclude2dot/) is
	a GPL C/C++-only tool, which apparently has support for
	grouping, but not for transitive reduction.  Should I have
	searched better, and found it a couple of months ago, maybe
	graph-includes would have never been developped :)
	</simpara>
      </listitem>
      <listitem>
	<simpara>http://www.tarind.com/depgraph.html has a dependency
	grapher for python, without transitive reduction as well.  It
	does however allow customisation of project classes, somewhat
	similar to graph-includes.</simpara>
      </listitem>
      <listitem>
	<simpara>OptimalAdvisor
	(http://javacentral.compuware.com/pasta/) is a refactoring
	tool, which goes far beyond simple dependency analysis, but is
	non-free/libre/open-source (also they have a
	functionally-limited free/gratis edition) and seems to support
	only java.</simpara>
      </listitem>
      <listitem>
	<simpara>codeproject.com has some VisualStudio(tm) plugins
	targetting C++, which I cannot test, but appear to scale badly
	for large projects
	(http://www.codeproject.com/csharp/DependencyGraph.asp).</simpara>
      </listitem>
    </itemizedlist>
  </section>

  <section id="todo">
    <title>TODO</title>
    <section>
      <title>general</title>

      <itemizedlist>
	<listitem>
	  <simpara>continue merging the verbose/debug behaviour into
	  the global report file</simpara>
	</listitem>
	<listitem>
	  <simpara>continue moving classes into DEPS namespace as the
	  API stabilizes</simpara>
	</listitem>
	<listitem>
	  <simpara>finalize filename portability support, using
	  File::Spec volume information (or, possibly better but using
	  non-core module, using Path::Class)</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>core engine</title>

      <itemizedlist>
	<listitem>
	  <simpara>allow to associate attributes to files (eg. an ARCH
	  attribute for multi-architecture trees, like kernels,
	  development tools and emulators)</simpara>
	</listitem>
	<listitem override="+">
	  <simpara>modularization (finish the restructuring into a
	  cleaner and more modular design)</simpara>
	</listitem>
	<listitem>
	  <simpara>allow passing options to modules (-O param=value
	  ?)</simpara>
	</listitem>
	<listitem>
	  <simpara>allow to define several views in a project,
	  several of which can be generated by default.</simpara>
	</listitem>
	<listitem>
	  <simpara>find out whether we can declare
	  protocols/pure-virtual-classes in some way, to cleanup the
	  class graph</simpara>
	</listitem>
	<listitem>
	  <simpara>generalize --prefix-strip</simpara>
	</listitem>
	<listitem>
	  <simpara>give consistent access to all commonly-needed
	  features through command-line and class
	  customization</simpara>
	</listitem>
	<listitem>
	  <simpara>Maybe allow to use as nodes other objects than
	  files (eg. URI objects ?), for ultimate
	  generalization.</simpara>
	</listitem>
	<listitem>
	  <simpara>Provide a mechanism for any transform to record
	  informations for the report file.</simpara>
	</listitem>
	<listitem>
	  <simpara>Allow to plug verifiers (probes ?) in a clean way
	  (eg. verify the transitive reduction, or any consistency
	  check)</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>graph-includes tool</title>

      <itemizedlist>
	<listitem>
	  <simpara>allow to run from a build directory</simpara>
	</listitem>
	<listitem>
	  <simpara>--class does not allow to find the project file
	  (need to set PERL5LIB)</simpara>
	</listitem>
	<listitem>
	  <simpara>caller must prepend path to source tree to
	  --prefixstrip and all relative -I flags</simpara>
	</listitem>
	<listitem override="+">
	  <simpara>find the accessory classes as easily as possible
	  (like bugzilla ?)</simpara>
	</listitem>
	<listitem>
	  <simpara>better robustness to incorrect arguments
	  (eg. --consolidate 1:2)</simpara>
	</listitem>
	<listitem>
	  <simpara>automate --help production (see Pod::Usage
	  ?)</simpara>
	</listitem>
	<listitem override="+">
	  <simpara>multi-sheet paper support may be broken</simpara>
	</listitem>
	<listitem>
	  <simpara>use an existing source of paper formats (libpaper,
	  LC_PAPER, whatever)</simpara>
	</listitem>
	<listitem>
	  <simpara>maybe use graphviz' tred(1) to check our transitive
	  reductions, and/or as alternative transform plugin.</simpara>
	</listitem>
	<listitem>
	  <simpara>autodetection of the language to use based on
	  filenames</simpara>
	</listitem>
	<listitem>
	  <simpara>provide an initial list of system directories to
	  avoid repeating them (ask compiler)</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>extractors</title>
      <itemizedlist>
	<listitem override="+">
	  <simpara>allow -I syntax for programs using eg. -I. from
	  source subdirectory</simpara>
	</listitem>
	<listitem>
	  <simpara>write other extractors (java, python,
	  &#8230;)</simpara>
	</listitem>
	<listitem>
	  <itemizedlist>
	    <title>C-like extractor</title>
	    <listitem>
	      <simpara>some support for CPP symbol conditionals
	      (mostly #ifdef), perhaps coupling this with
	      attributes</simpara>
	    </listitem>
	    <listitem>
	      <itemizedlist>
		<title>write an openc++-based dependency
		extractor</title>
		<listitem>
		  <simpara>extract more fine-grained dependency
		  (depending on a header does not necessarily imply
		  depending on code)</simpara>
		</listitem>
		<listitem>
		  <simpara>handle (warn about) the case where the
		  declarations for a given implementation file are
		  scattered in more than one header</simpara>
		</listitem>
	      </itemizedlist>
	    </listitem>
	  </itemizedlist>
	</listitem>
	<listitem>
	  <simpara>detect undeclared dependencies (eg. manually
	  inserted prototypes)</simpara>
	</listitem>
	<listitem>
	  <simpara>check necessity of declared includes</simpara>
	</listitem>
	<listitem>
	  <itemizedlist>
	    <title>perl extractor</title>
	    <listitem>
	      <simpara>remove arbitrary limitations</simpara>
	    </listitem>
	    <listitem>
	      <simpara>report use/require lines we could not
	      completely parse</simpara>
	    </listitem>
	    <listitem>
	      <simpara>do some invariant analysis when importing a
	      module using a variable name.  Investigate drawing 1->n
	      links in this case.</simpara>
	    </listitem>
	  </itemizedlist>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>project classes</title>

      <itemizedlist>
	<listitem>
	  <simpara>proper way to define include paths in project
	  class</simpara>
	</listitem>
	<listitem>
	  <simpara>make default project-class consider multiple levels
	  of directories as group levels, but only if they
	  (consistently ?) have multiple subgroups ?</simpara>
	</listitem>
	<listitem>
	  <simpara>write a linux-kernel class and others as examples
	  :)</simpara>
	</listitem>
	<listitem>
	  <simpara>provide a simple hash-based filelabel
	  implementation</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>grouping</title>

      <itemizedlist>
	<listitem>
	  <simpara>Abstract the grouping process into GroupHierarchy
	  objects</simpara>
	</listitem>
	<listitem>
	  <simpara>Provide a generic regexp-based grouper</simpara>
	</listitem>
	<listitem>
	  <simpara>provide tools for automatic grouping (eg. using
	  cycles, or selected external deps, or from leaves)</simpara>
	</listitem>
	<listitem>
	  <itemizedlist>
	    <title>keep group members as close as possible</title>
	    <listitem>
	      <simpara>(style) give more weight to intra-group edges</simpara>
	    </listitem>
	    <listitem>
	      <simpara>Improve transitive reductions implying multiple
	      edges into a single cycle, to prefer an edge into the
	      lowest-level common group</simpara>
	    </listitem>
	  </itemizedlist>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>styling</title>

      <itemizedlist>
	<listitem>
	  <simpara>specify restrictions on allowed deps at a given
	  grouping level, and show violations in red.</simpara>
	</listitem>
	<listitem>
	  <simpara>allow styling through font color, node shape
	  (dot/tulip), number of peripheries (dot)</simpara>
	</listitem>
	<listitem>
	  <simpara>allow to draw non-consecutive group levels
	  (eg. --consolidate 1,3) (tool issue to be resolved by
	  deps-grapher)</simpara>
	</listitem>
	<listitem>
	  <simpara>allow different node shapes when mixing high-level
	  nodes with lower-level ones through the default singleton
	  groups (Node::PerGraph style, similar to PerGroup)</simpara>
	</listitem>
	<listitem override="+">
	  <simpara>optionally show labels (using attributes ?) for
	  node ingredients and color edges according to them</simpara>
	</listitem>
	<listitem>
	  <simpara>optionally show external deps (deps on files not on
	  command-line)</simpara>
	</listitem>
	<listitem>
	  <simpara>limit graph to one or more given group(s) of files
	  (specified by &lt;level&gt;:&lt;label&gt;)</simpara>
	</listitem>
	<listitem>
	  <simpara>draw cycles in a given color</simpara>
	</listitem>
	<listitem>
	  <simpara>draw a specific path</simpara>
	</listitem>
	<listitem>
	  <simpara>provide automatic coloring schemes</simpara>
	</listitem>
	<listitem>
	  <simpara>color intra-group edges with the same color as
	  nodes (post-processing ?)</simpara>
	</listitem>
	<listitem>
	  <simpara>allow to request drawing of who in a high-level
	  node points to another node (ie. violates some
	  constraint)</simpara>
	</listitem>
	<listitem>
	  <simpara>propagate excuses in some way when they are dropped
	  by the transitive reducer</simpara>
	</listitem>
	<listitem>
	  <simpara>investigate candidate tools for hyperbolic layout
	  ?</simpara>
	</listitem>
	<listitem>
	  <simpara>allow to show the count of deps in a given edge
	  using line width instead of labels</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>documentation</title>
      <itemizedlist>
	<listitem>
	  <simpara>write more documentation</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>testsuite</title>

      <itemizedlist>
	<listitem>
	  <simpara>write a testsuite.</simpara>
	</listitem>
	<listitem>
	  <simpara>ensure that all provided non-abstract classes are
	    self-contained</simpara>
	</listitem>
      </itemizedlist>
    </section>

    <section>
      <title>gui</title>

      <para>The standard GUI should be able to navigate the project
      definition, visualizing the hierarchy of groups, (un)folding
      groups and edges, displaying single culprits from a group, and
      anything you can think of.</para>

      <para>For an engine, maybe with graphviz' lefty, or write a
      specialized tulip gui ?  A self-customizable GUI like entity may
      be a good idea.  Since entity has support for OpenGL areas,
      maybe it can be made to embed tulip graphs.</para>

    </section>
  </section>

  <section id="bugs">
    <title>Known bugs</title>

    <itemizedlist>
      <listitem>
	<simpara>volume names in paths are not handled yet (eg. on
	windows)</simpara>
      </listitem>
      <listitem>
	<simpara>on windows, backslash path separator in -prefixstrip
	argument interferes with regexp ("Trailing \ in regex m/ ... at
	wesnoth.pm line 21")</simpara>
      </listitem>
      <listitem>
	<simpara>(is that really a bug, or just a normal side-effect
	of cycles ?) traversal counts on edges are unreasonably high
	on cycles (real graph traversal issue, but unlikely to get
	fixed before we get another visual way to spot cycles
	;)</simpara>
      </listitem>
      <listitem>
	<simpara>the colored style of a node of level &lt; min is not
	shown when that node is displayed because it is not part of
	any node &gt; min and &lt; max. (is that still true now that
	we use transforms and the like ?)</simpara>
      </listitem>
      <listitem>
	<simpara>--showdropped mode draws too many edges as dropped
	(ie. does not consider marked edges as dropped when deciding
	whether to consider subsequent edges as dropped)</simpara>
      </listitem>
      <listitem>
	<simpara>transitive reduction may not be complete, some more
	edges could possibly be dropped - wesnoth tree at 2005-03-25
	exhibits the problem with the "display -&gt; builder -&gt;
	animated -&gt; image" path</simpara>
      </listitem>
    </itemizedlist>
  </section>

</article>
