- graceful shutdown on kill (i.e. disconnect all clients, disconnect all
  connections to FTP servers

- Maybe should be server that can have commands entered at STDIN for debugging
  and stats info dumps to STDOUT or a log file.

- perhaps add signals to dump stats on server connection queue as well as
  ftp cache

- Protocol stuff:
	client will send first an action code saying what it wants done
	it then sends necessary parameters for that action
	end of parameter field indicated by .<CR><LF>
	individual functions for each call necessary to correctly parse
		character stream into parameters for liburl call
	end of message indicated by !<CR><LF>
	server parses out message into parameter blocks and makes URL call
	parse fields of URL struct, need to separate, try using \<CR><LF>

- !!! Make socket library calls for this and ftp code...getting conflicts and
  looks sloppy as all hell to dup code
	- Find all sock code and urld code that could be "utils" and put
	  in utils lib
	- make a urld_utils.h in ../include

- Reallocation woes with buffers in AddParam and GetParam

- Client stub will not pass buffers to server (no shit), but hold buffer,
  request data from server, and then fill buffer for caller

- to make it crash...first open URL, then Retrieve, then Read, dies in
  ccache while retrieving sock...dies in CompareURL for 1st argument...
  all other DEBUG's removed.

Organization:

harvest/src/common/url
	- liburl.a uses the connection cache to parse and retrieve URLs.
	  contains all of the ccache code. clients include url.h

harvest/src/gatherer/urld
	- liburld.a is the stub version of the URL library that connects
	  to the urld process on the current host.  Clients include urld.h
		
urld - NOTES

- directories
	./daemon - holds code for daemon process (main server loop, etc)
	./util - file d_util.c added, holds common code for urld and client stub
	./cstub - code for stub calls, compiles into library called
		  libstub.a inside of ./lib
	./ccache - changed SockGetData() call to accept a URL struct
		   instead of a url in a character buffer.  Eliminates
		   need for ccache to use liburl calls to parse
		   data into URL structs for ftp retrieval
	./url - ftp cache integrated, file url.c changed so ftp retrievals
	    	use cache.  Two calls added - one to init cache and one
		to destroy cache.
	./include - .h's necessary for compilation
	./test_driver - holds a test driver for client stub code.  Given a 
			url, it allows you to read, open, close, and
			retrieve url via urld.  urld must be running on
			same machine in order for it to function.

- Points of interest
	./daemon - No clean exit from server currently exists.  Just kill
		   it.  A method for allowing clean shutdown needs to be
		   added (either commands issued from STDIN, or signal)
	./cstub - You can control if the client will utilize the client
		  connection queue in urld.  urld queues all client 
		  connections so they don't have to be reestablished after
		  first use.  However, this will require any code using the
		  urld stub to add a call for closing the connection after
		  completion, rather than having it transparently compile
		  into existing apps using liburl.

- BUGS!!!
	1.) FTP connection cache integration doesn't work quite right.  If you
	try two consecutive operations that require use of the cache
	(for example, if using the driver in ./test_driver, first 
	retrieve the URL, then try and read it, urld will segmentation
	fault).  This has been traced in the second use of the cache
	down to a function called CompareURL in ./ccache/hash.c.  If you
	look in the function you will see printf's if DEBUG is defined.
	One of the parameters (I believe the first one) causes the
	fault.  I think this is somehow due to the fact that all the major
	data structs of the ccache are global (the hash table and expiration
	lists are defined at the top of hash.c).

	2.) No clean shutdown of urld.  Need to add support for either signal
	or STDIN so it may shut down cleanly

	3.) All sockets are blocking I/O.  This can cause a problem if the
	server is waiting for the end of a data stream (indicated by 
	'!<CR><LF>') and the client crashes (or vice versa).  The server
	will block forever looking for '!<CR><LF>'.  Some kind of time out
	needs to be added.  NOTE that the connection cache is already using
	sigalrm, so that is probably not an option.

	4.) Not really a bug, but:  some Makefiles are kinda hacked together...
	I also noticed cc is used for some libs, and gcc for others.  Some kind
	of uniform methods for making should be developed.

	5, 6, 7...yet to be discovered, but I'm sure they're there.	

-David Merkel, June 1994
