
                                  SimControl

                             Version 1.1, January 2005


------------------------------------------------------------------------------------

Copyright @ 2004 by Jeff Miller <miller@psy.otago.ac.nz>.  All rights reserved.

Permission is granted to anyone to use this software for any purpose. If you
use this software, it would be appreciated if you would cite or acknowledge it,
but that is not required.

DISCLAIMER: THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DAMAGES ARISING IN ANY WAY FROM THE
USE OF THIS SOFTWARE.

Redistribution of this program and its documentation is permitted provided that
the redistribution retains: (a) the above notice of authorship and copyright,
(b) the above permission statement and disclaimer; and (c) this redistribution
condition. 

------------------------------------------------------------------------------------

Purpose:

This collection of programs (exe and bat files) is designed to manage the
running of a number of tasks (e.g., statistical simulations) across one or more
computers.  Each task is carried out by a batch file, and that batch file is
run in a cmd window without operator intervention.  No more than one task will
run at a time on a single computer.  All tasks are run at low priority, so
people using a computer for other purposes generally notice little if any
degradation of its performance.

------------------------------------------------------------------------------------

Overview:

- The user prepares a list of the tasks to be carried out in a file called
  TaskList.Txt.  Each task is associated with a batch file that will run it.
- The user starts the program Master.Bat on one computer.  This program
  assigns new tasks to one or more free slave computers as they come online
  or finish their previous tasks.
- The user starts the program Slave.Bat on each slave computer.  This program
  sets up the slave computer to request tasks from the master computer and
  to perform those tasks when they are assigned to it.
- The same computer can serve as both the Master and a Slave.
- If there are multiple Slave computers, they must all have access to a
  shared directory (i.e., on a network).  The path to this directory is called
  the MasterPath.
- Optionally, the user can set up the files so that email notifications will be
  sent (a) each time a simulation is finished, and/or (b) when there is at
  least one idle computer needing a new task assignment.

------------------------------------------------------------------------------------

Summary of steps.  For detailed instructions, see below:

0 Set up the programs for simulation management (one time only).
1 Set the paths in SetPaths.Bat
2 Create a TaskList.txt file listing the individual tasks.
3 If batch files or templates are used in step 2, it will be necessary to
  create the batch or template files to carry out the individual tasks.
4 Reset or delete TaskList.Dat
5 Start Master.bat on any one machine.
6 Start Slave.bat on each slave machine.

------------------------------------------------------------------------------------

Step 0: Detailed step-by-step instructions for initial setup.

These steps only have to be done for the first set of simulations. After they
have been done once, everything remains in place (unless you delete the
files and directories set up in this step, of course).

(Step 0.a) Make a master directory, referred to herein as the MasterPath
directory. If you are doing simulations on several machines, this should be a
network directory that they all can see.  For example, I use the directory
L:\Sims\Control for this purpose. Copy all of the files that came in
SimControl.zip into this MasterPath directory.

(Step 0.b) This step is optional.  You can skip it if you do NOT want the
simulations to send you any emailed notifications.

If you do want such notifications, you need to find the program called "blat"
on the web (www.blat.net, as of Nov 2004) and install it on the master computer
and every slave computer. As instructed in the readme file that comes with
blat, it is installed by copying the exe file to any directory in the path and
then running it once with a command like:
   C:> blat -install YOUR.EMAIL.SERVER YOUR.EMAIL@ADDRESS
where you fill in the appropriate info for the stuff in upper case.

After installing blat, you need to modify the files SlaveRun.bat and/or
MasterLoop.bat as described within those files. Look for the lines marked ***
in each of those files for further instructions.

(Step 0.c) Make a slave directory on a local disk of each computer.  For
example, I use the directory C:\Sims\Control for this purpose.  Copy the files
S*.bat from the MasterPath directory into the slave directory.  This copying
can be done simply by running the batch file NewSlave on each slave computer,
starting it within the network-visible master directory, after SetPaths
has been set.

(Step 0.d) Make sure that an environment variable "ComputerName" is defined for
each slave machine.  **Be sure to use a name containing only characters that
are allowed in file names.**  and **Make sure that every slave has a different
ComputerName**.
   Under Windows 2000 and XP, you can check the computer name by going to the
Start button => Settings => Control Panel => System. Then look at the "Computer
Name" tab.  If you see something in the field for "Full computer name", then
you are done.  If there is nothing there, you have to click "Change" to assign
a name to this field.
   Under older versions of Windows, you may have to define a ComputerName
environment variable with a command like this before you start each slave:
set ComputerName=Slave01

(Step 0.e) Copy the included executable files to some directory that is in the
path for every machine, including both the master computer and all the slaves.
These are the necessary executable files:
  ExistDir.exe
  dirc.exe
  MasterStart.Exe
  Sleep.exe
  cdd1.exe
  cdd.bat
You may simply leave these files in the MasterPath directory, and add that
directory to the path of each machine.

------------------------------------------------------------------------------------

Steps 1-6: Detailed step-by-step instructions for starting a new set of simulations
(after the initial setup is done).

1. Set the paths in SetPaths.Bat.
   As shown in this batch file, two environment variables need to be set to
specify paths for the slave computers. "MasterPath" specifies the directory
that can be accessed by both the slave computer(s) and the master computer.
Both computers need permission to write files to this directory.  "SlavePath"
specifies the directory in which the slave program Slave.bat is started
on each slave computer.
   You can also specify the program search path in this file if you have not
already set it via the control panel.

2. Create a TaskList.txt file.
   This file is just a plain text (ASCII) file that you can create with Notepad
or some other plain text editor.  Tasks can be specified using 3 different
methods, and a single TaskList.txt file can use all 3 different methods
interspersed in any fashion.  For an example, look at the file TaskListEx1.Txt.

    Step 2, Method 1: One command per task.
    With this method, a task is specified on a single line of the TaskList.Txt
    file.  The line is exactly what you would typed at the command prompt to
    carry out that task.  For example, a task might be specified by including
    the following line in the TaskList.txt file:
        myprogram myparm1 myparm2
    This line would run the indicated program.
    Note that if this program is itself a batch file, you have to specify it
    using "call", like this:
        call mybat myparm1 myparm2

    Step 2, Method 2: N commands per task.
    With this method, a task is specified on 2 or more lines of the
    TaskList.Txt file.  The first line used to specify the task is the number
    of lines.  Then, the remaining lines are exactly what you would type at
    the command prompt to carry out that task.  For example, a task might be
    specified with the following lines in the TaskList.txt file:
        3
        myprogram1 myparm1
        myprogram2 myparm1 myparm2
        myprogram3
    These lines would run the three indicated programs.
    Note that if these programs are themselves batch files, you have to
    specify them using "call", like this:
        3
        call mybat1 myparm1
        call mybat2 myparm1 myparm2
        call mybat3

    Step 2, Method 3: Batch and template files.
    With this method, each task is specified on a single line of the
    TaskList.Txt file (like method 1).  In the simplest case, each line simply
    contains the name of the batch file that carries out the desired task.  So,
    for example, the file TaskList.txt might simply contain these five lines:
        @go1
        @go2
        @go3
        @go4
        @go5
    Note the use of the @: It signals a batch or template file.
    Each line corresponds to one task, and each task is defined by its
    batch file go1.bat, go2.bat, ...
    Crucially, each batch file must end with an "Exit" command.  If the
    different tasks are similar to one another, the "template" capabilities
    described below may be helpful.

3. Create batch and/or template files to carry out the individual tasks.
This step depends entirely on what individual tasks you want the computer(s)
to carry out.  The main requirements are that each batch file:
 (a) be self-sufficient--that is, able to run without operator intervention.
 (b) run correctly when started in the directory defined by SlavePath.
 (c) end with an Exit command.
As noted above, you may be able to use the "batch template" capabilities
described below if you want to run many similar batch files (e.g., just
changing a few program names or parameters).  For an example, look at the
file Template.Bat.

4. Reset or delete TaskList.Dat
   Each time you start up a new set of tasks (i.e., with a new TaskList.txt
file), you should simply delete the file TaskList.Dat.  The function of this
file is described later for advanced users, but for now this is all you need to
know.  Alternatively, if you start a new set of tasks by adding them to the
end of a preexisting TaskList.Txt file, then you should not do anything to
TaskList.Dat.

5. Start Master.bat on any one machine.
   Open a cmd window and change into the directory specified by MasterPath.
Type "Master" at the command prompt to run the batch file Master.Bat.
That machine is now the master.  Do NOT start more than one master!

6. Start Slave.bat on each slave machine.
   On each slave machine, open a cmd window and change into the directory
specified by SlavePath.  Type "Slave" at the command prompt to run the batch
file Slave.bat (which automatically starts other processes as needed).

------------------------------------------------------------------------------------

Stopping the simulations:

   There is no special provision for stopping the simulations.  Just use
control-break or close the windows.  Shutdown also works, as does turning
off the power. :)

------------------------------------------------------------------------------------

Batch Template Capabilities

Sometimes the different tasks in a list are all variants of the same basic
form.  I have built in some capability to ease preparation of batch files for
these cases.  This capability works almost exactly like using command-line
parameter with batch files.

As a trivial example, suppose I wanted to run the following 3 batch files:

   REM Batch file 1      REM Batch file 2      REM Batch file 3
   Myprog Run1           Myprog Run2           Myprog Run3     
   Exit                  Exit                  Exit            

Rather than making up 3 separate batch files, I can simply produce a single
batch file template (call it MyTemplate.bat) that looks like this:
   REM Batch file %A
   Myprog Run%A
   Exit
Then, I set up TaskList.txt to contain the 3 lines:
@MyTemplate 1
@MyTemplate 2
@MyTemplate 3

The 1, 2, and 3 at the ends of these lines are parameters to be substituted
for %A in the template file.  That is, when it is time to generate an actual
batch file for a slave to execute, all instances of %A in the template are
replaced by the first parameter following the batch file name in the TaskList
file.  Naturally, you can use multiple parameters on each line--up to 26, in
fact, corresponding to %A, %B, %C, etc. (These are case sensitive).

An important restriction is that the parameters cannot contain spaces.  So,
for example, the following line in TaskList.Txt:
MyTemplate "Documents and Settings"
will not work as you would like.  This line will be parsed as having three
arguments: (1) "Documents (2) and (3) Settings"

To work around this restriction, you can use some other character and specify
that it should be converted to spaces during the parameter substitution process.
For example, you could include in TaskList.Txt the line:
MyTemplate Documents#and#Settings -s#
The special parameter -s# at the end of the line indicates that all occurrences
of the character # within any parameter should be replaced by spaces.  Or,
if you need to use the # for real, you can use any other character instead.
For example, any of these would work the same way:
@MyTemplate Documents_and_Settings -s_
@MyTemplate Documents!and!Settings -s!
@MyTemplate Documents%and%Settings -s%

------------------------------------------------------------------------------------

Additional remarks and hints:

- IMPORTANT TO REMEMBER: A batch or batch template file must EXIT when it
finishes.

- You can adjust the Sleep times (in seconds) in the *.bat files to make the
master and/or slaves check more often for new tasks.  This would be a good idea
if the tasks are short, so that you won't waste too much time waiting for
tasks to start.

- There is no guarantee that the tasks will finish in any particular order, so
they should all be independent of one another (i.e., the output of one should
NOT be needed as the input to another).

- It is not a good idea to remove the apparently superfluous file Dummy.Don
from the MasterPath, although it is only needed for cosmetic reasons.

- The TaskList.Dat file is really just a plain text (ASCII) file with an
integer in it.  This integer is a count of the number of already-completed
tasks from the file TaskList.Txt.  Each time the master computer assigns a
new task, it simply increments this count.  So, you can add tasks to the
tasklist while master is running.  You can also reset the count to some
larger or smaller number by hand if you want to skip or rerun some of the
tasks in the list.  This has to be done carefully, of course.

- You can add Slave.bat to the startup directory in each slave computer
so that each slave automatically becomes "ready to go" when it is booted up
(or when someone logs on, depending on your setup).

- The *.wrk files in the master directory are the batch files that are "in
progress" on the slaves.  They are retained so that you can easily check from
one machine which tasks are underway at any given moment.

- The file MasterRecord.txt has a record of the tasks started by the Master,
just as are shown on the Master screen.  It should be cleared occasionally.
Don't change the name of this file--it is hardwired into the MasterStart
program.

- If you want to be emailed each time a task finishes, add an appropriate blat
command to the end of the batch file for the task, just before the exit.

- It is sometimes convenient to visually mark off groups of lines in the
TaskList.Txt file, so that you can more easily see what you've got.
As a marker, you can use any line with five or more identical nonalphanumeric
characters OTHER THAN @.  Master will simply skip over this line when assigning tasks.

- When no simulations are in progress, it is a good idea to delete the file
  TaskList.Txt. That way, the Master will complain less about the lack of tasks
  to assign, and the Slaves will not keep reminding the master that they are done.

- The easiest way to pause a set of simulations is to rename Tasklist.* to
  HoldTask.*. The master will then assign no new tasks until these files are
  renamed back to tasklist.*.  Of course, in-progress slaves will continue
  their current tasks until these are done.

------------------------------------------------------------------------------------

History:

1.0
1.1 Added:
    blat usage for emailing.
    \LOW priority so tasks would not slow down machines.
    Marker lines (e.g., ######) in TaskList.Txt.
    Simulations not in progress if no TaskList.Txt & .Dat
