Alex - a Global Filesystem
Vincent Cate


School of Computer Science
Carnegie Mellon University
Pittsburgh, Pennsylvania 15213-3890


Abstract


The Alex filesystem provides users and applications transparent read
access to files in anonymous FTP sites on the Internet. Today there are
thousands of anonymous FTP sites with a total of a few millions of files
and roughly a terabyte of data. The standard approach to accessing these
files involves logging in to the remote machine. This means that an
application can not access remote files like local files. This also means
that users do not have any of their aliases or local tools available.
Users who want to use an application on a remote file first have to
manually make a local copy of the file. There is no mechanism for
automatically updating this local copy when the remote file changes. The
users must keep track of where they get their files from and check to see
if there are updates, and then fetch these. In this approach many
different users at the same site may have made copies of the same remote
file each using up disk space for the same data.


Alex addresses the problems with the existing approach while remaining
within the existing FTP protocol so that the large collection of currently
available files can be used. To get reasonable performance long term file
caching must be used. Thus consistency must be addressed.  Traditional
solutions to the cache consistency problem do not work in the Internet FTP
domain: callbacks are not an option as the FTP protocol has no provisions
for this and polling over the Internet is slow. Therefore, Alex relaxes
file cache consistency semantics, on a per file basis, and uses special
caching algorithms that take into account the properties of the files and
of the network to allow a simple stateless filesystem to scale to the size
of the Internet.



1. Introduction


Today it is possible to access a tremendous amount of data in FTP archives
on the Internet; however, the current method of accessing this data is
primitive. Users must run the FTP program and then login to the remote
machine. Once they have done this, they can not use any of their local
tools or aliases. Applications on a user's machine can not access files on
remote machines, the user must manually copy them first. If the remote
file changes the user has to find out somehow and make another copy. Since
each user is making copies, there may be many copies of a file at a given
site. Since a user can not easily tell if their own copy of a file is out
of date they may use a very old version of a file. Thus the existing
mechanism is cumbersome, slow, inefficient, and provides only limited
access to remote data.

To address this problem, and explore some ideas in wide area filesystems,
I am designing and implementing a filesystem called "Alex". The goal of
this system is to let users access files in FTP sites all over the world
just as they access files on their local system. Ideally, both the way it
is used and its performance should match that of a local filesystem.
Caching is the key to performance but leads to a consistency problem. This
paper addresses the consistency problem and my solution to it.

The name Alex was chosen because of the similarity between the way Alex
gathers information from around the world into one logical place (/alex)
and the way the ancient library at Alexandria collected information from
around the world into one physical place.

In this paper, I describe the Alex architecture, how users interact with
it, and what they need to understand in order to use it. I then describe
the current NFS server implementation. Next I describe some of the useful
and novel applications that can be built on top of this filesystem. I
conclude with the current status of Alex and my research agenda.


2. User View of Alex

2.1. Standard Filesystem Interface


One of my goals was to make Alex easy to use and think about. The hope is
to give users a clean and simple view of Alex. To a user or application,
Alex is just a normal filesystem. Any command that works on local files
will work on Alex files. Since Alex is a real filesystem, nothing needs to
be recompiled and no libraries are changed. Thus, users can apply all of
their existing skills and tools for using files.


2.2. Naming

The user sees a filesystem with a hierarchical name space. At the top
level (/alex) there are top-level Internet domains like "edu", "com",
"uk", and "jp". Each component of the hostname becomes a directory name as
shown in Figure 1. Then the remote path is added at the end. If the user
does a "ls /alex/edu/berkeley" he sees some machine names such as "ucbvax"
and "sprite" and some directories on berkeley.edu. From the "ls" it is not
clear what is where. The user may or may not be aware of host boundaries.


Figure 1. Example Alex Name


2.3. Control of Consistency


By default, the system guarantees to the user that the only updates that
might not yet show up are ones that have happened in the last 5% of the
reported age of the file. For example, if the user sees a file as being 20
hours old the system promises that the data given to the user will be at
most 1 hour stale. If a file was last mod- ified 20 days ago the file may
be up to 1 day stale. There is a maximum of 1 month and a minimum of 5
min- utes. The user can override these rules and ask for an update of any
directory at any time.


3. Alex Implementation


As shown in Figure 2, Alex is currently implemented as an NFS server. NFS
was chosen because it makes it easy to add Alex to a wide range of
machines. Most machines can simply use the mount command. Another
motivation for choosing NFS was that it is simple and stateless. Because
NFS is stateless it is fine if Alex dies and restarts (for example, when
installing a new version of the server). Since it takes only a couple of
seconds, none of the clients notice.


Figure 2 Alex is an NFS Server


3.1. Background


A few important characteristics of the Alex domain motivated my design
choices. Key to the Alex approach is using existing FTP sites without any
change to the FTP protocol. This determines a number of aspects of the
domain.


FTP gives us whole file transfer like AFS [Howard87]. Whole file transfer
is even more important in the case of a WAN than in the case of a LAN
where it has already proven its merit. The FTP protocol does not do call-
backs as does AFS (FTP is not a filesystem protocol, though I use it as
one). Thus, Alex is fully responsible for checking the consistency of its
cache with the FTP site. The FTP protocol returns directory information as
a unit (not one file at a time). This, like whole file transfer, reduces
the number of times you pay the network latency cost. This is a big
performance win in a WAN.


Also important are characteristics of the domain FTP operates in. This
domain is truly heterogeneous. While FTP hides some of this it also
exposes plenty. Both the syntax for specifying names and the directory
informa- tion returned varies among the different types of FTP sites.
Another problem is that a substantial fraction of the FTP sites are either
down or inaccessible at any one time. 


The type of data stored in FTP sites and the access patterns for this data
are very different from that of normal files. The data change much less
often than do normal files. This is a very important aspect of the domain.


3.2. Naming


A path name in Alex has three parts. The first is "/alex". The second part
is an Internet host name, with the most significant piece first (such as
"edu") and the "."s replaced by "/"s. The third part is the path on the
remote machine relative to the ftp starting directory on that machine. An
example path is "/alex/edu/berkeley/ pub/virus.patch".


There is an ambiguity in the name in that "pub" in the above path is a
directory on "berkeley.edu" but there is a "pub.berkeley.edu" that gets
the same name in Alex. To resolve a name Alex first matches with the
longest matching hostname it knows about (similar to Sprite's prefix
tables [Welch86]). Next, it sees if the rest of the path works. If it does
not Alex will create a possible hostname and call a nameserver. This
approach seems to work almost everywhere. In fact, pub.berkeley.edu is the
only site found so far for which this does not work (fortunately there is
nothing there to FTP).


One problem with this type of naming is that a user who has set up his
account to search the current directory for commands before other places
such as "/bin" gets very poor performance at times. For example, if such a
user wants to run "ls" while in /alex/edu/berkeley the shell will check to
see if there is a /alex/edu/berkeley/ls.  This causes the Alex server to
check for a host called "ls.berkeley.edu" (assuming there was no ls file
visible).  Doing this takes a call to a remote nameserver, which is
expensive. As long as users know this they can set up their search path to
avoid this problem.


3.3.  Caching


To get reasonable performance a large number of things are cached in
various ways. Some of these are listed here.


1 Machine names - Alex caches machine names so it can quickly resolve file
names (saving calls to the nameserver).


2 Open FTP connections - Sometimes fetching a second small file is so fast
it looks like it was in the file cache. 


3 Type of FTP site - Alex needs to know how the type of FTP site it is
talking to in order to specify names of files and directories in the
appropriate syntax.


4 Time zone information - In order to convert from the FTP local time on
file modifi- cations to the Greenwich time needed for NFS. To get the time
zone of a machine may take two calls to servers on that machine. 


5 Directory information and files - These are the most important thing
Alex caches.  In general when I refer to caching this is the type of
caching I mean.


6 Failures of certain types - If the nameserver says that it does not know
of some host we cache that fact. Also, if a host says "no anonymous ftp"
we cache that fact. If the FTP cannot connect we cache that for a shorter
period. It has turned out that caching failure information is very
important for good performance. Failures are detected with timeouts, which
can take awhile. Also, NFS retries can occur while waiting for a timeout
from some FTP site. If the failures are not cached, each of the NFS
retries would cause another long attempt to contact the FTP site. 


3.4. Directories


Alex organizes directories to use disks more efficiently than a normal
Unix filesystem. When a user does an "ls -F" or any operation which
inspects the meta-data for each file in a directory, the Unix filesystem
requires a disk seek for each file whose meta-data is not already cached
in RAM. Since Alex stores all the directory infor- mation together
sequentially on the disk, accesses to it are much faster. This is simple
to do since there are no hard links to deal with. We don't actually store
any details of disk allocation, so much of normal meta-data information is
not needed. Inlining all file information only about doubles the directory
size.


3.5. Handling Heterogeneity


There are a three ways in which heterogeneity affects Alex. These are
specifying files and directory names, parsing the directory information
returned, and deciding to transfer a file in binary or text mode.

It is possible to determine the name syntax an FTP server uses by its
response to the "pwd" command and other commands. Once this information is
known it is straightforward to specify files in the format that it wants.
We cache this information so that it only needs to be figured out the
first time or if the machine changes type, (e.g. switches from VMS to
Unix).

The FTP "dir" command returns directory information. On Unix machines this
is the result of an "ls -l". Simi- lar information is returned for other
operating systems. Alex uses this to create all of the information for the
NFS meta-data request. Alex can parse directory information from a variety
of machine types, some examples are shown in Figure 3. File sizes and
dates are used to determine the consistency of locally cached data. Thus,
Alex, as well as the user, requires this information. Some systems, such
as VMS FTP sites, give no indication of the size of a file. For these
sites, Alex can only give the size of a file after it has been fetched. In
the case of VM systems, Alex can only estimate the size of a file before
it is fetched (IBM provides only a maximum record size and number of
records). Despite these difficulties, Alex hides the heterogeneity so well
that a user can "cd" to a VMS machine and never know it.

DEC VMS:
00README.TXT;7 6 9-APR-1991 18:14 [ANONYMOUS] (RWED,RE,RE,RE)
PUB.DIR;1 8 18-OCT-1990 07:20 [ANONYMOUS] (RWED,RE,RE,RE)

Unix:
dr-xr-xr-x 3 system 320 Jul 16 14:25 0.7alpha
-rw-rw-rw- 1 system 853 Jul 16 13:41 directory

IBM VM/CMS
FUSION BIBLIO4 V 80 2219 45 11/25/91 7:00:07 FUSION
FUSION BIBLIO5 V 78 132 3 11/27/91 10:01:11 FUSION

Figure 3. Heterogeneous Directory Information


The descision to transfer data in binary or text format is made as
follows. On Unix machines we always use binary (they all seem to use
ASCII). On others, we use text unless the filename extension indicates
that the file is one of a few types, such as a .gif file, that need to be
transferred as binaries. This works most of the time.


4. .Building on Top of Alex


Since Alex has a single global namespace, each file has a unique global
name. This distinguishes Alex from a number of other systems such as
Prospero [Neuman89] and Jade [Rao91] that focus on user specific views.
This means that if one user says, "look around in /alex/org/eff/ftp"
another user can easily do so (possibly without ever knowing what the
machine name really was). While Alex nicely handles the naming and access-
ing issues, it does not address the indexing and organizing of the files.
Since there is so much information available, this is very important.
Fortunately, since Alex looks just like a filesystem it can easily be
combined with other applications to get indexing and organization along
with easy access. Systems such as Hyperbold [Weiner91], WAIS [Kahle91],
World Wide Web [Berners-Lee92], and archie [Emtage92] can be combined with
Alex to provide this functionality.


4.1. Power of Symbolic Links


Symbolic links become very powerful once directories and files all around
the world can be named. For exam- ple, we have a directory with symbolic
links to about 100 directories around the world containing computer
science technical reports. Symbolic links provide a user with the ability
to "customize his namespace" though not with the power of some the systems
focusing on this issues. A symbolic link to a remote file is as conve-
nient as a copy and has several advantages. First, it does not take up any
of the user's disk space. If the remote copy changes the user's copy would
be out of date but the symbolic link version can show the new data.
Finally, the symbolic link documents where the file comes from, making it
easy to go back later and look for related files. 


4.2. Combining Alex and Archie to get Archia


Archie [Emtage92] is a database that allows users to search for filenames
in FTP sites. For each match, archie returns a host, directory, and
filename. Archia combines these 3 pieces into an Alex path. So a user can
type archia jokes and get Alex pathnames with the word jokes in them. The
user can then easily access these files with Alex.


4.3. Combining Alex and WAIS


WAIS [Kahle91] stands for Wide Area Information Server. This package makes
it easy to set up a database server that people from around the world can
access with the client software. Since Alex lets you access files from all
around the world it becomes easy to use something like WAIS to index files
from all around the world.  I have created two databases to experiment
with this. One indexes README files, the other, computer science technical
papers.

I created the readmes database by indexing files with names including the
string "README". The initial set of pathnames was made using Archia. Since
then I have used the Unix find command and Alex to find more.  Currently I
have indexed about 25,000 README files from around the world. Users can
search on any words in any of these files. Since README files usually
describe files in the directory they are in, this can frequently help a
user find what they want.

For the cs-tech-reports database I have indexed a large number of computer
science technical papers from around the world. I started with the above
mentioned list of symbolic links to directories with technical papers.  I
used this list of directories and the Unix find command to get the
filenames for the papers. I then indexed these just like the README files.


5. Status


Alex is up and working. There are roughly 100 users. The server load is
usually small since the average user accesses Alex for less than a half
hour per week. The NFS protocol is fine for this level of load. Users have
vis- ited over 1,500 different FTP sites via Alex. Currently about 500 MB
of disk is being used to cache data. With current levels of activity files
usually stay in the cache more than a month past the time they are last
accessed.


Real performance results are not yet available (see research agenda in
next section) but things look very prom- ising. To give some idea of the
performance here are a few numbers. I did a "find /alex/edu -print" and
let it run for a day. It found about 10,000 files per hour (most of which
were probably not in the cache to start with) and had only finished a
small fraction of the edu hosts before I killed it. When Alex was reading
the 25,000 README files it averaged about 10 seconds per file (again most
were not in the cache to start with). The techreports did not take much
longer than this.


The readmes database is averaging more than 500 requests per day (searches
plus retrieves) from around the world. The cs-techreports database was
just recently released.


6. Research Agenda


There are many research issues related to Alex. Below are some of the main
ones I plan to explore.


6.1.  Caching Algorithms

In this domain, there are many factors to consider when deciding what to
keep in the cache and what to get rid of. As in other caching situations,
how recently and how frequently a file has been used, as well as its size,
are important factors. Additional new factors are past latency and
bandwidth experienced when accessing the source for file. The source's
availability is also a factor. Thus, caching algorithms are complex.

I need to determine what the important parameters are for caching
algorithms and how they relate to each other. In particular, how latency,
bandwidth, and availability should affect what is kept in the cache. It
will be very interesting to know what sort of hit ratios we can get for
the files and directories.

There are many other interesting things to measure, such as the number of
times a file is used once in the cache and how long files go between
usages.


6.2. Sharing of Cached Data Between Users


One of the benefits of this approach is that a cached copy of a file is
frequently available because some other user has accessed the file. It
will be interesting to see how often this happens. Preliminary indications
are that this will be very common for Alex users.

In [Ewing92] they found that just over half of the FTP traffic for users
at one site during a short 2 week period was for files that more than one
user accessed. Alex caches data for much longer than this and should get
even higher levels of inter-user sharing of cached data.


6.3. Characterization of Usage Pattern

FTP files are not typical user files and the access pattern to them is
very different. They are often archive files that don't ever change.
Frequently the ones that do change are replaced by a new file of a
slightly different name. For example pbmplus05oct91.tar.Z was replaced by
pbmplus10dec91.tar.Z. This is reminiscent of immutable files in the Cedar
filesystem [Gifford88]. 


6.4. Naming Ambiguity Problems

Among the systems visited, the only naming ambiguity noticed was
/alex/edu/berkeley/pub. It turns out that there is nothing to ftp on
pub.berkeley.edu so even this one instance has not been a problem.None of
my users have experienced the problem. However, I plan to do a careful
investigation looking for instances of this prob- lem.


6.5. Quantifying Consistency

Alex can tell when it has given out a stale file later when it updates
that directory. From this I can estimate the percentage of times users get
stale data. Alex can also tell how far out of date the stale files were. 

There are also cases where Alex can tell that it has stale data and clean
things up. For example, if a user tries to fetch a file that no longer
exists Alex quickly updates the directory. Also if it fetches a file and
the size is not what it was expecting it updates the directory information
for that directory. Measuring how often these events happen also gives us
some measure of consistency.

Any user can request the update of any directory. Users may desire this
because they have reason to believe that the directory information is out
of date (for example they saw a post saying a file was here but it does
not show up). It may also be that the user has had problems in the past
and does not trust Alex to keep the files con- sistent. So measuring how
often users do this is one way to quantify how well users feel Alex's
consistency algorithm is working.


6.6. Latency for Large Files

Right now when you do a "more bigfile" where bigfile is not yet in the
cache you do not see anything until the whole FTP has finished (since FTP
is whole file transfer). This means that the user sees long latency at
times.  However, I will probably modify Alex so that it starts using data
as soon as it comes in. Once this is done the user will see reasonable
latency even for large files not in the cache.


7. Conclusions

Alex is clearly a leap forward in accessing remote data. It makes it very
easy for users and applications to access files from all around the world.
Along with ease of use it also improves performance. While using FTP gives
one the feel of looking at a remote place through a narrow tube, Alex
gives one the feeling of actually visiting the remote place. World travel
has never been easier.

This technology can be combined in a very modular way with many different
systems. I have explored this using WAIS. Thus Alex is also an enabling
technology. As a platform, it can support some powerful systems.


This paper and other documents on Alex can be found in
/alex/edu/cmu/cs/sp/alex/doc.



8. Bibliography

Bob Alberti, Farhad Anklesaria, Paul Lindner, Mark McCahill, Daniel
Torrey, "Notes on the internet Gopher protocol", Microcomputer and
Workstation Networks Research Center, Spring 1991; Revised December 1991,
/alex/edu/umn/micro/boombox/pub/gopher/gopher_protocol.

Tim Berners-Lee, Robert Cailliau, Jean-Francois Groff, Bernd Pollermann,
"World-Wide Web: The Information Universe", CERN, 1211 Geneva 23,
Switzerland, /alex/ch/cern/nxoc01/pub/WWW/doc/Article_9202.ps

A. Emtage, P. Deutsch. archie - An Electronic Directory Service for the
Internet. Conference proceedings Usenix pp. 93-110, San Francisco, CA,
January 1992. /alex/ca/mcgill/cs/quiche/archie/doc/archie-usenix92-
paper.ps.

David J. Ewing, Richard S. Hall, Michael F. Schwartz, "A Measurement Study
of Internet File Transfer Traf- fic", January 1992, Department of Computer
Science, University of Colorado at Boulder, CU-CS-571-92, /alex/edu/
colorado/cs/bruno/pub/cs/techreports/schwartz/RD.Papers/PostScript/
FTP.Meas.ps.Z.

Herman Chung-Hwa Rao, "The Jade File System (Ph. D. Dissertation)",
Department of Computer Science, University of Arizona, TR 91-18. See also
"Accessing Files in an Internet: The Jade File System", /alex/edu/
arizona/cs/llp/jade.ps

David K. Gifford, Roger M. Needham, Michael D. Schroeder, "The Cedar File
System", Communications of the ACM, Vol 31, No 3, March 1988, p 288-298.

Brewster Kahle,"Wide Area Information Servers", Thinking Machines, April,
1991, /alex/com/think/quake/ wais/wais-discussion/overview.text.

J. Howard, M. Kazar, S. Menees,D.Nichols, M. Satyanarayana, R.
Sidebothamn, M. West, "Scale and Perfor- mance in a Distributed file
System", Operating Systems Review, Vol 21, No 5, November 1987.

B. Clifford Neuman, "The Virtual System Model for Large Distributed
Operating Systems", Department of Computer Science, University of
Washington, TR-89-01-07, /alex/edu/washington/cs/pub/pfs/doc/UW-CS-89-
01-07.PS.Z

Bob Weiner, "Hyperbole Manual - Hypertext for Everyday Work", Brown
University, December 1991, /alex/edu/brown/cs/wilma/pub/hyperbole/hypb.ps.Z

B. B. Welch and J. K. Ousterhout, "Prefix Tables: A Simple Mechanism for
Locating Files in a Distributed Filesystem", Proc. of the 6th
International Conference on Distributed Computing Systems, May 1986, pages
184-189.


