CedarBackup2.filesystem.FilesystemList

Package CedarBackup2 :: Module filesystem :: Class FilesystemList |
[hide private]
[frames] | no frames]

# Class FilesystemList

source code

    object --+
             |
          list --+
                 |
                FilesystemList

Known Subclasses:
  - BackupFileList
  - , PurgeItemList

Represents a list of filesystem items.

This is a generic class that represents a list of filesystem items. Callers can add individual files or directories to the list, or can recursively add the contents of a directory. The class also allows for up-front exclusions in several forms (all files, all directories, all items matching a pattern, all items whose basename matches a pattern, or all directories containing a specific "ignore file"). Symbolic links are typically backed up non-recursively, i.e. the link to a directory is backed up, but not the contents of that link (we don't want to deal with recursive loops, etc.).

The custom methods such as addFile will only add items if they exist on the filesystem and do not match any exclusions that are already in place. However, since a FilesystemList is a subclass of Python's standard list class, callers can also add items to the list in the usual way, using methods like `append()` or `insert()`. No validations apply to items added to the list in this way; however, many list-manipulation methods deal "gracefully" with items that don't exist in the filesystem, often by ignoring them.

Once a list has been created, callers can remove individual items from the list using standard methods like `pop()` or `remove()` or they can use custom methods to remove specific types of entries or entries which match a particular pattern.

Notes:

- Regular expression patterns that apply to paths are assumed to be bounded at front and back by the beginning and end of the string, i.e. they are treated as if they begin with `^` and end with `$`. This is true whether we are matching a complete path or a basename.
- Some platforms, like Windows, do not support soft links. On those platforms, the ignore-soft-links flag can be set, but it won't do any good because the operating system never reports a file as a soft link.

Instance Methods | [hide private]
new empty list |
__init__(self) Initializes a list with no configured exclusions. | source code
addFile(self, path) Adds a file to the list. | source code
addDir(self, path) Adds a directory to the list. | source code
addDirContents(self, path, recursive=True, addSelf=True, linkDepth=0, dereference=False) Adds the contents of a directory to the list. | source code
removeFiles(self, pattern=None) Removes file entries from the list. | source code
removeDirs(self, pattern=None) Removes directory entries from the list. | source code
removeLinks(self, pattern=None) Removes soft link entries from the list. | source code
removeMatch(self, pattern) Removes from the list all entries matching a pattern. | source code
removeInvalid(self) Removes from the list all entries that do not exist on disk. | source code
normalize(self) Normalizes the list, ensuring that each entry is unique. | source code
_setExcludeFiles(self, value) Property target used to set the exclude files flag. | source code
_getExcludeFiles(self) Property target used to get the exclude files flag. | source code
_setExcludeDirs(self, value) Property target used to set the exclude directories flag. | source code
_getExcludeDirs(self) Property target used to get the exclude directories flag. | source code
_setExcludeLinks(self, value) Property target used to set the exclude soft links flag. | source code
_getExcludeLinks(self) Property target used to get the exclude soft links flag. | source code
_setExcludePaths(self, value) Property target used to set the exclude paths list. | source code
_getExcludePaths(self) Property target used to get the absolute exclude paths list. | source code
_setExcludePatterns(self, value) Property target used to set the exclude patterns list. | source code
_getExcludePatterns(self) Property target used to get the exclude patterns list. | source code
_setExcludeBasenamePatterns(self, value) Property target used to set the exclude basename patterns list. | source code
_getExcludeBasenamePatterns(self) Property target used to get the exclude basename patterns list. | source code
_setIgnoreFile(self, value) Property target used to set the ignore file. | source code
_getIgnoreFile(self) Property target used to get the ignore file. | source code
_addDirContentsInternal(self, path, includePath=True, recursive=True, linkDepth=0, dereference=False) Internal implementation of `addDirContents`. | source code
verify(self) Verifies that all entries in the list exist on disk. | source code
Inherited from `list`: `__add__`, `__contains__`, `__delitem__`, `__delslice__`, `__eq__`, `__ge__`, `__getattribute__`, `__getitem__`, `__getslice__`, `__gt__`, `__iadd__`, `__imul__`, `__iter__`, `__le__`, `__len__`, `__lt__`, `__mul__`, `__ne__`, `__new__`, `__repr__`, `__reversed__`, `__rmul__`, `__setitem__`, `__setslice__`, `__sizeof__`, `append`, `count`, `extend`, `index`, `insert`, `pop`, `remove`, `reverse`, `sort` Inherited from `object`: `__delattr__`, `__format__`, `__reduce__`, `__reduce_ex__`, `__setattr__`, `__str__`, `__subclasshook__`

Class Variables | [hide private]
Inherited from `list`: `__hash__`

Properties | [hide private]
| excludeFiles Boolean indicating whether files should be excluded.
| excludeDirs Boolean indicating whether directories should be excluded.
| excludeLinks Boolean indicating whether soft links should be excluded.
| excludePaths List of absolute paths to be excluded.
| excludePatterns List of regular expression patterns (matching complete path) to be excluded.
| excludeBasenamePatterns List of regular expression patterns (matching basename) to be excluded.
| ignoreFile Name of file which will cause directory contents to be ignored.
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self) (Constructor) | source code
Initializes a list with no configured exclusions. Returns: new empty list Overrides: object.__init__

### addFile(self, path) | source code
Adds a file to the list. The path must exist and must be a file or a link to an existing file. It will be added to the list subject to any exclusions that are in place. Parameters: `path` (String representing a path on disk) - File path to be added to the list Returns: Number of items added to the list. Raises: `ValueError` - If path is not a file or does not exist. `ValueError` - If the path could not be encoded properly.

### addDir(self, path) | source code
Adds a directory to the list. The path must exist and must be a directory or a link to an existing directory. It will be added to the list subject to any exclusions that are in place. The ignoreFile does not apply to this method, only to addDirContents. Parameters: `path` (String representing a path on disk) - Directory path to be added to the list Returns: Number of items added to the list. Raises: `ValueError` - If path is not a directory or does not exist. `ValueError` - If the path could not be encoded properly.

### addDirContents(self, path, recursive=True, addSelf=True, linkDepth=0, dereference=False) | source code
Adds the contents of a directory to the list. The path must exist and must be a directory or a link to a directory. The contents of the directory (as well as the directory path itself) will be recursively added to the list, subject to any exclusions that are in place. If you only want the directory and its immediate contents to be added, then pass in `recursive=False`. Parameters: `path` (String representing a path on disk) - Directory path whose contents should be added to the list `recursive` (Boolean value) - Indicates whether directory contents should be added recursively. `addSelf` (Boolean value) - Indicates whether the directory itself should be added to the list. `linkDepth` (Integer value, where zero means not to follow any soft links) - Maximum depth of the tree at which soft links should be followed `dereference` (Boolean value) - Indicates whether soft links, if followed, should be dereferenced Returns: Number of items recursively added to the list Raises: `ValueError` - If path is not a directory or does not exist. `ValueError` - If the path could not be encoded properly. Notes: If a directory's absolute path matches an exclude pattern or path, or if the directory contains the configured ignore file, then the directory and all of its contents will be recursively excluded from the list. If the passed-in directory happens to be a soft link, it will be recursed. However, the linkDepth parameter controls whether any soft links within the directory will be recursed. The link depth is maximum depth of the tree at which soft links should be followed. So, a depth of 0 does not follow any soft links, a depth of 1 follows only links within the passed-in directory, a depth of 2 follows the links at the next level down, etc. Any invalid soft links (i.e. soft links that point to non-existent items) will be silently ignored. The excludeDirs flag only controls whether any given directory path itself is added to the list once it has been discovered. It does not modify any behavior related to directory recursion. If you call this method on a link to a directory that link will never be dereferenced (it may, however, be followed).

### removeFiles(self, pattern=None) | source code
Removes file entries from the list. If `pattern` is not passed in or is `None`, then all file entries will be removed from the list. Otherwise, only those file entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries). This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all files, then you will be better off setting excludeFiles to `True` before adding items to the list. Parameters: `pattern` - Regular expression pattern representing entries to remove Returns: Number of entries removed Raises: `ValueError` - If the passed-in pattern is not a valid regular expression.

### removeDirs(self, pattern=None) | source code
Removes directory entries from the list. If `pattern` is not passed in or is `None`, then all directory entries will be removed from the list. Otherwise, only those directory entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries). This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all directories, then you will be better off setting excludeDirs to `True` before adding items to the list (note that this will not prevent you from recursively adding the contents of directories). Parameters: `pattern` - Regular expression pattern representing entries to remove Returns: Number of entries removed Raises: `ValueError` - If the passed-in pattern is not a valid regular expression.

### removeLinks(self, pattern=None) | source code
Removes soft link entries from the list. If `pattern` is not passed in or is `None`, then all soft link entries will be removed from the list. Otherwise, only those soft link entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries). This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all soft links, then you will be better off setting excludeLinks to `True` before adding items to the list. Parameters: `pattern` - Regular expression pattern representing entries to remove Returns: Number of entries removed Raises: `ValueError` - If the passed-in pattern is not a valid regular expression.

### removeMatch(self, pattern) | source code
Removes from the list all entries matching a pattern. This method removes from the list all entries which match the passed in `pattern`. Since there is no need to check the type of each entry, it is faster to call this method than to call the removeFiles, removeDirs or removeLinks methods individually. If you know which patterns you will want to remove ahead of time, you may be better off setting excludePatterns or excludeBasenamePatterns before adding items to the list. Parameters: `pattern` - Regular expression pattern representing entries to remove Returns: Number of entries removed. Raises: `ValueError` - If the passed-in pattern is not a valid regular expression. Note: Unlike when using the exclude lists, the pattern here is not bounded at the front and the back of the string. You can use any pattern you want.

### removeInvalid(self) | source code
Removes from the list all entries that do not exist on disk. This method removes from the list all entries which do not currently exist on disk in some form. No attention is paid to whether the entries are files or directories. Returns: Number of entries removed.

### _setExcludeFiles(self, value) | source code
Property target used to set the exclude files flag. No validations, but we normalize the value to `True` or `False`.

### _setExcludeDirs(self, value) | source code
Property target used to set the exclude directories flag. No validations, but we normalize the value to `True` or `False`.

### _setExcludeLinks(self, value) | source code
Property target used to set the exclude soft links flag. No validations, but we normalize the value to `True` or `False`.

### _setExcludePaths(self, value) | source code
Property target used to set the exclude paths list. A `None` value is converted to an empty list. Elements do not have to exist on disk at the time of assignment. Raises: `ValueError` - If any list element is not an absolute path.

### _setExcludePatterns(self, value) | source code
Property target used to set the exclude patterns list. A `None` value is converted to an empty list.

### _setExcludeBasenamePatterns(self, value) | source code
Property target used to set the exclude basename patterns list. A `None` value is converted to an empty list.

### _setIgnoreFile(self, value) | source code
Property target used to set the ignore file. The value must be a non-empty string if it is not `None`. Raises: `ValueError` - If the value is an empty string.

### _addDirContentsInternal(self, path, includePath=True, recursive=True, linkDepth=0, dereference=False) | source code
Internal implementation of `addDirContents`. This internal implementation exists due to some refactoring. Basically, some subclasses have a need to add the contents of a directory, but not the directory itself. This is different than the standard `FilesystemList` behavior and actually ends up making a special case out of the first call in the recursive chain. Since I don't want to expose the modified interface, `addDirContents` ends up being wholly implemented in terms of this method. The linkDepth parameter controls whether soft links are followed when we are adding the contents recursively. Any recursive calls reduce the value by one. If the value zero or less, then soft links will just be added as directories, but will not be followed. This means that links are followed to a constant depth starting from the top-most directory. There is one difference between soft links and directories: soft links that are added recursively are not placed into the list explicitly. This is because if we do add the links recursively, the resulting tar file gets a little confused (it has a link and a directory with the same name). Parameters: `path` - Directory path whose contents should be added to the list. `includePath` - Indicates whether to include the path as well as contents. `recursive` - Indicates whether directory contents should be added recursively. `linkDepth` - Depth of soft links that should be followed `dereference` - Indicates whether soft links, if followed, should be dereferenced Returns: Number of items recursively added to the list Raises: `ValueError` - If path is not a directory or does not exist. Note: If you call this method on a link to a directory that link will never be dereferenced (it may, however, be followed).

### verify(self) | source code
Verifies that all entries in the list exist on disk. Returns: `True` if all entries exist, `False` otherwise.

Property Details | [hide private]

### excludeFiles Boolean indicating whether files should be excluded. Get Method: _getExcludeFiles(self) - Property target used to get the exclude files flag. Set Method: _setExcludeFiles(self, value) - Property target used to set the exclude files flag.

### excludeDirs Boolean indicating whether directories should be excluded. Get Method: _getExcludeDirs(self) - Property target used to get the exclude directories flag. Set Method: _setExcludeDirs(self, value) - Property target used to set the exclude directories flag.

### excludeLinks Boolean indicating whether soft links should be excluded. Get Method: _getExcludeLinks(self) - Property target used to get the exclude soft links flag. Set Method: _setExcludeLinks(self, value) - Property target used to set the exclude soft links flag.

### excludePaths List of absolute paths to be excluded. Get Method: _getExcludePaths(self) - Property target used to get the absolute exclude paths list. Set Method: _setExcludePaths(self, value) - Property target used to set the exclude paths list.

### excludePatterns List of regular expression patterns (matching complete path) to be excluded. Get Method: _getExcludePatterns(self) - Property target used to get the exclude patterns list. Set Method: _setExcludePatterns(self, value) - Property target used to set the exclude patterns list.

### excludeBasenamePatterns List of regular expression patterns (matching basename) to be excluded. Get Method: _getExcludeBasenamePatterns(self) - Property target used to get the exclude basename patterns list. Set Method: _setExcludeBasenamePatterns(self, value) - Property target used to set the exclude basename patterns list.

### ignoreFile Name of file which will cause directory contents to be ignored. Get Method: _getIgnoreFile(self) - Property target used to get the ignore file. Set Method: _setIgnoreFile(self, value) - Property target used to set the ignore file.