{
  "name": "Duperemove",
  "tagline": "Tools for deduping file systems",
  "body": "# Duperemove\r\n\r\nDuperemove is a simple tool for finding duplicated extents and\r\nsubmitting them for deduplication. When given a list of files it will\r\nhash their contents on a block by block basis and compare those hashes\r\nto each other, finding and categorizing extents that match each\r\nother. When given the -d option, duperemove will submit those\r\nextents for deduplication using the Linux kernel extent-same ioctl.\r\n\r\nDuperemove can store the hashes it computes in a 'hashfile'. If\r\ngiven an existing hashfile, duperemove will only compute hashes\r\nfor those files which have changed since the last run.  Thus you can run\r\nduperemove repeatedly on your data as it changes, without having to\r\nre-checksum unchanged data.\r\n\r\nDuperemove can also take input from the [fdupes](https://github.com/adrianlopezroche/fdupes) program.\r\n\r\nSee [the duperemove man page](http://markfasheh.github.io/duperemove/duperemove.html) for further details about running duperemove.\r\n\r\n\r\n# Requirements\r\n\r\nThe latest stable code can be found in [v0.10-branch](https://github.com/markfasheh/duperemove/tree/v0.10-branch).\r\n\r\nKernel: Duperemove needs a kernel version equal to or greater than 3.13\r\n\r\nLibraries: Duperemove uses glib2 and sqlite3.\r\n\r\n\r\n# FAQ\r\n\r\nPlease see the FAQ file [provided in the duperemove\r\nsource](https://github.com/markfasheh/duperemove/blob/master/FAQ.md)\r\n\r\nFor bug reports and feature requests please use [the github issue tracker](https://github.com/markfasheh/duperemove/issues)\r\n\r\n\r\n# Examples\r\n\r\nPlease see the examples section of [the duperemove man\r\npage](http://markfasheh.github.io/duperemove/duperemove.html#7)\r\nfor a complete set of usage examples, including hashfile usage.\r\n\r\n## A simple example, with program output\r\n\r\nDuperemove takes a list of files and directories to scan for\r\ndedupe. If a directory is specified, all regular files within it will\r\nbe scanned. Duperemove can also be told to recursively scan\r\ndirectories with the '-r' switch. If '-h' is provided, duperemove will\r\nprint numbers in powers of 1024 (e.g., \"128K\").\r\n\r\nAssume this abitrary layout for the following examples.\r\n\r\n    .\r\n    ├── dir1\r\n    │   ├── file3\r\n    │   ├── file4\r\n    │   └── subdir1\r\n    │       └── file5\r\n    ├── file1\r\n    └── file2\r\n\r\nThis will dedupe files 'file1' and 'file2':\r\n\r\n    duperemove -dh file1 file2\r\n\r\nThis does the same but adds any files in dir1 (file3 and file4):\r\n\r\n    duperemove -dh file1 file2 dir1\r\n\r\nThis will dedupe exactly the same as above but will recursively walk\r\ndir1, thus adding file5.\r\n\r\n    duperemove -dhr file1 file2 dir1/\r\n\r\n\r\nAn actual run, output will differ according to duperemove version.\r\n\r\n    Using 128K blocks\r\n    Using hash: murmur3\r\n    Using 4 threads for file hashing phase\r\n    csum: /btrfs/file1 \t[1/5] (20.00%)\r\n    csum: /btrfs/file2 \t[2/5] (40.00%)\r\n    csum: /btrfs/dir1/subdir1/file5 \t[3/5] (60.00%)\r\n    csum: /btrfs/dir1/file3 \t[4/5] (80.00%)\r\n    csum: /btrfs/dir1/file4 \t[5/5] (100.00%)\r\n    Total files:  5\r\n    Total hashes: 80\r\n    Loading only duplicated hashes from hashfile.\r\n    Hashing completed. Calculating duplicate extents - this may take some time.\r\n    Simple read and compare of file data found 3 instances of extents that might benefit from deduplication.\r\n    Showing 2 identical extents of length 512.0K with id 0971ffa6\r\n    Start\t\tFilename\r\n    512.0K\t\"/btrfs/file1\"\r\n    1.5M\t\"/btrfs/dir1/file4\"\r\n    Showing 2 identical extents of length 1.0M with id b34ffe8f\r\n    Start\t\tFilename\r\n    0.0\t\"/btrfs/dir1/file4\"\r\n    0.0\t\"/btrfs/dir1/file3\"\r\n    Showing 3 identical extents of length 1.5M with id f913dceb\r\n    Start\t\tFilename\r\n    0.0\t\"/btrfs/file2\"\r\n    0.0\t\"/btrfs/dir1/file3\"\r\n    0.0\t\"/btrfs/dir1/subdir1/file5\"\r\n    Using 4 threads for dedupe phase\r\n    [0x147f4a0] Try to dedupe extents with id 0971ffa6\r\n    [0x147f770] Try to dedupe extents with id b34ffe8f\r\n    [0x147f680] Try to dedupe extents with id f913dceb\r\n    [0x147f4a0] Dedupe 1 extents (id: 0971ffa6) with target: (512.0K, 512.0K), \"/btrfs/file1\"\r\n    [0x147f770] Dedupe 1 extents (id: b34ffe8f) with target: (0.0, 1.0M), \"/btrfs/dir1/file4\"\r\n    [0x147f680] Dedupe 2 extents (id: f913dceb) with target: (0.0, 1.5M), \"/btrfs/file2\"\r\n    Kernel processed data (excludes target files): 4.5M\r\n    Comparison of extent info shows a net change in shared extents of: 5.5M\r\n\r\n\r\n# Links of interest\r\n\r\n[The duperemove wiki](https://github.com/markfasheh/duperemove/wiki)\r\nhas both design and performance documentation.\r\n\r\n[duperemove-tests](https://github.com/markfasheh/duperemove-tests) has\r\na growing assortment of regression tests.\r\n\r\n[Duperemove web page](http://markfasheh.github.io/duperemove/)\r\n",
  "note": "Don't delete this file! It's used internally to help with page regeneration."
}