# gfal2-util

## Description

GFAL 2 utils are a group of command line tools for file manipulations with any protocol managed by gfal2

## Installation

    yum install gfal2-util gfal2-all

## List of command line tools

gfal-cat
  Concatenate a list of files into the standard output.
gfal-chmod
  Change permissions for a file or directory.
gfal-copy
  Copy files from source to destination(s).
gfal-bringonline
  Perform a staging operation on the given SURL.
gfal-archivepoll
  Perform an archive polling operation on a given SURL.
gfal-legacy-register
  Register a new replica into a catalog.
gfal-legacy-replicas
  List the replicas of an entry in a catalog.
gfal-legacy-unregister
  Remove a replica from a catalog.
gfal-ls
  Equivalent to the system `ls` command.
gfal-mkdir
  Create a directory.
gfal-rename
  Rename a file or directory.
gfal-rm
  Remove a file or directory (with -r).
gfal-save
  Write standard input to a file.
gfal-stat
  Show extended information about a file or directory.
gfal-sum
  Retrieve the file checksum.
gfal-xattr
  List or display a file's extended attributes.

## Third party copy

To understand some parts of this documentation, it is useful to know the difference between a third party copy and a regular one.

Third party copy: When doing a copy between two remote endpoints, the data is sent directly between the two participating storages.
Non-third party copy: The data goes through the client doing the copy.

Gfal2 is capable of doing both, depending on the protocol combination. Some protocols do support third party copy (see below), some others do not. In any case, when copying between two different protocols, the copy will not be third party copy (e.g from GridFTP to DAV).
SRM is an exception to this.

## Protocol support

Gfal2 relies on plugins to add support for different protocols. Depending on your installation, some may not be available.
To use a given protocol, the general syntax is protocol://host[:port]/path. For instance:

- gsiftp://dpm.cern.ch/dpm/cern.ch/home/dteam
- davs://dpm.cern.ch/dpm/cern.ch/home/dteam
- davs://dpm.cern.ch:443/dpm/cern.ch/home/dteam

SRM is an exception to this.

You can check what is installed using rpm -qa | grep gfal2-plugin. The normally available protocols are:

file://
  Local file access. It can be omitted (i.e. gfal-ls /).
gsiftp://
  GridFTP protocol. It has third-party copy support.
srm://
  SRM protocol. Its support for third-party copies depends on the transfer URL negotiated with the endpoint, and the protocol used for the other peer.
  gfal2 is capable of resolving the full SURL using the information system (e.g srm://host/path), but otherwise, you can also use the full SURL (e.g srm://host:8446/srm/managerv2?SFN=/path)
root://
  XRootD protocol. Some storages support third party copies, others do not.
dav(s)://
  WebDAV protocol. Some storages support third party copies, but to enable it, you need to use davs+3rd://
https://
  Plain HTTPS protocol. Some storages support third party copies, but to enable it, you need to use https+3rd://
s3://
  S3 protocol. Does not support third party copy, but it could still be achieved if the peer is a davs+3rd endpoint
lfc://
  LFC endpoints. Requires the endpoint to be part of the url (lfc://lfc.cern.ch/path)
guid:
  LFC endpoints, by GUID. It uses the LFC_HOST configured under /etc/gfal2.d/lfc_plugin.conf
rfio://
  RFIO protocol.
dcap://
  DCAP protocol, only supported by dCache endpoints.

These plugins may not be normally installed.

sftp://
  Experimental. SFTP support. Works with an SSH private key, instead of a X509 proxy.
mock://
  Useful for testing. Allows to force a given reply depending on the URL used. See its documentation.

## Configuration

All plugin configuration files are stored under /etc/gfal2.d/. You can navigate them and have a look at the different configuration options on the gfal2 repository.

In any case, for all commands, any value can be overridden by using the flag -D"GROUP:PARAM=VALUE"

For example, let's say we want to do a listing of an SRM endpoint, but using a different infosys endpoint.

    gfal-ls -D"BDII:LCG_GFAL_INFOSYS=myinfosys.cern.ch:2170" srm://endpoint.cern.ch/dpm/cern.ch/home/dteam/

## Examples

### Uploading a file to the Grid

    gfal-copy -fpK /home/doe/file1 srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1

In this example, the flag -f tells gfal-copy to overwrite the destination if it already exists, -p enables automatic directory creation, and -K enables checksum validation.

### Copying files out of the Grid

gfal-copy can be used for uploading, downloading, and also third party copies. Same syntax applies.

    gfal-copy -fpK srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1 /home/doe/file1

### Listing files and directories

    gfal-ls -l srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/
    -rw-rw-r--   1 46    45          192 Jun 21  2016 853.test
    -rw-rw-r--   1 46    45       641064 Jun 16  2016 aaaabbaaa
    -rw-rw-r--   1 46    48            0 Jun  6  2016 build.out
    -rw-rw-r--   1 46    49        49185 Nov 18 16:04 dmc-901.test
    -rw-rw-r--   1 46    49        49185 Nov 18 16:05 dmc-902.test
    -rw-rw-r--   1 46    50      5279801 Apr 28 16:20 test

In this example, -l enables long listing, which displays then mode, owner, size and mtime next to each entry. Without -l, the short version is displayed instead: only the names.

gfal-ls uses the environment variable LS_COLORS when -l is passed, if the terminal supports it.

### Get information about a file or directory

    gfal-stat srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/
      File: 'srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch'
      Size: 0	directory
    Access: (0755/drwxr-xr-x)	Uid: 45	Gid: 45
    Access: 1970-01-01 01:00:00.000000
    Modify: 2015-06-10 18:26:00.000000
    Change: 2017-05-09 11:42:52.000000

### Get the checksum of a file

    gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa adler32
    srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa c88e70d6

    gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa md5
    srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa 64fd36ae477e970b06949fe90d0c8c20

### Get information about space tokens

    gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken
    ["65e3ad04-ada1-11e6-b531-c860001bd938","bd1dae50-108c-11e7-82fa-c860001bd938","77056126-194f-11e7-82fa-c860001bd938","4ad03b22-2a92-11e7-82fa-c860001bd938","45e35c78-2b52-11e7-82fa-c860001bd938"]

    gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken.token?65e3ad04-ada1-11e6-b531-c860001bd938
    { "spacetoken": "65e3ad04-ada1-11e6-b531-c860001bd938", "owner": "root", "totalsize": 14937625, "unusedsize": 112193261767, "usedsize": 0, "guaranteedsize": 107374182400, "lifetimeleft": -1, "retention": "CUSTODIAL", "accesslatency": "NEARLINE" }

## Apendix: lcg-util equivalence table

Description | lcg-util | gfal-util
--- | --- | ---
Add an alias for a given GUID | lcg-aa | Deprecated
Bring SURLs online | lcg-bringonline | gfal-bringonline
Poll archiving status of SURLs | gfal-archivepoll
Copy files with no catalog involved | lcg-cp | gfal-copy
Copy and register a file | lcg-cr | gfal-copy
Delete a file / directory | lcg-del | gfal-rm
Get the file checksum | lcg-get-checksum | gfal-sum
Get the TURLs for given SURLs | lcg-getturls | gfal-xattr "user.replicas"
Get the TURL for a given SURL | lcg-gt | gfal-xattr "user.replicas"
List aliases for a given LFN/GUID | lcg-la | Deprecated
Get the GUID for a given LFN | lcg-lg | gfal-xattr "user.guid"
Lists the replicas for a given LFN | lcg-lr | gfal-xattr "user.replicas"
List information / directory | lcg-ls | gfal-ls
Remove an alias | lcg-ra | Deprecated
Copy between Ses using the catalog | lcg-rep | gfal-copy
Register a file in a catalog | lcg-rf | gfal-copy
Set a file status to done | lcg-sd | Deprecated*
Get space tokens associated to a description | lcg-stmd | gfal-xattr "spacetoken"/"spacetoken.token?$tok"/"spacetoken.description?$desc"
Unregister a file | lcg-uf | gfal-legacy-unregister
Create a directory |  | gfal-mkdir
Dump the standard input into a file |  | gfal-save
Dump a file into the standard output |  | gfal-cat

(*) These actions are actually performed internally by the SRM plugin when needed.

## License

This software is licensed under the Apache 2 License. See LICENSE for details.

Copyright (c) 2013-2022 CERN Copyright (c) 2012-2013 Members of the EMI Collaboration See http://www.eu-emi.eu/partners for details on the copyright holders.

## Release Notes

See RELEASE-NOTES for a detailed changelog.

## Contact

You can notify bugs or ask for feature requests via

Tracker
  https://its.cern.ch/jira/browse/DMC/component/12918
Mailing list
  dmc-support@cern.ch