transfer.sh/README.md

transfer.sh

Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance.

Transfer.sh currently supports the s3 (Amazon S3) and local file system (local).

Usage

Upload:

$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt

Encrypt & Upload:

$ cat /tmp/hello.txt|gpg -ac -o-|curl -X PUT --upload-file "-" https://file.unsha.re/test.txt

Download & Decrypt:

$ curl https://file.unsha.re/1lDau/test.txt|gpg -o- > /tmp/hello.txt

Deleting

$ curl -X DELETE <X-Url-Delete Response Header URL>

Request Headers

Max-Downloads

$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt -H "Max-Downloads: 1" # Limit the number of downloads

Max-Days

$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt -H "Max-Days: 1" # Set the number of days before deletion

Response Headers

X-Url-Delete

The URL used to request the deletion of a file and returned as a response header.

curl -sD - --upload-file ./hello https://file.unsha.re/hello.txt | grep 'X-Url-Delete'
X-Url-Delete: https://file.unsha.re/hello.txt/BAYh0/hello.txt/PDw0NHPcqU

Examples

See good usage examples on examples.md

Link aliases

Create direct download link:

https://file.unsha.re/1lDau/test.txt --> https://file.unsha.re/get/1lDau/test.txt

Inline file:

https://file.unsha.re/1lDau/test.txt --> https://file.unsha.re/inline/1lDau/test.txt

Usage

Parameter	Description	Value	Env
listener	port to use for http (:80)		LISTENER
profile-listener	port to use for profiler (:6060)		PROFILE_LISTENER
force-https	redirect to https	false	FORCE_HTTPS
tls-listener	port to use for https (:443)		TLS_LISTENER
tls-listener-only	flag to enable tls listener only		TLS_LISTENER_ONLY
tls-cert-file	path to tls certificate		TLS_CERT_FILE
tls-private-key	path to tls private key		TLS_PRIVATE_KEY
http-auth-user	user for basic http auth on upload		HTTP_AUTH_USER
http-auth-pass	pass for basic http auth on upload		HTTP_AUTH_PASS
ip-whitelist	comma separated list of ips allowed to connect to the service		IP_WHITELIST
ip-blacklist	comma separated list of ips not allowed to connect to the service		IP_BLACKLIST
temp-path	path to temp folder	system temp	TEMP_PATH
web-path	path to static web files (for development or custom front end)		WEB_PATH
proxy-path	path prefix when service is run behind a proxy		PROXY_PATH
proxy-port	port of the proxy when the service is run behind a proxy		PROXY_PORT
email-contact	email contact for the front end		EMAIL_CONTACT
provider	which storage provider to use	(s3 or local)
aws-access-key	aws access key		AWS_ACCESS_KEY
aws-secret-key	aws access key		AWS_SECRET_KEY
bucket	aws bucket		BUCKET
s3-endpoint	Custom S3 endpoint.		S3_ENDPOINT
s3-region	region of the s3 bucket	eu-west-1	S3_REGION
s3-no-multipart	disables s3 multipart upload	false	S3_NO_MULTIPART
s3-path-style	Forces path style URLs, required for Minio.	false	S3_PATH_STYLE
basedir	path storage for local provider		BASEDIR
lets-encrypt-hosts	hosts to use for lets encrypt certificates (comma seperated)		HOSTS
log	path to log file		LOG
cors-domains	comma separated list of domains for CORS, setting it enable CORS		CORS_DOMAINS
rate-limit	request per minute		RATE_LIMIT
max-upload-size	max upload size in kilobytes		MAX_UPLOAD_SIZE
purge-days	number of days after the uploads are purged automatically		PURGE_DAYS
purge-interval	interval in hours to run the automatic purge for (not applicable to S3)		PURGE_INTERVAL
random-token-length	length of the random token for the upload path (double the size for delete path)	6	RANDOM_TOKEN_LENGTH

If you want to use TLS using lets encrypt certificates, set lets-encrypt-hosts to your domain, set tls-listener to :443 and enable force-https.

If you want to use TLS using your own certificates, set tls-listener to :443, force-https, tls-cert-file and tls-private-key.

Development

Switched to GO111MODULE

go run main.go --provider=local --listener :8080 --temp-path=/tmp/ --basedir=/tmp/

Build

$ git clone git@github.com:zenofile/transfer.sh.git
$ cd transfer.sh
$ go build -o transfersh main.go

Docker

For easy deployment, we've created an official Docker container. There are two variants, differing only by which user runs the process.

The default one will run as root:

docker run --publish 8080:8080 zenofile/transfer.sh:latest --provider local --basedir /tmp/

The one tagged with the suffix -noroot will use 5000 as both UID and GID:

docker run --publish 8080:8080 zenofile/transfer.sh:latest-noroot --provider local --basedir /tmp/

Building the Container

You can also build the container yourself. This allows you to choose which UID/GID will be used, e.g. when using NFS mounts:

# Build arguments:
# * RUNAS: If empty, the container will run as root.
#          Set this to anything to enable UID/GID selection.
# * PUID:  UID of the process. Needs RUNAS != "". Defaults to 5000.
# * PGID:  GID of the process. Needs RUNAS != "". Defaults to 5000.

docker build -t transfer.sh-noroot --build-arg RUNAS=doesntmatter --build-arg PUID=1337 --build-arg PGID=1338 .

S3 Usage

For the usage with a AWS S3 Bucket, you just need to specify the following options:

• provider
• aws-access-key
• aws-secret-key
• bucket
• s3-region

If you specify the s3-region, you don't need to set the endpoint URL since the correct endpoint will used automatically.

Custom S3 providers

To use a custom non-AWS S3 provider, you need to specify the endpoint as defined from your cloud provider.

Shell script

See transfer.sh.

Now you can use transfer function

$ transfer hello.txt

Bash and zsh (with delete url, delete token output and prompt before uploading)

Add this to .bashrc or .zshrc or its equivalent

Expand

transfer()
{
    local file
    declare -a file_array
    file_array=("${@}")

    if [[ "${file_array[@]}" == "" || "${1}" == "--help" || "${1}" == "-h" ]]
    then
        echo "${0} - Upload arbitrary files to \"transfer.sh\"."
        echo ""
        echo "Usage: ${0} [options] [<file>]..."
        echo ""
        echo "OPTIONS:"
        echo "  -h, --help"
        echo "      show this message"
        echo ""
        echo "EXAMPLES:"
        echo "  Upload a single file from the current working directory:"
        echo "      ${0} \"image.img\""
        echo ""
        echo "  Upload multiple files from the current working directory:"
        echo "      ${0} \"image.img\" \"image2.img\""
        echo ""
        echo "  Upload a file from a different directory:"
        echo "      ${0} \"/tmp/some_file\""
        echo ""
        echo "  Upload all files from the current working directory. Be aware of the webserver's rate limiting!:"
        echo "      ${0} *"
        echo ""
        echo "  Upload a single file from the current working directory and filter out the delete token and download link:"
        echo "      ${0} \"image.img\" | awk --field-separator=\": \" '/Delete token:/ { print \$2 } /Download link:/ { print \$2 }'"
        echo ""
        echo "  Show help text from \"transfer.sh\":"
        echo "      curl --request GET \"https://file.unsha.re\""
        return 0
    else
        for file in "${file_array[@]}"
        do
            if [[ ! -f "${file}" ]]
            then
                echo -e "\e[01;31m'${file}' could not be found or is not a file.\e[0m" >&2
                return 1
            fi
        done
        unset file
    fi

    local upload_files
    local curl_output
    local awk_output
    local filename

    du --total --block-size="K" --dereference "${file_array[@]}" >&2
    # be compatible with "bash"
    if [[ "${ZSH_NAME}" == "zsh" ]]
    then
        read $'upload_files?\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m'
    elif [[ "${BASH}" == *"bash"* ]]
    then
        read -p $'\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m' upload_files
    fi

    case "${upload_files:-y}" in
        "y"|"Y")
            # for the sake of the progress bar, execute "curl" for each file.
            # the parameters "--include" and "--form" will suppress the progress bar.
            for file in "${file_array[@]}"
            do
                filename="${file##*/}"
                # show delete link and filter out the delete token from the response header after upload.
                # it is important to save "curl's" "stdout" via a subshell to a variable or redirect it to another command,
                # which just redirects to "stdout" in order to have a sane output afterwards.
                # the progress bar is redirected to "stderr" and is only displayed,
                # if "stdout" is redirected to something; e.g. ">/dev/null", "tee /dev/null" or "| <some_command>".
                # the response header is redirected to "stdout", so redirecting "stdout" to "/dev/null" does not make any sense.
                # redirecting "curl's" "stderr" to "stdout" ("2>&1") will suppress the progress bar.
                curl_output=$(curl --request PUT --progress-bar --dump-header - --upload-file "${file}" "https://file.unsha.re/${filename}")
                awk_output=$(awk \
                    'gsub("\r", "", $0) && tolower($1) ~ /x-url-delete/ \
                    {
                        delete_link=$2;
                        print "Delete command: curl --request DELETE " "\""delete_link"\"";

                        gsub(".*/", "", delete_link);
                        delete_token=delete_link;
                        print "Delete token: " delete_token;
                    }

                    END{
                        print "Download link: " $0;
                    }' <<< "${curl_output}")

                # return the results via "stdout", "awk" does not do this for some reason.
                echo -e "${awk_output}\n"

                # avoid rate limiting as much as possible; nginx: too many requests.
                if (( ${#file_array[@]} > 4 ))
                then
                    sleep 5
                fi
            done
            ;;

        "n"|"N")
            return 1
            ;;

        *)
            echo -e "\e[01;31mWrong input: '${upload_files}'.\e[0m" >&2
            return 1
    esac
}

Sample output

$ ls -lh
total 20M
-rw-r--r-- 1 <some_username> <some_username> 10M Apr  4 21:08 image.img
-rw-r--r-- 1 <some_username> <some_username> 10M Apr  4 21:08 image2.img
$ transfer image*
10240K  image2.img
10240K  image.img
20480K  total
Do you really want to upload the above files (2) to "transfer.sh"? (Y/n):
######################################################################################################################################################################################################################################## 100.0%
Delete command: curl --request DELETE "https://file.unsha.re/wJw9pz/image2.img/mSctGx7pYCId"
Delete token: mSctGx7pYCId
Download link: https://file.unsha.re/wJw9pz/image2.img

######################################################################################################################################################################################################################################## 100.0%
Delete command: curl --request DELETE "https://file.unsha.re/ljJc5I/image.img/nw7qaoiKUwCU"
Delete token: nw7qaoiKUwCU
Download link: https://file.unsha.re/ljJc5I/image.img

$ transfer "image.img" | awk --field-separator=": " '/Delete token:/ { print $2 } /Download link:/ { print $2 }'
10240K  image.img
10240K  total
Do you really want to upload the above files (1) to "transfer.sh"? (Y/n):
######################################################################################################################################################################################################################################## 100.0%
tauN5dE3fWJe
https://file.unsha.re/MYkuqn/image.img