325 lines
13 KiB
Markdown
325 lines
13 KiB
Markdown
# transfer.sh [](https://goreportcard.com/report/github.com/zenofile/transfer.sh) [](https://hub.docker.com/r/zeno/transfer.sh/) [](https://github.com/zenofile/transfer.sh/actions/workflows/test.yml?query=branch%3Amain)
|
|
|
|
Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance.
|
|
|
|
Transfer.sh currently supports the s3 (Amazon S3) and local file system (local).
|
|
|
|
## Usage
|
|
|
|
### Upload:
|
|
```bash
|
|
$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt
|
|
```
|
|
|
|
### Encrypt & Upload:
|
|
```bash
|
|
$ cat /tmp/hello.txt|gpg -ac -o-|curl -X PUT --upload-file "-" https://file.unsha.re/test.txt
|
|
````
|
|
|
|
### Download & Decrypt:
|
|
```bash
|
|
$ curl https://file.unsha.re/1lDau/test.txt|gpg -o- > /tmp/hello.txt
|
|
```
|
|
|
|
### Deleting
|
|
```bash
|
|
$ curl -X DELETE <X-Url-Delete Response Header URL>
|
|
```
|
|
|
|
## Request Headers
|
|
|
|
### Max-Downloads
|
|
```bash
|
|
$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt -H "Max-Downloads: 1" # Limit the number of downloads
|
|
```
|
|
|
|
### Max-Days
|
|
```bash
|
|
$ curl --upload-file ./hello.txt https://file.unsha.re/hello.txt -H "Max-Days: 1" # Set the number of days before deletion
|
|
```
|
|
|
|
## Response Headers
|
|
|
|
### X-Url-Delete
|
|
|
|
The URL used to request the deletion of a file and returned as a response header.
|
|
```bash
|
|
curl -sD - --upload-file ./hello https://file.unsha.re/hello.txt | grep 'X-Url-Delete'
|
|
X-Url-Delete: https://file.unsha.re/hello.txt/BAYh0/hello.txt/PDw0NHPcqU
|
|
```
|
|
|
|
## Examples
|
|
|
|
See good usage examples on [examples.md](examples.md)
|
|
|
|
## Link aliases
|
|
|
|
Create direct download link:
|
|
|
|
https://file.unsha.re/1lDau/test.txt --> https://file.unsha.re/get/1lDau/test.txt
|
|
|
|
Inline file:
|
|
|
|
https://file.unsha.re/1lDau/test.txt --> https://file.unsha.re/inline/1lDau/test.txt
|
|
|
|
## Usage
|
|
|
|
Parameter | Description | Value | Env
|
|
--- | --- | --- | ---
|
|
listener | port to use for http (:80) | | LISTENER |
|
|
profile-listener | port to use for profiler (:6060) | | PROFILE_LISTENER |
|
|
force-https | redirect to https | false | FORCE_HTTPS
|
|
tls-listener | port to use for https (:443) | | TLS_LISTENER |
|
|
tls-listener-only | flag to enable tls listener only | | TLS_LISTENER_ONLY |
|
|
tls-cert-file | path to tls certificate | | TLS_CERT_FILE |
|
|
tls-private-key | path to tls private key | | TLS_PRIVATE_KEY |
|
|
http-auth-user | user for basic http auth on upload | | HTTP_AUTH_USER |
|
|
http-auth-pass | pass for basic http auth on upload | | HTTP_AUTH_PASS |
|
|
ip-whitelist | comma separated list of ips allowed to connect to the service | | IP_WHITELIST |
|
|
ip-blacklist | comma separated list of ips not allowed to connect to the service | | IP_BLACKLIST |
|
|
temp-path | path to temp folder | system temp | TEMP_PATH |
|
|
web-path | path to static web files (for development or custom front end) | | WEB_PATH |
|
|
proxy-path | path prefix when service is run behind a proxy | | PROXY_PATH |
|
|
proxy-port | port of the proxy when the service is run behind a proxy | | PROXY_PORT |
|
|
email-contact | email contact for the front end | | EMAIL_CONTACT |
|
|
provider | which storage provider to use | (s3 or local) |
|
|
aws-access-key | aws access key | | AWS_ACCESS_KEY |
|
|
aws-secret-key | aws access key | | AWS_SECRET_KEY |
|
|
bucket | aws bucket | | BUCKET |
|
|
s3-endpoint | Custom S3 endpoint. | | S3_ENDPOINT |
|
|
s3-region | region of the s3 bucket | eu-west-1 | S3_REGION |
|
|
s3-no-multipart | disables s3 multipart upload | false | S3_NO_MULTIPART |
|
|
s3-path-style | Forces path style URLs, required for Minio. | false | S3_PATH_STYLE |
|
|
basedir | path storage for local provider | | BASEDIR |
|
|
lets-encrypt-hosts | hosts to use for lets encrypt certificates (comma seperated) | | HOSTS |
|
|
log | path to log file| | LOG |
|
|
cors-domains | comma separated list of domains for CORS, setting it enable CORS | | CORS_DOMAINS |
|
|
rate-limit | request per minute | | RATE_LIMIT |
|
|
max-upload-size | max upload size in kilobytes | | MAX_UPLOAD_SIZE |
|
|
purge-days | number of days after the uploads are purged automatically | | PURGE_DAYS |
|
|
purge-interval | interval in hours to run the automatic purge for (not applicable to S3) | | PURGE_INTERVAL |
|
|
random-token-length | length of the random token for the upload path (double the size for delete path) | 6 | RANDOM_TOKEN_LENGTH |
|
|
|
|
If you want to use TLS using lets encrypt certificates, set lets-encrypt-hosts to your domain, set tls-listener to :443 and enable force-https.
|
|
|
|
If you want to use TLS using your own certificates, set tls-listener to :443, force-https, tls-cert-file and tls-private-key.
|
|
|
|
## Development
|
|
|
|
Switched to GO111MODULE
|
|
|
|
```bash
|
|
go run main.go --provider=local --listener :8080 --temp-path=/tmp/ --basedir=/tmp/
|
|
```
|
|
|
|
## Build
|
|
|
|
```bash
|
|
$ git clone git@github.com:zenofile/transfer.sh.git
|
|
$ cd transfer.sh
|
|
$ go build -o transfersh main.go
|
|
```
|
|
|
|
## Docker
|
|
|
|
For easy deployment, we've created an official Docker container. There are two variants, differing only by which user runs the process.
|
|
|
|
The default one will run as `root`:
|
|
|
|
```bash
|
|
docker run --publish 8080:8080 zenofile/transfer.sh:latest --provider local --basedir /tmp/
|
|
```
|
|
|
|
The one tagged with the suffix `-noroot` will use `5000` as both UID and GID:
|
|
```bash
|
|
docker run --publish 8080:8080 zenofile/transfer.sh:latest-noroot --provider local --basedir /tmp/
|
|
```
|
|
|
|
### Building the Container
|
|
You can also build the container yourself. This allows you to choose which UID/GID will be used, e.g. when using NFS mounts:
|
|
```bash
|
|
# Build arguments:
|
|
# * RUNAS: If empty, the container will run as root.
|
|
# Set this to anything to enable UID/GID selection.
|
|
# * PUID: UID of the process. Needs RUNAS != "". Defaults to 5000.
|
|
# * PGID: GID of the process. Needs RUNAS != "". Defaults to 5000.
|
|
|
|
docker build -t transfer.sh-noroot --build-arg RUNAS=doesntmatter --build-arg PUID=1337 --build-arg PGID=1338 .
|
|
```
|
|
|
|
## S3 Usage
|
|
|
|
For the usage with a AWS S3 Bucket, you just need to specify the following options:
|
|
- provider
|
|
- aws-access-key
|
|
- aws-secret-key
|
|
- bucket
|
|
- s3-region
|
|
|
|
If you specify the s3-region, you don't need to set the endpoint URL since the correct endpoint will used automatically.
|
|
|
|
### Custom S3 providers
|
|
|
|
To use a custom non-AWS S3 provider, you need to specify the endpoint as defined from your cloud provider.
|
|
|
|
## Shell script
|
|
|
|
See [transfer.sh](contrib/bash/transfer.sh).
|
|
|
|
#### Now you can use transfer function
|
|
```
|
|
$ transfer hello.txt
|
|
```
|
|
|
|
|
|
### Bash and zsh (with delete url, delete token output and prompt before uploading)
|
|
##### Add this to .bashrc or .zshrc or its equivalent
|
|
|
|
<details><summary>Expand</summary><p>
|
|
|
|
```bash
|
|
transfer()
|
|
{
|
|
local file
|
|
declare -a file_array
|
|
file_array=("${@}")
|
|
|
|
if [[ "${file_array[@]}" == "" || "${1}" == "--help" || "${1}" == "-h" ]]
|
|
then
|
|
echo "${0} - Upload arbitrary files to \"transfer.sh\"."
|
|
echo ""
|
|
echo "Usage: ${0} [options] [<file>]..."
|
|
echo ""
|
|
echo "OPTIONS:"
|
|
echo " -h, --help"
|
|
echo " show this message"
|
|
echo ""
|
|
echo "EXAMPLES:"
|
|
echo " Upload a single file from the current working directory:"
|
|
echo " ${0} \"image.img\""
|
|
echo ""
|
|
echo " Upload multiple files from the current working directory:"
|
|
echo " ${0} \"image.img\" \"image2.img\""
|
|
echo ""
|
|
echo " Upload a file from a different directory:"
|
|
echo " ${0} \"/tmp/some_file\""
|
|
echo ""
|
|
echo " Upload all files from the current working directory. Be aware of the webserver's rate limiting!:"
|
|
echo " ${0} *"
|
|
echo ""
|
|
echo " Upload a single file from the current working directory and filter out the delete token and download link:"
|
|
echo " ${0} \"image.img\" | awk --field-separator=\": \" '/Delete token:/ { print \$2 } /Download link:/ { print \$2 }'"
|
|
echo ""
|
|
echo " Show help text from \"transfer.sh\":"
|
|
echo " curl --request GET \"https://file.unsha.re\""
|
|
return 0
|
|
else
|
|
for file in "${file_array[@]}"
|
|
do
|
|
if [[ ! -f "${file}" ]]
|
|
then
|
|
echo -e "\e[01;31m'${file}' could not be found or is not a file.\e[0m" >&2
|
|
return 1
|
|
fi
|
|
done
|
|
unset file
|
|
fi
|
|
|
|
local upload_files
|
|
local curl_output
|
|
local awk_output
|
|
local filename
|
|
|
|
du --total --block-size="K" --dereference "${file_array[@]}" >&2
|
|
# be compatible with "bash"
|
|
if [[ "${ZSH_NAME}" == "zsh" ]]
|
|
then
|
|
read $'upload_files?\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m'
|
|
elif [[ "${BASH}" == *"bash"* ]]
|
|
then
|
|
read -p $'\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m' upload_files
|
|
fi
|
|
|
|
case "${upload_files:-y}" in
|
|
"y"|"Y")
|
|
# for the sake of the progress bar, execute "curl" for each file.
|
|
# the parameters "--include" and "--form" will suppress the progress bar.
|
|
for file in "${file_array[@]}"
|
|
do
|
|
filename="${file##*/}"
|
|
# show delete link and filter out the delete token from the response header after upload.
|
|
# it is important to save "curl's" "stdout" via a subshell to a variable or redirect it to another command,
|
|
# which just redirects to "stdout" in order to have a sane output afterwards.
|
|
# the progress bar is redirected to "stderr" and is only displayed,
|
|
# if "stdout" is redirected to something; e.g. ">/dev/null", "tee /dev/null" or "| <some_command>".
|
|
# the response header is redirected to "stdout", so redirecting "stdout" to "/dev/null" does not make any sense.
|
|
# redirecting "curl's" "stderr" to "stdout" ("2>&1") will suppress the progress bar.
|
|
curl_output=$(curl --request PUT --progress-bar --dump-header - --upload-file "${file}" "https://file.unsha.re/${filename}")
|
|
awk_output=$(awk \
|
|
'gsub("\r", "", $0) && tolower($1) ~ /x-url-delete/ \
|
|
{
|
|
delete_link=$2;
|
|
print "Delete command: curl --request DELETE " "\""delete_link"\"";
|
|
|
|
gsub(".*/", "", delete_link);
|
|
delete_token=delete_link;
|
|
print "Delete token: " delete_token;
|
|
}
|
|
|
|
END{
|
|
print "Download link: " $0;
|
|
}' <<< "${curl_output}")
|
|
|
|
# return the results via "stdout", "awk" does not do this for some reason.
|
|
echo -e "${awk_output}\n"
|
|
|
|
# avoid rate limiting as much as possible; nginx: too many requests.
|
|
if (( ${#file_array[@]} > 4 ))
|
|
then
|
|
sleep 5
|
|
fi
|
|
done
|
|
;;
|
|
|
|
"n"|"N")
|
|
return 1
|
|
;;
|
|
|
|
*)
|
|
echo -e "\e[01;31mWrong input: '${upload_files}'.\e[0m" >&2
|
|
return 1
|
|
esac
|
|
}
|
|
```
|
|
|
|
</p></details>
|
|
|
|
#### Sample output
|
|
```bash
|
|
$ ls -lh
|
|
total 20M
|
|
-rw-r--r-- 1 <some_username> <some_username> 10M Apr 4 21:08 image.img
|
|
-rw-r--r-- 1 <some_username> <some_username> 10M Apr 4 21:08 image2.img
|
|
$ transfer image*
|
|
10240K image2.img
|
|
10240K image.img
|
|
20480K total
|
|
Do you really want to upload the above files (2) to "transfer.sh"? (Y/n):
|
|
######################################################################################################################################################################################################################################## 100.0%
|
|
Delete command: curl --request DELETE "https://file.unsha.re/wJw9pz/image2.img/mSctGx7pYCId"
|
|
Delete token: mSctGx7pYCId
|
|
Download link: https://file.unsha.re/wJw9pz/image2.img
|
|
|
|
######################################################################################################################################################################################################################################## 100.0%
|
|
Delete command: curl --request DELETE "https://file.unsha.re/ljJc5I/image.img/nw7qaoiKUwCU"
|
|
Delete token: nw7qaoiKUwCU
|
|
Download link: https://file.unsha.re/ljJc5I/image.img
|
|
|
|
$ transfer "image.img" | awk --field-separator=": " '/Delete token:/ { print $2 } /Download link:/ { print $2 }'
|
|
10240K image.img
|
|
10240K total
|
|
Do you really want to upload the above files (1) to "transfer.sh"? (Y/n):
|
|
######################################################################################################################################################################################################################################## 100.0%
|
|
tauN5dE3fWJe
|
|
https://file.unsha.re/MYkuqn/image.img
|
|
```
|