File Transfer¶

Storage¶

storage:tag_s3¶

Executes as PutObjectTagging on S3 objects under the specified location

Example:

{
    "tool": "storage:tag_s3",
    "parameters": {
        "location": "s3://{aws-credentials}@com.castlabs.vtk.test/contentpath/",
        "files": [
            "file.txt"
        ],
        "tags": {
            "foo": "bar",
            "baz": "qux"
        }
    }
}

Parameter	Properties	Type	Description
tags	required	dict	tags to apply
location	optional		A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]		Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional	str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional		The custom endpoint URL of a s3 bucket
http_credential_source	optional		HTTP Credential Source
az_acc_name	optional		Azure Storage Account Name
az_container	optional		Azure Storage Container
az_acc_key	optional		Azure Account Access Key
az_sas_token	optional		Azure Shared Access Token (?sv=..)
gc_adc_string	optional		Google Cloud ADC base64 (with padding)
role_arn	optional		The role to be assumed for executing the tool

storage:check¶

Checks if files in a remote location exist. This step either

fails if these files exist (mode: ‘FAIL_ON_PRESENCE’), fast-forwards the job to success without executing any further step (mode: ‘COMPLETE_ON_PRESENCE’) fails if the specified files do not exist (mode: FAIL_ON_ABSENCE’)

Example:

{ “tool”: “storage:check”, “parameters”: { “location”: “s3://{aws-credentials}@com.castlabs.vtk.test/contentpath”, “files”: [ “tos_3min_video_960x540_1mbps.mp4” ], “mode”: “COMPLETE_ON_PRESENCE” } },

Parameter	Properties	Type	Choice	Description
mode	optional		‘FAIL_ON_PRESENCE’, ‘COMPLETE_ON_PRESENCE’, ‘FAIL_ON_ABSENCE’	FAIL_ON_PRESENCE: If the files are already present the job will fail.
COMPLETE_ON_PRESENCE: If the files are already present the job will succeed without executing any subsequent steps.
FAIL_ON_ABSENCE: If the files are not there (e.g. if subsequent steps require those files) the job will fail.
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional	str		A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

storage:get¶

File Downloader Tool for S3/http(s)/ftp(s)/GS and Azure Blob Storage

The following schemes are supported and can be used as the location:

s3:// - AWS S3 (HTTPS)
http:// - HTTP
https:// - HTTPS
ftp:// - FTP
ftps:// - FTP with TLS
gs:// - Google Cloud Storage

For Azure Blob Storage location may not be used, instead provide:

az_acc_name
az_acc_key / az_sas_token
az_container

Example with S3:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Example using the FTP protocol for download:

{
"tool": "storage:get",
"parameters": {
    "location": "ftp://{USERNAME}:{PASSWORD}@{HOST}{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Downloading from Backblaze B2 storage using their S3 compatible API:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Downloading from custom s3 storage using their S3 compatible API:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ],
    "s3_endpoint_url": "{ENDPOINT_URL}"
}
}

Example using Azure account name and access key:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_acc_key": "account_access_key",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example using Azure account name and SAS Token:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example using Azure account name and SAS Token:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example downloading blobs from Google Cloud Storage:

{
"tool": "storage:get",
    "parameters": {
        "location": "gs://bucket-name/",
        "gc_adc_string": "{google-adc-string-b64}",
        "files": [
            "*"
        ]
    }
}

Parameter	Properties	Default	Type	Description
remove_query_string	optional	false	bool	HTTP sources only: remove the query string from the output filename
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional		str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

storage:mount¶

Mount S3 bucket and make the specified files available locally

Requires role_arn to be set at the job level.

NOTE: Deprecated. Use the job-level mounts array instead, which mounts read-only S3 prefixes into every task on demand (no per-task tool, supports credential dispensers and directory mounts). See the Jobs documentation, section “Mounts”.

Example:

{
"tool": "storage:mount",
"parameters": {
    "location": "s3://{BUCKET}/{PATH}/",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Parameter	Properties	Default	Type	Choice	Description
location	required				S3 path which is mounted
files	required [list value]				Files that become available from the S3 location (list of strings). Wildcards not supported

storage:put¶

File Upload tool for AWS-S3 or Azure Blob Storage

The tool understands the wildcard expansion characters:

\* - matches everything
? - matches any single character

Example with S3:

{
"tool": "storage:put",
"parameters": {
    "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4",
        "*.vtt", ...
    ],
    "s3_endpoint_url": "{ENDPOINT_URL}"
}
}

Example using Azure account name and access key:

{
"tool": "storage:put",
"parameters": {
    "az_acc_name": "account_name",
    "az_acc_key": "account_access_key",
    "az_container": "container",
    "overwrite": False,
    "files": [
        "hls/*"
    ]
}
}

Example using Azure account name and SAS token:

{
"tool": "storage:put",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "overwrite": False,
    "files": [
        "hls/*"
    ]
}
}

Files can be uploaded under a different name using rename. The keys are local files (exactly as listed in files, relative to base_dir), the values are the names the files get at the destination, relative to the location:

{
"tool": "storage:put",
"parameters": {
    "location": "s3://{BUCKET}/{PATH}",
    "files": [
        "input/kdjshkjfhsd.mp4"
    ],
    "rename": {
        "input/kdjshkjfhsd.mp4": "video.mp4"
    }
}
}

Files that are not mentioned in rename keep their local name. A rename target may contain a sub-directory ("hls/video.mp4") but must not be absolute or traverse upwards.

Parameter	Properties	Default	Type	Description
overwrite	optional	false	bool	If set to true then existing files will be overwritten without any further precaution.
rename	optional		dict	Optional mapping of local file name (relative to `base_dir`, as listed in `files`) to the name the file is stored under at the destination (relative to `location`). Files not listed here keep their local name.
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional		str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

Next topic: Packaging

Previous topic: Transcoding