File Transfer¶

Storage¶

storage:put¶

File Upload tool for AWS-S3 or Azure Blob Storage

The tool understands the wildcard expansion characters:

\* - matches everything
? - matches any single character

Example with S3:

{
"tool": "storage:put",
"parameters": {
    "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4",
        "*.vtt", ...
    ],
    "s3_endpoint_url": "{ENDPOINT_URL}"
}
}

Example using Azure account name and access key:

{
"tool": "storage:put",
"parameters": {
    "az_acc_name": "account_name",
    "az_acc_key": "account_access_key",
    "az_container": "container",
    "overwrite": False,
    "files": [
        "hls/*"
    ]
}
}

Example using Azure account name and SAS token:

{
"tool": "storage:put",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "overwrite": False,
    "files": [
        "hls/*"
    ]
}
}

Parameter	Properties	Default	Type	Description
overwrite	optional	false	bool	If set to true then existing files will be overwritten without any further precaution.
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional		str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

storage:mount¶

Mount S3 bucket and make the specified files available locally

Requires role_arn to be set at the job level.

Example:

{
"tool": "storage:mount",
"parameters": {
    "location": "s3://{BUCKET}/{PATH}/",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Parameter	Properties	Default	Type	Choice	Description
location	required				S3 path which is mounted
files	required [list value]				Files that become available from the S3 location (list of strings). Wildcards not supported

storage:tag_s3¶

Executes as PutObjectTagging on S3 objects under the specified location

Example:

{
    "tool": "storage:tag_s3",
    "parameters": {
        "location": "s3://{aws-credentials}@com.castlabs.vtk.test/contentpath/",
        "files": [
            "file.txt"
        ],
        "tags": {
            "foo": "bar",
            "baz": "qux"
        }
    }
}

Parameter	Properties	Type	Description
tags	required	dict	tags to apply
location	optional		A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]		Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional	str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional		The custom endpoint URL of a s3 bucket
http_credential_source	optional		HTTP Credential Source
az_acc_name	optional		Azure Storage Account Name
az_container	optional		Azure Storage Container
az_acc_key	optional		Azure Account Access Key
az_sas_token	optional		Azure Shared Access Token (?sv=..)
gc_adc_string	optional		Google Cloud ADC base64 (with padding)
role_arn	optional		The role to be assumed for executing the tool

storage:get¶

File Downloader Tool for S3/http(s)/ftp(s)/GS and Azure Blob Storage

The following schemes are supported and can be used as the location:

s3:// - AWS S3 (HTTPS)
http:// - HTTP
https:// - HTTPS
ftp:// - FTP
ftps:// - FTP with TLS
gs:// - Google Cloud Storage

For Azure Blob Storage location may not be used, instead provide:

az_acc_name
az_acc_key / az_sas_token
az_container

Example with S3:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Example using the FTP protocol for download:

{
"tool": "storage:get",
"parameters": {
    "location": "ftp://{USERNAME}:{PASSWORD}@{HOST}{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Downloading from Backblaze B2 storage using their S3 compatible API:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ]
}
}

Downloading from custom s3 storage using their S3 compatible API:

{
"tool": "storage:get",
"parameters": {
    "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
    "files": [
        "video.mp4",
        "audio_en.mp4",
        "audio_es.mp4"
    ],
    "s3_endpoint_url": "{ENDPOINT_URL}"
}
}

Example using Azure account name and access key:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_acc_key": "account_access_key",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example using Azure account name and SAS Token:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example using Azure account name and SAS Token:

{
"tool": "storage:get",
"parameters": {
    "az_acc_name": "account_name",
    "az_sas_token": "sas_token",
    "az_container": "container",
    "files": [
        "file.mxf"
    ]
}
}

Example downloading blobs from Google Cloud Storage:

{
"tool": "storage:get",
    "parameters": {
        "location": "gs://bucket-name/",
        "gc_adc_string": "{google-adc-string-b64}",
        "files": [
            "*"
        ]
    }
}

Parameter	Properties	Default	Type	Description
remove_query_string	optional	false	bool	HTTP sources only: remove the query string from the output filename
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional		str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

storage:check¶

Checks if files in a remote location exist. This step either

fails if these files exist (mode: ‘FAIL_ON_PRESENCE’), fast-forwards the job to success without executing any further step (mode: ‘COMPLETE_ON_PRESENCE’) fails if the specified files do not exist (mode: FAIL_ON_ABSENCE’)

Example:

{ “tool”: “storage:check”, “parameters”: { “location”: “s3://{aws-credentials}@com.castlabs.vtk.test/contentpath”, “files”: [ “tos_3min_video_960x540_1mbps.mp4” ], “mode”: “COMPLETE_ON_PRESENCE” } },

Parameter	Properties	Type	Choice	Description
mode	optional		‘FAIL_ON_PRESENCE’, ‘COMPLETE_ON_PRESENCE’, ‘FAIL_ON_ABSENCE’	FAIL_ON_PRESENCE: If the files are already present the job will fail.
COMPLETE_ON_PRESENCE: If the files are already present the job will succeed without executing any subsequent steps.
FAIL_ON_ABSENCE: If the files are not there (e.g. if subsequent steps require those files) the job will fail.
location	optional			A base URL for the files. For HTTP get operations, can also be the complete URL.
files	optional [list value]			Filenames to be downloaded, uploaded, or checked, relative to the base URL. Not used for HTTP sources with a complete URL as the location.
base_dir	optional	str		A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

Next topic: Packaging

Previous topic: Transcoding