File Transfer

File Transfer¶

Storage¶

storage:tag_s3¶

Executes as PutObjectTagging on S3 objects under the specified location

Example: { “tool”: “storage:tag_s3”, “parameters”: { “location”: “s3://{aws-credentials}@com.castlabs.vtk.test/contentpath/”, “files”: [ “file.txt” ], “tags”: { “foo”: “bar”, “baz”: “qux” } } }

Parameter	Properties	Type	Description
tags	required	dict	tags to apply
files	required [list value]		Filenames to be downloaded before processing (list of strings)
location	optional		A base URL from which files are downloaded (string)
base_dir	optional	str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional		The custom endpoint URL of a s3 bucket
http_credential_source	optional		HTTP Credential Source
az_acc_name	optional		Azure Storage Account Name
az_container	optional		Azure Storage Container
az_acc_key	optional		Azure Account Access Key
az_sas_token	optional		Azure Shared Access Token (?sv=..)
gc_adc_string	optional		Google Cloud ADC base64 (with padding)
role_arn	optional		The role to be assumed for executing the tool

storage:get¶

File Downloader Tool for S3/http(s)/ftp(s)/GS and Azure Blob Storage

The following schemes are supported and can be used as the location:

s3:// - AWS S3 (HTTPS)
http:// - HTTP
https:// - HTTPS
ftp:// - FTP
ftps:// - FTP with TLS
gs:// - Google Cloud Storage

For Azure Blob Storage location may not be used, instead provide:

az_acc_name
az_acc_key / az_sas_token
az_container

Example with S3:

{
    "tool": "storage:get",
    "parameters": {
        "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
        "files": [
            "video.mp4",
            "audio_en.mp4",
            "audio_es.mp4", ...
        ]
    }
}

Example using the FTP protocol for download:

{
    "tool": "storage:get",
    "parameters": {
        "location": "ftp://{USERNAME}:{PASSWORD}@{HOST}{PATH}",
        "files": [
            "video.mp4",
            "audio_en.mp4",
            "audio_es.mp4", ...
        ]
    }
}

Downloading from Backblaze B2 storage using their S3 compatible API:

{
    "tool": "storage:get",
    "parameters": {
        "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
        "files": [
            "video.mp4",
            "audio_en.mp4",
            "audio_es.mp4", ...
        ]
    }
}

Downloading from custom s3 storage using their S3 compatible API:

{
    "tool": "storage:get",
    "parameters": {
        "location": "s3://{keyID}:{applicationKey}@{ENDPOINT}/{BUCKET_NAME}/{PATH}",
        "files": [
            "video.mp4",
            "audio_en.mp4",
            "audio_es.mp4", ...
        ],
        "s3_endpoint_url": "{ENDPOINT_URL}"
    }
}

Example using Azure account name and access key:

    {
      "tool": "storage:get",
      "parameters": {
        "az_acc_name": "account_name",
        "az_acc_key": "account_access_key",
        "az_container": "container",
        "files": [
            "file.mxf"
        ]
      }
    }

Example using Azure account name and SAS Token:

    {
      "tool": "storage:get",
      "parameters": {
        "az_acc_name": "account_name",
        "az_sas_token": "sas_token",
        "az_container": "container",
        "files": [
            "file.mxf"
        ]
      }
    }

Example using Azure account name and SAS Token:

    {
      "tool": "storage:get",
      "parameters": {
        "az_acc_name": "account_name",
        "az_sas_token": "sas_token",
        "az_container": "container",
        "files": [
            "file.mxf"
        ]
      }
    }

Example downloading blobs from Google Cloud Storage:

    {
        "tool": "storage:get",
            "parameters": {
                "location": "gs://bucket-name/",
                "gc_adc_string": "{google-adc-string-b64}",
                "files": [
                    "*"
                ]
            }
    },

Parameter	Properties	Type	Description
files	required [list value]		Filenames to be downloaded before processing (list of strings)
location	optional		A base URL from which files are downloaded (string)
base_dir	optional	str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional		The custom endpoint URL of a s3 bucket
http_credential_source	optional		HTTP Credential Source
az_acc_name	optional		Azure Storage Account Name
az_container	optional		Azure Storage Container
az_acc_key	optional		Azure Account Access Key
az_sas_token	optional		Azure Shared Access Token (?sv=..)
gc_adc_string	optional		Google Cloud ADC base64 (with padding)
role_arn	optional		The role to be assumed for executing the tool

storage:mount¶

Mount S3 bucket and make the specified files available locally

Requires role_arn to be set at the job level.

Example: { “tool”: “storage:mount”, “parameters”: { “location”: “s3://{BUCKET}/{PATH}/”, “files”: [ “video.mp4”, “audio_en.mp4”, “audio_es.mp4”, … ] } }

Parameter	Properties	Default	Type	Choice	Description
location	required				S3 path which is mounted
files	required [list value]				Files that become available from the S3 location (list of strings). Wildcards not supported

storage:put¶

File Upload tool for AWS-S3 or Azure Blob Storage

The tool understands the wildcard expansion characters: - \* - matches everything - ? - matches any single character

Example with S3:

{
    "tool": "storage:put",
    "parameters": {
        "location": "s3://{AWS_ACCESS_KEY}:{AWS_SECRET_KEY}@{BUCKET}/{PATH}",
        "files": [
            "video.mp4",
            "audio_en.mp4",
            "audio_es.mp4",
            "*.vtt", ...
        ],
        "s3_endpoint_url": "{ENDPOINT_URL}"
    }
}

Example using Azure account name and access key:

    {
      "tool": "storage:put",
      "parameters": {
        "az_acc_name": "account_name",
        "az_acc_key": "account_access_key",
        "az_container": "container",
        "overwrite": False,
        "files": [
            "hls/*"
        ]
      }
    }

Example using Azure account name and SAS token:

    {
      "tool": "storage:put",
      "parameters": {
        "az_acc_name": "account_name",
        "az_sas_token": "sas_token",
        "az_container": "container",
        "overwrite": False,
        "files": [
            "hls/*"
        ]
      }
    }

Parameter	Properties	Default	Type	Description
files	required [list value]			Filenames to be downloaded before processing (list of strings)
overwrite	optional	false	bool	If set to true then existing files will be overwritten without any further precaution.
location	optional			A base URL from which files are downloaded (string)
base_dir	optional		str	A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

storage:check¶

Checks if files in a remote location exist. This step either

fails if these files exist (mode: 'FAIL_ON_PRESENCE'),
fast-forwards the job to success without executing any further step (mode: 'COMPLETE_ON_PRESENCE')
fails if the specified files do not exist (mode: FAIL_ON_ABSENCE')

Example:

{
  "tool": "storage:check",
  "parameters": {
    "location": "s3://{aws-credentials}@com.castlabs.vtk.test/contentpath",
    "files": [
      "tos_3min_video_960x540_1mbps.mp4"
    ],
    "mode": "COMPLETE_ON_PRESENCE"
  }
},

Parameter	Properties	Type	Choice	Description
files	required [list value]			Filenames to be downloaded before processing (list of strings)
mode	optional		‘FAIL_ON_PRESENCE’, ‘COMPLETE_ON_PRESENCE’, ‘FAIL_ON_ABSENCE’	FAIL_ON_PRESENCE: If the files are already present the job will fail.
COMPLETE_ON_PRESENCE: If the files are already present the job will succeed without executing any subsequent steps.
FAIL_ON_ABSENCE: If the files are not there (e.g. if subsequent steps require those files) the job will fail.
location	optional			A base URL from which files are downloaded (string)
base_dir	optional	str		A local directory to be used as target or source (default = ‘’)
s3_endpoint_url	optional			The custom endpoint URL of a s3 bucket
http_credential_source	optional			HTTP Credential Source
az_acc_name	optional			Azure Storage Account Name
az_container	optional			Azure Storage Container
az_acc_key	optional			Azure Account Access Key
az_sas_token	optional			Azure Shared Access Token (?sv=..)
gc_adc_string	optional			Google Cloud ADC base64 (with padding)
role_arn	optional			The role to be assumed for executing the tool

Previous topic: Transcoding

Next topic: Packaging