Utilities

Utilities¶

utils¶

utils:tar¶

Tar file(s)

Example:

{
  "tool": "utils:tar",
  "parameters": {
    "archive": "test.tar.bz2",
    "compress": true,
    "bzip2": true,
    "verbose": true,
    "files": "subtitles/*"
  }
},

Parameter	Properties	Default	Type
archive	required
verbose	optional	false	bool
compress	optional	false	bool
extract	optional	false	bool
gzip	optional	false	bool
bzip2	optional	false	bool
xz	optional	false	bool
lzma	optional	false	bool
workdir	optional
files	optional

utils:rename¶

Rename file(s)

The pattern is a regex expression.

Files in the working folder and its sub-folders are looped and the pattern is evaluated against each name (a relative path to the working folder).

An important thing to note to avoid unexpected matches to the pattern: The matching of the pattern is not enforced to be from the beginning of the string, but the filename is scanned, looking for the first location where the pattern produces a match.

Taking the following working folder structure as an example. video.mp4 audio.mp4 out/video.mp4

Specifying only video.mp4 as the pattern would match both, the video.mp4 in the root of the folder and also in the out sub-folder. To match only the files in the root, use regex expressions at your disposal to make the regex pattern more specific.

For this specific example, a regex of ^video.mp4 would only match the file at the root of the folder, since it specifies the name must occur at the very beginning of the string.

Example:

{
    "tool": "utils:rename",
    "parameters": {
        "pattern": '-(?P<lang>\\w+).aac$',
        "to": "out/1234567891234567890123456789012_{lang}.aac"
    }
}

Above will find all the AAC tracks and use the language extracted from each to form the output filename. For the input files:

test-en.aac
test-es.aac

Following files will be available after tool execution:

out/123456789123456789012345678901295en.aac
out/123456789123456789012345678901295es.aac

Parameter	Properties	Default	Type	Choice	Description
pattern	optional
to	optional

utils:env¶

Define local environment variables

Maps variables in key pairs once during task startup. Useful for easier job templating to streamline input and output parameters throughout task layouts.

Example:

{
  "tool": "utils:env",
  "parameters": {
    "videoIn": "string",
    "source-base-path": "s3.castlabs.com",
    "asset_id": "DRMtoday_Asset_ID_0bacbfd4-c173-4d5c-bcd4-d677cf4b4c6e"
  }
},
...
{
    "tool": "ffmpeg:cmd",
    "parameters": {
            "-i": "{videoIn}",
            ...
        }
}

Parameter	Properties	Default	Type	Choice	Description

utils:modjson¶

Modify an S3 based JSON file object with jq

Example:

 {
  "tool": "utils:modjson",
  "parameters": {
    "location": "s3://{target-bucket}/{json-base-bath}/{asset_id}-{job_id}.json",
    "jq": ".assetId = "{asset_id}" | .jobId = "{job_id}" | .status = ["Started"]"
  }
},

This can update a JSON object with:

"status": [
    "Started"
],

Parameter	Properties	Default	Type	Description
default	required	{}	str
jq	required	.	str
role_arn	optional			The role to be assumed for executing the tool

utils:http_request¶

Do an HTTP GET/POST request to the specified URL

url: REQUIRED, URL to use
method: OPTIONAL, HTTP method to use. Valid values: GET, POST
headers: OPTIONAL, HTTP headers to send
body: OPTIONAL, base64-encoded body to POST when using HTTP POST. Ignored if HTTP GET is used.
params: OPTIONAL, URI params
ignore_errors: OPTIONAL, ignore errors during the HTTP call. Default: false
save_as: OPTIONAL, saves the response body to a local file. Default: None

Example:

{
  "tool": "utils:http_request",
  "parameters": {
    "url": "https://www.google.com",
    "method": "POST",
    "headers": {"Authorization": "my-auth-token"},
    "body": "aGVsbG8sIHdvcmxkIQ==",
    "params": {"id": "1"},
    "ignore_errors": true,
    "save_as": "response.json"
  }
},

Parameter	Properties	Default	Type	Choice
url	required
method	optional	POST	str	‘GET’, ‘POST’
body	optional		str
headers	optional
params	optional
ignore_errors	optional	false	bool
save_as	optional

utils:file_checksum¶

Uses the algorithm specified in the `algorithm` parameter to generate a checksum for the file provided in the `input_file` string.
Exposes it in the environment as the specified name.

If output file is provided, the checksum will be written to the file.

Supported algorithms: md5, sha1, sha256, sha512

Example:

{
  "tool": "utils:file_checksum",
  "parameters": {
    "algorithm": "md5",
    "input_file": "file1.txt",
    "env_name": "files_checksum",
    "output_file": "file1.txt.md5"
  }
},

Parameter	Properties	Default	Type	Choice
input_file	required
algorithm	optional	md5	str	‘md5’, ‘sha1’, ‘sha256’, ‘sha512’
env_name	optional
output_file	optional

utils:md5sum¶

Generates an MD5 sum from the provided string and exposes it in the environment as the specified name.
If `uuid_format` is `false`, the value will be a hex string (bcab7a3c076642c28c0258fe95443c7e)
If `uuid_format` is `true`, the value will be a hex string in a UUID format (bcab7a3c-0766-42c2-8c02-58fe95443c7e)

Example:

{
  "tool": "utils:md5sum",
  "parameters": {
    "input_string": "string-to-hash",
    "env_name": "my_key",
    "uuid_format": true
  }
},

Parameter	Properties	Default	Type
input_string	optional
env_name	optional
uuid_format	optional	false	bool

utils:write_text¶

Write text to file

Write text to file. Input is a base64-encoded value of text to be written to the file.

Example:

{
    "parameters": {
        "base64value": "U29tZSBzYW1wbGUgdGV4dCBoZXJlLgpXaXRoIG5ldyBsaW5lcyEKClRoZSBlbmQu",
        "outputfile": "out/sample.txt"
    },
    "tool": "utils:write_text"
}

Parameter	Properties	Default	Type	Choice	Description
outputfile	required
base64value	optional
textvalue	optional

utils:mkdir¶

Create a directory in the Job`s default path during runtime

Example:

{
  "tool": "utils:mkdir",
  "parameters": {
    "dir": "subtitles"
  }
},

Parameter	Properties	Default	Type	Choice	Description
dir	required				The directory to be created. Path is relative to current job’s execution directory

utils:mkfifo¶

Create a named pipe in the Job`s default path during runtime

Example:

{
  "tool": "utils:mkfifo",
  "parameters": {
    "name": "namedpipe"
  }
},

Parameter	Properties	Default	Type	Choice	Description
pipe_name	required				The named pipe to be created. Path is relative to current job’s execution directory

utils:zip¶

Zip file(s)

Example:

{
    "parameters": {
        "input_patterns": [
          "video/a.txt",
          "video/b.txt"
        ],
        "outputfile": "out.zip"
    },
    "tool": "utils:zip"
}

Parameter	Properties	Default	Type	Choice	Description
input_patterns	required [list value]
outputfile	required

utils:notify_email¶

Custom Notification Email, sending JSON object as body

Example:

{
    "parameters": {
        "recipients": ["info@castlabs.com", "info@above.aero"]
        "infojson": {
            "key1": "value1",
            "key2": "value2",
        },
    },
    "tool": "utils:notify_email"
}

Parameter	Properties	Default	Type	Choice	Description
recipients	required [list value]
infojson	required		dict

utils:copy¶

Copy file(s) The pattern is a regex expression.

Example:

{
    "tool": "utils:copy",
    "parameters": {
        "pattern": "purchaseorder/processdata/(?P<webvttfiles>[\w\-]+).vtt$",
        "to": "delivery/{webvttfiles}.vtt"
    }
}

Above will find all the WebVTT files in purchaseorder/processdata/ and will copy them to the delivery folder. If the delivery folder does not exist it will be created.

Parameter	Properties	Default	Type	Choice	Description
pattern	optional
to	optional

utils:write_bytes¶

Write Bytes to file

Write Bytes to files, can be used to create key files for Unified Packager tool for example

Example:

{
    "parameters": {
        "hexvalue": "aabbccddeeff0011223344556677889900",
        "outputfile": "out/iv.key"
    },
    "tool": "utils:write_bytes"
}

Parameter	Properties	Default	Type	Choice	Description
hexvalue	required
outputfile	required

utils:derive-key-playready¶

Derive a key from a seed and key id, using the PlayReady derivation method.

The key is exposed as two environment variables with a named prefix: {name_prefix}_hex {name_prefix}_b64

Example:

{
    "parameters": {
        "seed_hex": "00112233445566778899aabbccddeeff00112233445566778899aabbccdd",
        "key_id": "089b3d59-080b-4b99-a633-a69bdec0eef5",
        "name_prefix": "my_key"
    },
    "tool": "utils:derive-key-playready"
}

Parameter	Properties	Default	Type	Choice	Description
seed_hex	required
key_id	required
name_prefix	required

utils:createlink¶

Generate a AWS-S3 HTTPS link for specified files and print it to log.

Parameter	Properties	Default	Type	Description
location	required			S3 base URL from which files are downloaded (string)
files	required [list value]			Filenames to be included (list of strings)
expires_in	optional	3600	int	Link expires in N seconds
role_arn	optional			The role to be assumed for executing the tool

utils:echo¶

Echo a string or environment variable during runtime

Example

{
    "tool": "utils:echo",
    "parameters": {
        "text": "audio_id_hex: {audio_id_hex}"
    }
},

Parameter	Properties	Default	Type	Choice	Description
text	required

utils:random-uuid¶

Generates a random UUID and exposes it in the environment with the key as given via name parameter.

Example:

{
  "tool": "utils:random-uuid",
  "parameters": {
    "name": "uuid-1",
    "lower_case": true
  }
},

Parameter	Properties	Default	Type	Choice	Description
name	optional	uuid	str
lower_case	optional	false	bool

utils:env_jq¶

Define local environment variables from JSON Uses jq expressions to get a value from a local json and sets it in the env var.

Example:

{
  "tool": "utils:env_jq",
  "parameters": {
    "input_json": "api_response.json",
    "jq": ".token",
    "var_name": "MY_API_AUTH_TOKEN"
  }
},
...
{
  "tool": "utils:echo",
  "parameters": {
    "text": "{MY_API_AUTH_TOKEN}"
  }
}

Parameter	Properties	Default	Type	Choice	Description
input_json	required
jq	required
var_name	required

Previous topic: Packaging

Next topic: DRMtoday