Jobs

Jobs¶

A Job is an instruction set that can be submitted to Video Toolkit. Usually such jobs consist of downloading files for processing, processing and uploading of the results. Each Job holds a collection of tasks which are one or more tool definitions.

Job definition¶

The structure of a job is simply a list of tasks:

{
    "tasks": [{
        "tool": "<tool1>",
        "parameters": [ ... ]
    },
    {
        "tool": "<tool2>",
        "parameters": [ ... ]
    },
    {
        ...
    }]
}

AWS Region selection for jobs¶

Define the AWS region the job shall be started in (usually a resource pool was created upfront for your organization_id and targeted AWS regions)

{
    "region": "aws:eu-central-1",
    "tags": [
        "Tag ID",
        "Tag ID 2",
        ...
    ],
    "tasks": [{
        "tool": "<tool1>",
        "parameters": [ ... ]
    }]
}

Organizations¶

If a user is part of multiple organizations simultaneously it is required to specify the organization_id or organization name to assign the job to a specific organization:

{
    "organization_id": organization id
        --- alternatively
    "organization": organization name
    "tasks": [
      ...
    ]
}

Naming/Tagging Jobs¶

Jobs don’t have names or any other human-readable identifier by default. That can make it very difficult to keep track of jobs especially if you try to get oversight by using the UI. Jobs can have tags to help with that. We suggest to use only a few tags per job and made good experience with two tags: One for the category (e.g. 2017-03-Encodings) and one for the asset ID (e.g. Metropolis_1927) as it is referenced in your media asset management system.

{
    "tasks": [{
        "tool": "<tool1>",
        "parameters": [ ... ]
    },
    {
        ...
    }],
    "tags": [
      "2017-03-Encodings",
      "Metropolis_1927"
    ]
}

Jobs API¶

Submit Job¶

Submit a job via

API¶

HTTPS POST to the ‘/api/jobs/’ endpoint, with the job as JSON data. Required Headers:

'content-type: application/json'

'Authorization: api-key <your API key>'

UI¶

Simply drag&drop the JSON file after logging in.

Get job¶

If you know the job ID, you can retrieve information about that specific job by executing an HTTP GET to the /api/jobs/JOB_ID/ endpoint.

curl -v -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H Content-Type:application/json https://vtks.castlabs.com/api/jobs/abcDEFGHIjk/
> GET /api/jobs/abcDEFGHIjk/ HTTP/1.1
> Host: vtks.castlabs.com
> User-Agent: curl/7.64.1
> Accept: */*
> Authorization: Ticket ***
> content-type:application/json
>
< HTTP/1.1 200 OK
< Allow: GET, OPTIONS, HEAD, POST
< Content-Type: application/json
{
"id":"abcDEFGHIjk",
"url":"https://vtks.castlabs.com/api/jobs/abcDEFGHIjk/",
"tasks":[
  {
    "id":12345,
    "tool":"storage:get",
    "parameters":{...},
    "status":2,
    "status_text":"Completed"
  }
],
"notify":null,
"organization":"some_org",
"organization_id":1,
"user":"some_user",
"status":3,
"status_text":"Failed",
"created":"2018-04-26T17:27:43Z",
"modified":"2018-04-26T17:30:41Z",
"project_offer":null,
"purchase_order":null,
"purchase_order_id":null,
"job_bundle":null,
"job_bundle_id":null,
"tags":[],
"region":"aws:us-east-1",
"role_arn":null
}

Get job metrics¶

Executing an HTTP GET to the /api/jobs/JOB_ID/metrics/ endpoint returns basic metrics as JSON for a given job_id. The job in question must have the status COMPLETED, or FAILED.

{"id": "F8QDEYg1qos", "cus": 50.26, "video_minutes_in": 60, "video_minutes_out": 240, "gb_in": 50.055132323000000004, "gb_out": 8.104970481}

Restart/Resubmit job¶

Restarting a job will reset the tasks, and start over under the same job_id, by issuing a POST to this endpoint: /api/jobs/{jobId}/restart/.

Resubmitting a job will reset the tasks, but will start the same job under a new job_id: /api/jobs/{jobId}/resubmit/

curl -v -X POST -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' https://vtks.castlabs.com/api/jobs/2w9geAylAqq/restart/
*   Trying 3.94.1.190:443...
* TCP_NODELAY set
* Connected to vtks.castlabs.com (3.94.1.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=vtks.castlabs.com
*  start date: Jan 14 00:00:00 2022 GMT
*  expire date: Feb 12 23:59:59 2023 GMT
*  subjectAltName: host "vtks.castlabs.com" matched cert's "vtks.castlabs.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> POST /api/jobs/2w9geAylAqq/restart/ HTTP/1.1
> Host: vtks.castlabs.com
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Allow: OPTIONS, POST
< Content-Type: text/html; charset=utf-8
< Date: Thu, 24 Feb 2022 12:40:40 GMT
< Referrer-Policy: same-origin
< Server: nginx
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Cookie
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host vtks.castlabs.com left intact

Getting task’s log output¶

To get a log output of a given task, append /logs/. For the above example the full URL becomes https://vtks.castlabs.com/api/jobs/abcDEFGHIjk/tasks/12345/logs/

curl -v -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H Content-Type:application/json https://vtks.castlabs.com/api/jobs/abcDEFGHIjk/tasks/12345/logs/
> GET /api/jobs/abcDEFGHIjk/tasks/12345/logs/ HTTP/1.1
> Host: vtks.castlabs.com
> User-Agent: curl/7.64.1
> Accept: */*
> Authorization: Ticket ***
> content-type:application/json
>
< HTTP/1.1 200 OK
< Allow: GET, OPTIONS, HEAD, POST
< Content-Type: application/json
{
  "logs":
  [
    {
      "timestamp": 1607374149,
      "message": "Running tool..."
    },
    ...
  ]
}

Note: this call is throttled to 20 calls/minute and 1000/day.

Job Bundles (Purchase Orders, Bundles)¶

Jobs can be bundled to so called Job Bundles for better tagging and grouping of multiple jobs. Adding a job to a job bundle by adding the bundle id to the job definition:

{
    "job_bundle_id": job_bundle_id
        --- alternatively
    "job_bundle": job_bundle_name
    "tasks": [{
      ...
    }]
}

Job Bundle Creation¶

Note: Users creating Job Bundles must have the ‘CONFIG’ role.

Create a Job Bundle by posting the following JSON job description to $APP_URL/job_bundle/ for example:

Content of jb.json:

{
    "name" : bundle_name,
    "organization_name": your_organization_name
}

cURL command line:

curl -v --data @jb.json -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H content-type:application/json  $APP_URL/job_bundle/

Job Bundle Listing¶

Bundles of jobs can be listed, for example with cURL:

curl -v -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H content-type:application/json  $APP_URL/job_bundle/

Search for names of bundles when listing:

curl -v -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H content-type:application/json  $APP_URL/job_bundle/?search={bundle_name}

Search by name:

curl -v -H 'Authorization: api-key AbabBbbAB.XyYYXYxxYxXYxYXYXYXxYxYXYXYxY' -H content-type:application/json  $APP_URL/job_bundle/?name={bundle_name}

Previous topic: Example Job Execution

Next topic: Tasks