Utility Tiler AWS Step Function

This step function utility-tiler-step-function is used to run a tiling process by distributing tile tasks across lambdas and step functions. The step function attempts to save itself from reaching DataLimits. Batchsizes and concurrency can be controlled from the step function through the invocation and step function template.

Lambdas (dev)

The lambdas orchestrated by the Step function are the following:

• A UtilityTaskHelperTiler function reads the task json content from s3.
• A UtilityTaskControllerForTiler function that construct the Task items and determines whether to use a batchsize or unlimited power (within AWS limits).
• A UtilityTilerFunction function that performs tiling based on a search window defined in the task.

Step Function (dev)

In case the step function expects too many tasks it will fall back on a recovery step function branch to redirect its tasks to separate step functions.

• utility-tiler-auxiliary-step-function

Invocation description

The following keys are required:

prefix: str, path to source data
bucket: str, name of s3 bucket
output_directory: str, path to output s3 key. Must be a directory since we're outputting tiles
tilesize: int, dimension of tiles. Tiles are always squared, so length == width, hence the use of a single integer.
options: str(dict), escaped dictionary object as string. Can be left empty, would not recommend.
basename: str, prefix for tile filename (e.g. 20210101_RGB will result in 20210101_RGB-0-0.tiff). Using basename: "" will create a basename based on the prefix used.

Example

{
  "prefix": "../20210101/20210101_RGB.tif",
  "bucket": "nosi-data-processed-dev",
  "output_directory": "02_data/01_clients/<client_folder>/01/01_data/01_aerial/01_rgb/20210101/",
  "tilesize": 5000,
  "options": "{\"compression\": \"DEFLATE\", \"tiled\": \"True\"}",
  "basename": "20210101_RGB"
}