Postprocessing Build Contourlines layer AWS Step Function

This step function postprocessing-build-contourlines-step-function is used to orchestrate the building of a contourlines and uploading the results to a specified Geoserver. The function requires a succesfull DTM, run postprocessing-build-dem-step-function if that is not the case. Processing the contourlines is expensive since DEM data is quite large and we need to perform some blurring to smooth out the high resolution DEMs. Therefore this step function leverages an EC2 to run the extensive computations. Contourlines are isolines on the edges of elevations differences. We run with an interval of 1m but this can be adjusted in the EC2 python code.

Step Functions

The step functions that are used within this step function:

• A PostprocessingStartECS step function that starts an EC2 within the ECS cluster based on set parameters.

Lambdas

The lambdas orchestrated by the Step function are the following:

• A UtilityGetPrefixFunction to generate the prefix used to find the input files and where data will be stored in S3.
• A UtilityTaskControllerForTiler function to create the payload jsons for parallel processing inside EC2s.
• A PostprocessingGetInstanceRequirements function to estimate EC2 resource requirements.
• A PostprocessingBuildVectorFunction function to process, configure and upload vector based datasets.
• A PostprocessingPublishVectorFunction function to publish the vector data in the AuroraDB.
• A UtilityNotificationClickupFunction function to handle ClickUp notifications.

Warning Do not use this lambda against a raw contourlines dataset, in case of large feature counts (>100000) the lambda could timeout. Make sure the data is already inserted into Geoserver DB and set the layer status in DynamoDB to unpublished.

Invocation description

The following keys are required:

job_id: str, job id reference
client_tag: str, client tag to determine costing

The following keys are optional:

legacy: bool, switch to prepare data for Geoserver of Serverless infrastructure, defaults to false.
upload: bool, switch to allow step function to upload to either infrastructure, defaults to true
options: dict, a object of options which are used in contourlines postprocessing
  blur: bool, switch on blurring, defaults to true
  blur_stat: str, choose a blurring algorithm, defaults to gaussian
  compression: str, choose a compression algorithm, defaults to DEFLATE
  factor: int, blurring factor, defaults to 500
  interval: int, interval in meters used in contourlines
  minimal_length: int, clean out lines with total length or less than of minimal_length, defaults to 50
  nodata: int, nodata value of output raster
  overwrite: bool, switch to overwrite existing shapefiles, defaults to false
  smooth: bool, switch on vector smoothing, defaults to true
  smooth_iteration: int, number of iterations used in vector smoothing, defaults to 2

Example

{
  "job_id": "20220520151302-1327-0ad42aeec1014698b5efc953b83b5825",
  "client_tag": "Some Client",
  "legacy": false,
  "upload": true,
  "options": {
    "blur": true,
    "blur_stat": "gaussian",
    "compression": "DEFLATE",
    "factor": 500,
    "interval": 1,
    "minimal_length": 50,
    "nodata": 0,
    "overwrite": false,
    "smooth": true,
    "smooth_iterations": 2
  }
}