Postprocessing Build DEM layers AWS Step Function

This step function postprocessing-build-dem-step-function is used to orchestrate the upload and configuration of Digital Surface Model (DSM) and Digital Terrain Model (DTM) deliverables. The DSM and DTM are both produced in the core processing by metashape.py. The step function runs a parallel process for either DEM deliverable. The default upload behaviour of this step function is different than the others in that by default it does not upload unless upload or legacy is defined.

Step functions

The step functions that are used within this step function:

• A PostprocessingRasterOptimization step function that generates the COG dataset.

Lambdas

The lambdas orchestrated by the Step function are the following:

• A UtilityGetPrefixFunction to generate the prefix used to find the input files and where data will be stored in S3.
• A PostprocessingBuildPropertiesFunction function to build properties files necessary for Geoserver coverage configuration.

./datastore.properties  # contains database schema information
./indexer.properties  # contains coverage information (EPSG)
./timeregex.properties  # contains regex string to determine date
./footprint.properties  # contains trigger for COG hosting

• A PostprocessingGetInstanceRequirements function to estimate EC2 resource requirements.
• A PostprocessingRunSsmCommandFunction function to run particular commands against the EC2 running the Geoserver. In this case the lambda runs the upload command moving data from S3 to EC2.
• A PostprocessingCheckSsmCommandFunction function polls the Geoserver data directory for the presence of all data. Basically the checksum.
• A PostprocessingPublishRaster function to generate the publication information for Serverless infrastructure.
• A PostprocessingBuildRasterStats to calculate the quantiles used for YSLD styling.
• A PostprocessingBuildRaster function call the Geoserver REST API to fullfill the ImageMosaic plugin generation. After this step the imagery should be visible online through Geoserver UI and front end App.
• A PostprocessingBuildCache function calls the GeoWebCache API to start creating TileLayers and pre-seeding.

Invocation description

The following keys are required:

job_id: str, job id reference
client_tag: str, client tag to determine costing

The following keys are required:

legacy: bool, switch to prepare data for Geoserver of Serverless infrastructure, defaults to false.
upload: bool, switch to allow step function to upload to either infrastructure, defaults to true.
options: object, define all options or it will default to default options.
  colours: array, list of html colour codes. Defaults to Geoserver colour code set.
  nodata: number, overwriting the nodata value. Defaults to tiff nodata
  si_unit: str, measure unit used in the legend. Defaults to "m" for meters.

Example

{
  "job_id": "20220520151302-1327-0ad42aeec1014698b5efc953b83b5825",
  "client_tag": "Some Client",
  "legacy": false,
  "upload": true,
  "options": {
    "colours": [
      "#2c7bb6",
      "#abd9e9",
      "#ffffbf",
      "#fdae61",
      "#d7191c"
    ],
    "nodata": -32767,
    "si_unit": "m"
  }
}