Postprocessing Build Raster layer AWS Step Function

This step function postprocessing-build-raster-step-function is used to orchestrate building of a custom raster layer. This process is a lot more involving than the preset postprocessing step functions. The step function attempt to grasp all information from incoming data layers, but cannot make all the assumptions based on that. Therefore the admin is required to fill out more information. Use this function to upload any type of GeoTIFF from any location in the data processed bucket. DynamoDB controls the status of this layer, by adding it to the layer item and, if present, reading its status to skip to certain process stages.

Step Functions

The step functions that are used within this step function:

• A PostprocessingRasterOptimization step function that generates the COG dataset for Geoserver.

Lambdas

The lambdas orchestrated by the Step function are the following:

• A PostprocessingBuildPropertiesFunction function to build properties files necessary for Geoserver coverage configuration.

./datastore.properties  # contains database schema information
./indexer.properties  # contains coverage information (EPSG)
./timeregex.properties  # contains regex string to determine date
./footprint.properties  # contains trigger for COG hosting

• A PostprocessingGetInstanceRequirements function to estimate EC2 resource requirements.
• A PostprocessingPublishRaster function to generate the publication information for Serverless infrastructure.
• A PostprocessingRunSsmCommandFunction function to run particular commands against the EC2 running the Geoserver. In this case the lambda runs the upload command moving data from S3 to EC2.
• A PostprocessingCheckSsmCommandFunction function polls the Geoserver data directory for the presence of all data. Basically the checksum.
• A PostprocessingBuildRasterStats to calculate the quantiles used for YSLD styling.
• A PostprocessingBuildRaster function call the Geoserver REST API to fullfill the ImageMosaic plugin generation. After this step the imagery should be visible online through Geoserver UI and front end App.
• A PostprocessingBuildCache function calls the GeoWebCache API to start creating TileLayers and pre-seeding.

Invocation description

The following keys are required:

job_id: str, job id reference
layername: str, input layer name
prefix: str, path to layer data
calculate_stats: bool, switch to determine whether to let Lambda calculate the statistics. These statistics are: breaks, range_type, nodata. Colours will be returned all grey if colours is null
data_type: str, choice between imagery and data. Use imagery when you are uploading a 3-stack GeoTIFF (like RGB). Look at raster optimization docs for more information.
colours: list, an input array of string containing the colours, needs to be of similar length as breaks, default is all grey.
breaks: list, an input array of where you like to break up the values.
sample_size: int, number of iterations the statistics will you to determine breaks.
unit: str, list, display ISO unit in YSLD, when an array you can add a label per break
nodata: int or float, input number used to set the transparant value in YSLD
range_type: str, choice between ramp (continuous) and values (discrete), default is ramp
mode: str, choice between quantiles and ... "WIP"

Example

{
  "job_id": "20220520151302-1327-0ad42aeec1014698b5efc953b83b5825",
  "layername": "some_custom_layer2",
  "prefix": "some/prefix/",
  "calculate_stats": false,
  "data_type": "data",
  "colours": [
    "#2c7bb6",
    "#abd9e9",
    "#ffffbf",
    "#fdae61",
    "#d7191c"
  ],
  "breaks": [
    0,
    5,
    20,
    50,
    100
  ],
  "sample_size": 1,
  "unit": "m",
  "nodata": 0,
  "range_type": "ramp",
  "mode": "quantiles"
}

{
  "job_id": "20220520151302-1327-0ad42aeec1014698b5efc953b83b5825",
  "layername": "some_custom_layer",
  "prefix": "some/prefix/",
  "calculate_stats": true,
  "data_type": "data",
  "colours": [
    "#2c7bb6",
    "#abd9e9",
    "#ffffbf",
    "#fdae61",
    "#d7191c"
  ],
  "sample_size": 1,
  "unit": "m",
  "mode": "quantiles"
}

{
  "job_id": "20230421140759-952-2d70952abf634155a9b4af2f37ce627f",
  "layername": "problemareas",
  "prefix": "02_data/01_clients/059_NLD_eaglesensing/08/01_data/06_plantation/07_problemareas/20160101/",
  "calculate_stats": false,
  "data_type": "data",
  "colours": [
    "#b6587d",
    "#3ac830",
    "#e9e20a",
    "#c25304"
  ],
  "breaks": [
    1,
    2,
    3,
    4
  ],
  "sample_size": 1,
  "unit": [
    "Other",
    "Healthy Pineapple",
    "Unhealthy Pineapple",
    "Plantation Gap"
  ],
  "nodata": 0,
  "range_type": "values",
  "mode": "quantiles"
}