Skip to main content

index

Objects​

BaseInput​

@dataclass
class BaseInput()

Input for running the main lambda handler.

Attributes:

  • prefix str - object path in which to find the files.
  • batch int, optional - size of a single batch. This will generate a List[List] with every outer List of the same length as batch. The last list in the List will contain the leftovers and can be of a different length.
  • bucket str, optional - the bucket to search in. Defaults to SOURCE_BUCKET.
  • exclude List[str], optional - an array of string patterns to exclude from the files in prefix.
  • s3_payload bool, optional - switch to send the data as S3 payloads to the payload bucket. Defaults to False.

Functions​

lambda_handler​

@tracer.capture_lambda_handler
@logger.inject_lambda_context()
def lambda_handler(event: dict, context: dict) -> list

Retrieves contents of s3 bucket. Used in a step function to prep for a batcher.

Arguments:

  • event dict - should fit the BaseInput.
  • context dict - lambda context object.

Raises:

  • e - raised when listing fails for some reason.

Returns:

  • list - a list of strings | objects | list as an iterable. With s3_payload = True this will be a list of payload jsons.

Examples:

{
    "prefix": "some/object/path/",
    "batch": 10,
    "bucket": "some-bucket",
    "exclude": ["*.tif.aux", "Thumbs"],
    "s3_payload": true
}