Add metadata at item level for Disasters (event name and monty-stac schema)

[kyle-lesinger]

Summary:

In order to effectively filter through data within veda-ui, additional property information is needed at the item level. This will assist with restricting data within the E&A to only data that is relevant for a specific event.

The event name (or activation name) is specific to each activation event and is unique. The goal of these changes is to allow for easier filtering when a specific component is clicked (TBD where this component will be).

Changes

For event name only

• Added new function _parse_filename which extracts disasters specific activation name and info from filename.
• Add a new flag to discovery_items to allow for backwards compatibility "disasters:extract_event_name": true,. If the key/value pair is absent, airflow will run as normal. If the key/value pair is present and is true, then the filename will be extracted
• Adds the new event:name to the properties section for each item.

For monty stac extension

• Add a new flag to discovery_items to allow for backwards compatibility "disasters:monty": true,. If the key/value pair is absent, airflow will run as normal. If the key/value pair is present and is true, then the monty stac spec info will be added
• scrape information from file name and add the following properties (see example below)

"monty:corr_id": "20230208-TUR-EQ-1-GCDB",
"monty:hazard_codes": ["EQ","GH0101"],
"monty:country_codes": ["TUR"]

Can test locally with the following config

{
    "collection": "alos2-all-vars-daily-TEST-addMetadata",
    "dashboard:is_periodic": false,
    "dashboard:time_interval": "P1D",
    "description": "The Advanced Land Observing Satellite-2 (ALOS-2, DAICHI-2) is a follow-on mission from the ALOS Daichi. ALOS has contributed to cartography, regional observation, disaster monitoring, and resource surveys, since its launch in 2006. ALOS-2 will succeed this mission with enhanced capabilities. JAXA conducted research and development activities for ALOS-2 to improve wide and high-resolution observation technologies developed for ALOS in order to further fulfill social needs. These social needs include: 1) Disaster monitoring of damage areas, both in considerable detail, and when these areas may be large 2) Continuous updating of data archives related to national land and infrastructure information 3) Effective monitoring of cultivated areas 4) Global monitoring of tropical rain forests to identify carbon sinks.",
    "title": "ALOS2 All Variables daily timestep",
    "data_type": "cog",
    "spatial_extent": {
        "xmin": -125,
        "ymin": -24,
        "xmax": -66,
        "ymax": 49
    },
    "temporal_extent": {
        "startdate": "2020-09-14T00:00:00Z",
        "enddate": null
    },
    "item_assets": {
        "dpm": {
            "description": "Damage Proxy Map derived from ALOS-2 SAR data showing changes in radar backscatter that indicate potential structural damage or building destruction from disaster events.",
            "roles": [
                "data",
                "layer"
            ],
            "title": "Damage Proxy Map",
            "type": "image/tiff; application=geotiff; profile=cloud-optimized"
        },
        "dpm-cvd": {
            "description": "Damage Proxy Map with colorblind-friendly color scheme, derived from ALOS-2 SAR data showing changes in radar backscatter indicating potential structural damage or building destruction.",
            "roles": [
                "data",
                "layer"
            ],
            "title": "Damage Proxy Map (CVD)",
            "type": "image/tiff; application=geotiff; profile=cloud-optimized"
        }
    },
    "license": "CC0-1.0",
    "links": [],
    "providers": [
        {
            "name": "NASA"
        }
    ],
    "sample_files": [
        ""
    ],
    "renders": {
        "dashboard": {
            "assets": [
                "dpm"
            ],
            "bidx": [1,2,3,4],
            "rescale": [
                [
                    0,
                    255
                ]
            ]
        },
        "dpm": {
            "assets": [
                "dpm"
            ],
            "bidx": [1,2,3,4],
            "rescale": [
                [
                    0,
                    255
                ]
            ]
        },
        "dpm-cvd": {
            "assets": [
                "dpm-cvd"
            ],
            "bidx": [1,2,3,4],
            "rescale": [
                [
                    0,
                    255
                ]
            ]
        }
    },
    "stac_extensions": [
        "https://stac-extensions.github.io/render/v1.0.0/schema.json",
        "https://stac-extensions.github.io/item-assets/v1.0.0/schema.json",
        "https://ifrcgo.org/monty-stac-extension/v1.1.0/schema.json"
    ],
    "stac_version": "1.0.0",
    "discovery_items": [
        {
            "collection": "alos2-all-vars-daily-TEST-addMetadata",
            "disasters:extract_event_name": true,
            "disasters:monty": true,
            "bucket": "nasa-disasters",
            "prefix": "drcs_activations_new/ALOS2/",
            "cogify": false,
            "discovery": "s3",
            "dry_run": false,
            "filename_regex": ".*_day.tif$",
            "assets": {
                "dpm": {
                    "description": "Damage Proxy Map derived from ALOS-2 SAR data showing changes in radar backscatter that indicate potential structural damage or building destruction from disaster events.",
                    "regex": "^(?!.*(_cvd_)).*DPM.*\\.tif$",
                    "title": "Damage Proxy Map"
                },
                "dpm-cvd": {
                    "description": "Damage Proxy Map with colorblind-friendly color scheme, derived from ALOS-2 SAR data showing changes in radar backscatter indicating potential structural damage or building destruction.",
                    "regex": ".*DPM.*cvd.*\\.tif$",
                    "title": "Damage Proxy Map (CVD)"
                }
            },
            "id_regex": ".*(\\d{4}-\\d{2}-\\d{2}_day)\\.tif$",
            "id_template": "alos2-{}"
        }
    ]
}

PR Checklist

• Unit tests
• Ad-hoc testing - Deploy changes and test manually
• Integration tests
• Infrastructure changes - test using terraform validate and terraform plan

[kyle-lesinger]

@smohiudd here is the first part of the new updates for Disasters. If we fully incorporate monty stac extension, I'll have to scrape more properties from the filenames. But that is TBD.