IDS-DRR IMD Source Pipeline

[d-saikrishna]

@tauseefsshah
We have to write script(s):

1. To extract Rainfall data from IMD (Indian Meteorological Department)
2. Aggregate (mean) rainfall for each revenue circle in Assam - on a monthly basis

Resources:

1. imdlib is a Python Package useful for downloading IMD data. Documentation
2. Assam Revenue circle maps -- This is the map we have to use to aggregate IMD data at Revenue circle level.

Additional references:

1. Tutorial on using imdlib - You can find many other tutorials.
2. Zonal Statisitics - is the keyword you have to lookup on to aggregate IMD data in a revenue circle.

[d-saikrishna]

@tauseefsshah to solve where you got stuck:

1. Each tif file you downloaded from IMDLIB will have multiple bands. (Consider band = array). Each band/array for one date. So When you download tif for a month, you get 30/31 bands/arrays.
2. Now you have to take a mean of all 30/31 bands/arrays and create a single array. That is the array of mean rainfall in a month. You can get some idea of doing this from this code. raster.read(1) Opens only one band/array - You have to loop across bands/arrays and take mean.
3. It is this mean array that you have to send to the zonal_stats function.

Let me know if you get stuck again.

[tauseefsshah]

@d-saikrishna Still doesn't work. I have updated the code to have the average rainfall per month and saving that as tif but when I read it, it's still showing ValueError: negative dimensions are not allowed. When I check the tif in qgis, it looks good.

[d-saikrishna]

Interesting issues that we can propose:

1. imdlib package (which serves IMD data through Python) is using a bad standard to serve the rasters. The rasters are good for visualisation - but fail when we want to calculate zonal statistics out of it. We fixed it. We can propose this as an issue or write a blog about it.
2. While calculating zonal stats, raster stats package is not taking a weighted average of pixels. As a result, it might miss a lot of pixels that are on the borders of a polygon (a problem especially if the pixel size is high). We solved it through resampling rasters to lesser spatial resolution. But if we can develop a way to take weighted average based on overlap %- that will be very useful

Edit: There is already a PR on rasterstats for the issue. Hopefully it gets merged :) perrygeo/python-rasterstats#136

[d-saikrishna]

@tauseefsshah

Once you are done with the scripts, create a small ETL diagram that we use in the README section. You can refer to the diagrams used in SENTINEL, BHUVAN etc.

You can create a duplicate slide in the following Presentation and download the image in the docs folder within IMD folder.
https://docs.google.com/presentation/d/1YQ4t21dduS18D9CLj6zddQ5v9KdEenzoqPSaZu2BhRo/edit#slide=id.g27c4711d02d_0_0

Feel free to detail things out in the README section :)

[tauseefsshah]

@d-saikrishna I have created a PR for merging the dev-rainfall branch into main, please review the code once and merge it. I'll create the etl diagram and README and push to the same branch so please don't delete it after merging

[d-saikrishna]

@tauseefsshah I have merged the code and made a few changes to suit the file naming convention. Every thing worked fine except for one issue.

The code is breaking when I run for 2023 year. Please check that once.

[d-saikrishna]

@tauseefsshah could you check it for 2023 year once?

[d-saikrishna]

Data is not available for 2023 yet. Source: Link

[d-saikrishna]

https://imdpune.gov.in/cmpg/Realtimedata/Rainfall/Rain_Download.html# -- for realtime data - need a way to integrate this

[d-saikrishna]

Data for 2023 is now available ad added