Pgwatch - Patroni Rest API Monitoring Enhancement

[kylorend3r]

Hi there,

I want to find out if we can monitor patroni rest api's and store the measurements in metric database. In other words, I would like to collect patroni metrics and visualise them.

I reviewed the codebase and it seems pgwatch doesn't retrieve patroni rest api metrics.

My idea is adding capability of adding a patroni rest endpoint monitoring source without affecting the previous patroni sources and collect the metrics from rest endpoints.

We can also create a patroni dashboard to visualise metrics as a final step.

Let me know your thoughts and opinions/feedback.

https://patroni.readthedocs.io/en/latest/rest_api.html#monitoring-endpoint

[pashagolub]

Hello! Yes, we have this feature in out TODO list! Thanks for confirming it's interesting for you.

To implement it we need first to introduce the additional kind/type of the source, e.g. http or rest.

Or maybe we can add another kind of machinery, similar to RPC sink, that will send measurements directly to pgwatch instance. This way we can collect the data from different kind of scripts, exporters, etc.

[kylorend3r]

The idea is promising. I'd like to restrict the input with patroni https endpoint format.

Because, there might be different http sources with different data formats. This is easy to guess but good to mention. My first feedback is to restrict/limit the input of the source so user will only be able to monitor patroni https.

Secondly, there might be different auth. methods to patroni http adresses. We can start with the basic one or we can provide a platform integrated with crendetials, SSL etc..

Do you have plan in your mind ? Do you need an initial draft ?

[pashagolub]

You don't need to restrict anything if you treat http(s) metrics as a regular one. But instead of SQL we might use jsonpath, e.g.
to list a leader

'$.members[*] ? (@.role == "leader").name'

or to list replicas

'$.members[*] ? (@.role == "replica").name'

Something like this. I'm still in a stage where I'm trying to think about it more before trying to code it. To make it as extendable as possible for future use.

And yes, auth and all the things must be solved as well. That's why I'm leaning to have http sources/metrics as first class citizen instead of additional hard-coded kludge. :-)

If you have any ideas/drafts, you are more than welcome to share those!