**BROKEN** plugin-cln-renepay: Unable to parse sendpay_failure

[grubles]

At commit 98a188b

Running tests/test_renepay.py::test_self_pay results in this error. I checked the flaky tracker and the occurrence of failures for this test is pretty low (2.26%) in the GH CI but on my machine it fails much more frequently.

lightningd-1 2025-12-16T12:54:37.584Z TRACE   plugin-cln-renepay: Calling modifier compute_routes
lightningd-1 2025-12-16T12:54:37.584Z TRACE   plugin-cln-renepay: Calling modifier send_routes
lightningd-1 2025-12-16T12:54:37.584Z DEBUG   plugin-cln-renepay: Sent route request: partid=1 amount=10000msat prob=0.980 fees=0msat delay=0 path=
lightningd-1 2025-12-16T12:54:37.584Z DEBUG   plugin-cln-renepay: notify msg info: Sent route request: partid=1 amount=10000msat prob=0.980 fees=0msat delay=0 path=
lightningd-1 2025-12-16T12:54:37.584Z TRACE   plugin-cln-renepay: Calling modifier sleep
lightningd-1 2025-12-16T12:54:37.585Z DEBUG   plugin-cln-renepay: renesendpay called: {"payment_hash":"6d6306f32990b154bd3c62262aea3f1345ad400778870ef4106c1a9a8552688a","partid":1,"groupid":1,"invoice":"lnbcrt100n1p55z45dsp57jeshyc5w0ckw4rr8qs3sft359ty3e9avlljfcvp5mj383q48dzspp5d43sduefjzc4f0fuvgnz463lzdz66sq80zrsaaqsdsdf4p2jdz9qdqgw3jhxapjxqyjw5qcqp99qxpqysgqnz86ly70v4aa2ertppkxtfuz9rlkvvg6frd3wj6fxq306advppsxucxu55ze0sx8zzqzu4wl6yhfjcygng603vwz6p3hhyaex7uasaqqx8qgwl","destination":"0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518","amount_msat":10000,"total_amount_msat":10000,"final_cltv":5,"route":[],"payment_secret":"f4b30b931473f16754633821182571a15648e4bd67ff24e181a6e513c4153b45"}
lightningd-1 2025-12-16T12:54:37.586Z DEBUG   lightningd: payment failed: Unknown invoice 6d6306f32990b154bd3c62262aea3f1345ad400778870ef4106c1a9a8552688a
lightningd-1 2025-12-16T12:54:37.586Z DEBUG   hsmd: Client: Received message 1 from client
lightningd-1 2025-12-16T12:54:37.587Z DEBUG   plugin-cln-renepay: JSON reply with unknown id '{"jsonrpc":"2.0","id":"-c:renepay#52/cln:renepay#101/cln-renepay:renesendpay#14/cln:renesendpay#102/cln-renepay:injectpaymentonion#16","error":{"code":218,"message":"Malformed error reply","data":{"onionreply":"947af89619cb06b4f3829789eb08c22ddcc86767009a8ba1028b83f4e6f6c6e62091121d09a9ac1fdf8002cc2347a98f912cf3d806b23d50274a51c908e85268d44dbdac4e1ffede49d4db4b410169e782f5fcaf84ed4cbe2fda18240b279c5a450af2e13e5c44acb8c9e303756caebf92fdfe7c13ad14b12ba1762fee986a67a03256f0318deb0c97b378ac28022d60bdc866f98d2e2feea0eb4b2f32b140812e93f137f24e6c2dd8af38641f0655e2b9dd40809361acb195ca3b3d6fadbca6d854d42f1fc26107528ed3b883d8215350552581ca502a6acdf281ad71fd353c53ff0555ad47b7047d62c1d36180b8da28ad2308f9be33b1a315f98e1da3366e940d7222e0a2bbddddda27f21f8189b6e5967215d3ea176da0288fea89ce52bdc6e1f0c6"}}}'
lightningd-1 2025-12-16T12:54:37.587Z DEBUG   plugin-cln-renepay: sendpay_failure notification: {"sendpay_failure":{"code":202,"message":"Malformed error reply","data":{"created_index":2,"id":2,"payment_hash":"6d6306f32990b154bd3c62262aea3f1345ad400778870ef4106c1a9a8552688a","groupid":1,"partid":1,"destination":"0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518","amount_msat":10000,"amount_sent_msat":10000,"created_at":1765889677,"status":"pending","bolt11":"lnbcrt100n1p55z45dsp57jeshyc5w0ckw4rr8qs3sft359ty3e9avlljfcvp5mj383q48dzspp5d43sduefjzc4f0fuvgnz463lzdz66sq80zrsaaqsdsdf4p2jdz9qdqgw3jhxapjxqyjw5qcqp99qxpqysgqnz86ly70v4aa2ertppkxtfuz9rlkvvg6frd3wj6fxq306advppsxucxu55ze0sx8zzqzu4wl6yhfjcygng603vwz6p3hhyaex7uasaqqx8qgwl","onionreply":"39556d401e7fb8c7b848a5f7decab03d712c9d9766fd7f48dab40b30e634aeac000e400f00000000000027100000006c00f20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}}
lightningd-1 2025-12-16T12:54:37.587Z **BROKEN** plugin-cln-renepay: Unable to parse sendpay_failure: {"code":202,"message":"Malformed error reply","data":{"created_index":2,"id":2,"payment_hash":"6d6306f32990b154bd3c62262aea3f1345ad400778870ef4106c1a9a8552688a","groupid":1,"partid":1,"destination":"0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518","amount_msat":10000,"amount_sent_msat":10000,"created_at":1765889677,"status":"pending","bolt11":"lnbcrt100n1p55z45dsp57jeshyc5w0ckw4rr8qs3sft359ty3e9avlljfcvp5mj383q48dzspp5d43sduefjzc4f0fuvgnz463lzdz66sq80zrsaaqsdsdf4p2jdz9qdqgw3jhxapjxqyjw5qcqp99qxpqysgqnz86ly70v4aa2ertppkxtfuz9rlkvvg6frd3wj6fxq306advppsxucxu55ze0sx8zzqzu4wl6yhfjcygng603vwz6p3hhyaex7uasaqqx8qgwl","onionreply":"39556d401e7fb8c7b848a5f7decab03d712c9d9766fd7f48dab40b30e634aeac000e400f00000000000027100000006c00f20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}
lightningd-1 2025-12-16T12:54:37.588Z INFO    plugin-cln-renepay: Killing plugin: exited during normal operation
DEBUG:root:Received response for renepay call: {'jsonrpc': '2.0', 'id': '-c:renepay#52', 'error': {'code': -4, 'message': 'Plugin terminated before replying to RPC call.'}}
{'run_id': 259295488937635840, 'github_repository': 'ElementsProject/lightning', 'github_sha': '98a188b3945ed4c7179e9b89ec286079bb06777c', 'github_ref': None, 'github_ref_name': 'master', 'github_run_id': None, 'github_head_ref': None, 'github_run_number': None, 'github_base_ref': None, 'github_run_attempt': None, 'testname': 'test_self_pay', 'start_time': 1765889671, 'end_time': 1765889677, 'outcome': 'fail'}
---------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------
Unable to parse sendpay_failure: {"code":202,"message":"Malformed error reply","data":{"created_index":2,"id":2,"payment_hash":"6d6306f32990b154bd3c62262aea3f1345ad400778870ef4106c1a9a8552688a","groupid":1,"partid":1,"destination":"0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518","amount_msat":10000,"amount_sent_msat":10000,"created_at":1765889677,"status":"pending","bolt11":"lnbcrt100n1p55z45dsp57jeshyc5w0ckw4rr8qs3sft359ty3e9avlljfcvp5mj383q48dzspp5d43sduefjzc4f0fuvgnz463lzdz66sq80zrsaaqsdsdf4p2jdz9qdqgw3jhxapjxqyjw5qcqp99qxpqysgqnz86ly70v4aa2ertppkxtfuz9rlkvvg6frd3wj6fxq306advppsxucxu55ze0sx8zzqzu4wl6yhfjcygng603vwz6p3hhyaex7uasaqqx8qgwl","onionreply":"39556d401e7fb8c7b848a5f7decab03d712c9d9766fd7f48dab40b30e634aeac000e400f00000000000027100000006c00f20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}
============================================================================================= short test summary info =============================================================================================
FAILED tests/test_renepay.py::test_self_pay - AssertionError: Regex pattern did not match.
ERROR tests/test_renepay.py::test_self_pay - ValueError: 

Happy to provide more logs if needed.

[Lagrang3]

I am able to reproduce it locally.

This test has three steps: a self payment that must succeed, then a repeated payment to the same invoice that also succeeds
and then try to pay an invoice that has been deleted, for which we expect a failure with a certain error message.

But in the last instance renepay is not able to parse the sendpay_failure notification.
By manual inspection I think it is not a formatting issue with the sendpay_failure but that renepay lacks the secrets
to decode the onionreply field, unfortunately this information is not provided in the logs.

As to why renepay lacks those secrets I think the issue can be traced back to this line:

lightningd-1 2025-12-16T12:54:37.587Z DEBUG   plugin-cln-renepay: JSON reply with unknown id '{"jsonrpc":"2.0","id":"-c:renepay#52/cln:renepay#101/cln-renepay:renesendpay#14/cln:renesendpay#102/cln-renepay:injectpaymentonion#16","error":{"code":218,"message":"Malformed error reply","data":{"onionreply":"947af89619cb06b4f3829789eb08c22ddcc86767009a8ba1028b83f4e6f6c6e62091121d09a9ac1fdf8002cc2347a98f912cf3d806b23d50274a51c908e85268d44dbdac4e1ffede49d4db4b410169e782f5fcaf84ed4cbe2fda18240b279c5a450af2e13e5c44acb8c9e303756caebf92fdfe7c13ad14b12ba1762fee986a67a03256f0318deb0c97b378ac28022d60bdc866f98d2e2feea0eb4b2f32b140812e93f137f24e6c2dd8af38641f0655e2b9dd40809361acb195ca3b3d6fadbca6d854d42f1fc26107528ed3b883d8215350552581ca502a6acdf281ad71fd353c53ff0555ad47b7047d62c1d36180b8da28ad2308f9be33b1a315f98e1da3366e940d7222e0a2bbddddda27f21f8189b6e5967215d3ea176da0288fea89ce52bdc6e1f0c6"}}}'

After calling renesesendpay we should get a response with the secrets while the payment is pending.
Instead we are getting a "JSON reply with unknown id".

Probably the same reported here: #8345.

The problem could also be a race between getting a reply from renesendpay from which we get the secrets and
getting the notification that the payment has failed.

Actions:

• we need to print more information into the logs as to why parsing a sendpay_failure fails,
• investigate why are we failing to receive replies from lightningd with "JSON reply with unknown id",
• make sure we don't call sendonion before we are ready to read a sendpay_failure notification,

[grubles]

Here are the full logs for lightningd-1 and lightningd-2:

test_renepay_log2.txt
test_renepay_log1.txt