RGS crashing every few minutes. Result::unwrap() on an Err value: Elapsed(()) #104
Description
I am getting lots of Result::unwrap() errors since a week. RGS crashes every few minutes. This is happening since 1 week.
I am running the current main branch.
Seems like the insert times out. The server has plenty of memory/disk space/cpu available.
My channel_updates table has 67,409,921 rows. There seems to be a unique contraint on this table which I highly suspect slows things down to the extend that the timeout is triggered.
A simple select count(*) from channel_updates takes 17s already so the insert will likely take even longer.
rgs=# SELECT COUNT(*) FROM channel_updates;
count
----------
67409921
(1 row)
Time: 17512.966 ms (00:17.513)
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: thread 'tokio-runtime-worker' panicked at /home/rgs/rgs2/src/persistence.rs:297:18:
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: called `Result::unwrap()` on an `Err` value: Elapsed(())
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: stack backtrace:
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 0: 0x60cac57a134c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 1: 0x60cac55e41cb - core::fmt::write::hf5713710ce10ff22
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 2: 0x60cac57765a2 - std::io::Write::write_fmt::hda708db57927dacf
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 3: 0x60cac57a5df8 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 4: 0x60cac57a6c96 - std::panicking::rust_panic_with_hook::had2118629c312a4a
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 5: 0x60cac57a6795 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 6: 0x60cac57a66f9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 7: 0x60cac57a66e4 - rust_begin_unwind
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 8: 0x60cac55ac4a2 - core::panicking::panic_fmt::h3eea515d05f7a35e
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 9: 0x60cac55ac885 - core::result::unwrap_failed::h7c8d8bbbcf45dc13
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 10: 0x60cac5692804 - tokio::runtime::task::core::Core<T,S>::poll::h07188bddb2eac8bc
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 11: 0x60cac55cc134 - tokio::runtime::task::raw::poll::h2037b2fc13a43256
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 12: 0x60cac57b5035 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h098373b6dcc81fb6
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 13: 0x60cac57b363b - tokio::runtime::scheduler::multi_thread::worker::run::h06b00e2af4810ce2
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 14: 0x60cac57ac338 - tokio::runtime::task::raw::poll::hc895a065d1c83131
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 15: 0x60cac57adf35 - std::sys::backtrace::__rust_begin_short_backtrace::h6910da00ffe66e61
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 16: 0x60cac57badcd - core::ops::function::FnOnce::call_once{{vtable.shim}}::h73e9d65efeb1127e
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 17: 0x60cac57a80ab - std::sys::pal::unix::thread::Thread::new::thread_start::hcdbd1049068002f4
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 18: 0x7d119629caa4 - <unknown>
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 19: 0x7d1196329c6c - <unknown>
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: 20: 0x0 - <unknown>
Config:
# Environment variables can be set directly in the file
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_HOST=localhost"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_USER=postgres"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_PASSWORD=****"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_NAME=rgs"
Environment="RAPID_GOSSIP_SYNC_SERVER_NETWORK=mainnet"
Environment="RAPID_GOSSIP_SYNC_SERVER_SNAPSHOT_INTERVAL=10800"
Environment="RAPID_GOSSIP_SYNC_SERVER_CACHES_PATH=/var/rgs_cache"
Environment="BITCOIN_REST_DOMAIN=10.172.0.3"
Environment="BITCOIN_REST_PORT=8332"
Environment="BITCOIN_REST_PATH=/rest/"
Environment="LN_PEERS=***"
Environment="RUST_BACKTRACE=full"