-
Notifications
You must be signed in to change notification settings - Fork 215
Open
Labels
Description
Related to #3551
Currently dstack replicas take blocking advisory lock for processing gateway connections (check_or_restart() and try_collect_stats()):
| async def _process_active_connections(): | |
| connections = await gateway_connections_pool.all() | |
| # Two server processes on a single host cannot process | |
| # gateway connections and init gateway connections concurrently: | |
| # Race conditions cause conflicting tunnels being opened. | |
| async with get_session_ctx() as session: | |
| async with advisory_lock_ctx( | |
| bind=session, | |
| dialect_name=get_db().dialect_name, | |
| resource="gateway_tunnels", | |
| ): | |
| await asyncio.gather(*(_process_connection(conn) for conn in connections)) |
This means replicas can hold db connections open for some time potentially and that should be avoided as much as possible. A solution could be non-blocking pg_try_advisory_lock, but we need to ensure all the replicas can processes eventually.
Reactions are currently unavailable