Re-enable ignored tests: cross-node subscription and multi-node operations

[sanity]

Problem

Multiple tests are currently disabled with #[ignore] that indicate significant functionality issues, particularly around:

1. Cross-node subscription propagation - updates don't reliably reach subscribers on different nodes
2. Multi-node network operations - various failures in partially connected networks

These disabled tests represent real bugs that affect production use of Freenet.

Affected Tests

High Priority - Cross-Node Subscription (Core Functionality)

Test	File	Issue
test_multiple_clients_subscription	crates/core/tests/operations.rs:646	Client 3 on node-b never receives update notifications from node-a

Root Cause Analysis:
When running this test, the failure mode is:

• Client 1 on node-a PUTs a contract and subscribes
• Client 2 on node-a subscribes (works - same node)
• Client 3 on node-b GETs the contract and subscribes
• Client 1 sends an UPDATE
• Client 3 never receives the update notification

Logs show transaction timeouts. The issue is that cross-node subscription propagation doesn't properly register node-b as a network subscriber on the node seeding the contract.

This was previously tracked in #1798 (closed prematurely).

High Priority - Multi-Node Operations

Test	File	Issue
test_ping_multi_node	apps/freenet-ping/app/tests/run_app.rs:463	"currently fails"
test_ping_multi_node_with_update	apps/freenet-ping/app/tests/run_app.rs:1090	"currently fails"
test_ping_partially_connected_network	apps/freenet-ping/app/tests/run_app_partially_connected_network.rs:36	"nodes fail on startup with channel closed errors"

Medium Priority - Connection Handler

Test	File	Issue
simulate_nat_traversal_drop_first_packets_for_all	crates/core/src/transport/connection_handler.rs:1509	"should be fixed"
simulate_nat_traversal_b_to_a_no_first_response	connection_handler.rs:1692	"should be fixed"
simulate_nat_traversal_a_to_b_no_first_response	connection_handler.rs:1721	"should be fixed"
simulate_nat_traversal_repeat_first_packet	connection_handler.rs:1779	"Flaky in CI"
simulate_nat_traversal_drop_packets_for_both	connection_handler.rs:1911	"should be fixed"

Lower Priority - Application Tests

Test	File	Issue
test_basic_room_creation	crates/core/tests/ubertest.rs:263	"connection issues"
test_app_ubertest	crates/core/tests/ubertest.rs:389	"requires fixes still in progress"

Expected Outcome

1. Cross-node subscription propagation works reliably
2. Updates sent from one node reach all subscribed clients on other nodes
3. Multi-node tests pass consistently in CI

Technical Context

PR #2219 recently separated local client subscriptions from network peer subscriptions, which is architecturally correct. However, the cross-node test failure predates this change and is about the network-level subscription routing between nodes.

The subscription flow for cross-node updates should be:

1. Node-b receives subscribe request from local client
2. Node-b forwards RequestSub to peer closer to contract location (node-a or gateway)
3. That peer registers node-b as a network subscriber via add_subscriber()
4. When UPDATE happens, get_broadcast_targets_update() returns node-b
5. UPDATE message is sent to node-b
6. Node-b delivers to local subscribed clients

Something in this chain is broken for cross-node scenarios.

Acceptance Criteria

• test_multiple_clients_subscription passes reliably (not flaky)
• At least one multi-node ping test (test_ping_multi_node or test_ping_multi_node_with_update) passes
• Tests are re-enabled (remove #[ignore])

[AI-assisted - Claude]

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-bug (98% confidence)
• P-high (90% confidence)
• E-hard (80% confidence)
• A-networking (96% confidence)

Reasoning: The report describes multiple disabled tests that reveal broken cross-node subscription propagation and multi-node operation failures (updates not reaching subscribers on other nodes, transactions timing out, nodes failing to start in partially connected setups). This is a functional defect (T-bug) affecting core networking behavior and production use, so it merits high priority (P-high). The root causes span network-level subscription routing, peer registration, and NAT/connection handling, which are non-trivial and touch core transport logic and test infrastructure, so estimated effort is high (E-hard). Relevant areas are A-networking (peer/subscription routing, transport, NAT traversal) and A-developer-xp (tests/CI flakiness and re-enabling tests). Confidence scores reflect the strong evidence in the issue and described test failures.

Previous labels: none

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[sanity]

Detailed Root Cause Analysis

After extensive investigation with detailed tracing, I've identified the core issue with test_multiple_clients_subscription:

Problem: Subscription Tree Doesn't Match UPDATE Propagation Path

When Client 3 on node-b subscribes to a contract that was PUT on node-a:

1. node-b sends RequestSub to gateway
2. gateway uses k_closest_potentially_caching() to find a node with the contract
3. gateway may forward SeekNode to node-b itself (because node-b cached the contract during GET)
4. node-b processes SeekNode, registers the subscription locally, returns ReturnSub
5. gateway receives ReturnSub and should register node-b as a subscriber

When Client 1 on node-a sends UPDATE:

• UPDATE goes through gateway (central hub)
• gateway calls get_broadcast_targets_update() which checks subscribers_of(key)
• If gateway doesn't have node-b in its subscriber list, the UPDATE doesn't propagate to node-b

Race Condition Evidence

Looking at timestamps from a failed run:

19:05:31.987306Z - ReturnSub sent to node-b (subscription completing)
19:05:31.988725Z - UPDATE request sent (only 1.4ms later!)

The ReturnSub for Client 3's subscription is still being processed when the UPDATE starts. The subscription handler (add_subscriber at line 997 of subscribe.rs) may not have completed before get_broadcast_targets_update() is called.

Test Results

Approach	Result
No delay	~20-30% pass rate
5s delay before UPDATE	~33% pass rate (still flaky)

The 5s delay helps but doesn't fully fix the issue, suggesting there are additional problems beyond the immediate race condition:

1. Subscription tree may go to wrong node (cached copy holder instead of UPDATE origin path)
2. Multiple network paths can result in different subscription trees
3. No guarantee that intermediate nodes (gateway) have all subscribers registered

Proposed Protocol-Level Fix

The subscription tree architecture needs to guarantee that:

1. All subscribers are registered on nodes along the UPDATE propagation path
2. OR the UPDATE broadcast queries all potentially-caching nodes, not just the subscription tree

Possible approaches:

1. Confirmation-based subscription: Don't return SubscribeResponse until all intermediate nodes confirm registration
2. UPDATE broadcast to all caching nodes: Use k_closest_potentially_caching() for UPDATE broadcast, not just subscribers_of()
3. Centralized subscription registry: Maintain subscriber list at a deterministic location (contract hash location)

Files Investigated

• crates/core/src/operations/subscribe.rs - subscription flow and add_subscriber calls
• crates/core/src/operations/update.rs - get_broadcast_targets_update() at line 744
• crates/core/src/node/network_bridge/p2p_protoc.rs - message routing
• crates/core/tests/operations.rs - test implementation

Recommendation

This is a fundamental architectural issue that requires protocol-level changes. The test should remain #[ignore] on main until a proper fix is implemented. The fixes in PR #2222 (race conditions, state transitions) were necessary but not sufficient for reliable cross-node subscriptions.

[AI-assisted - Claude]