Skip to content

Commit 85f415c

Browse files
LucaButBoringLucaButBoring
authored
Merge branch 'main' into feat/tasks
2 parents b0a73d5 + 4fe8f3e commit 85f415c

File tree

14 files changed

+1558
-0
lines changed

14 files changed

+1558
-0
lines changed

‎.github/workflows/conformance.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
name: Conformance Tests
2+
3+
on:
4+
pull_request: {}
5+
push:
6+
branches: [main]
7+
workflow_dispatch:
8+
9+
jobs:
10+
server:
11+
name: Server Conformance
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Set up JDK 17
17+
uses: actions/setup-java@v4
18+
with:
19+
java-version: '17'
20+
distribution: 'temurin'
21+
cache: 'maven'
22+
23+
- name: Build and start server
24+
run: |
25+
mvn clean install -DskipTests
26+
mvn exec:java -pl conformance-tests/server-servlet -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet" &
27+
timeout 30 bash -c 'until curl -s http://localhost:8080/mcp > /dev/null 2>&1; do sleep 0.5; done'
28+
29+
- name: Run conformance tests
30+
uses: modelcontextprotocol/conformance@v0.1.11
31+
with:
32+
mode: server
33+
url: http://localhost:8080/mcp
34+
suite: active
35+
expected-failures: ./conformance-tests/conformance-baseline.yml
36+
37+
client:
38+
name: Client Conformance
39+
runs-on: ubuntu-latest
40+
strategy:
41+
matrix:
42+
scenario: [initialize, tools_call, elicitation-sep1034-client-defaults, sse-retry]
43+
steps:
44+
- uses: actions/checkout@v4
45+
46+
- name: Set up JDK 17
47+
uses: actions/setup-java@v4
48+
with:
49+
java-version: '17'
50+
distribution: 'temurin'
51+
cache: 'maven'
52+
53+
- name: Build client
54+
run: mvn clean install -DskipTests
55+
56+
- name: Run conformance test
57+
uses: modelcontextprotocol/conformance@v0.1.11
58+
with:
59+
mode: client
60+
command: 'java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar'
61+
scenario: ${{ matrix.scenario }}
62+
expected-failures: ./conformance-tests/conformance-baseline.yml

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ build/
1010
out
1111
/.gradletasknamecache
1212
**/*.flattened-pom.xml
13+
**/dependency-reduced-pom.xml
1314

1415
### IDE - Eclipse/STS ###
1516
.apt_generated

‎conformance-tests/VALIDATION_RESULTS.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# MCP Java SDK Conformance Test Validation Results
2+
3+
## Summary
4+
5+
**Server Tests:** 37/40 passed (92.5%)
6+
**Client Tests:** 3/4 scenarios passed (9/10 checks passed)
7+
8+
## Server Test Results
9+
10+
### Passing (37/40)
11+
12+
- **Lifecycle & Utilities (4/4):** initialize, ping, logging-set-level, completion-complete
13+
- **Tools (11/11):** All scenarios including progress notifications ✨
14+
- **Elicitation (10/10):** SEP-1034 defaults (5 checks), SEP-1330 enums (5 checks)
15+
- **Resources (4/6):** list, read-text, read-binary, templates-read
16+
- **Prompts (4/4):** list, simple, with-args, embedded-resource, with-image
17+
- **SSE Transport (2/2):** Multiple streams
18+
- **Security (1/2):** Localhost validation passes
19+
20+
### Failing (3/40)
21+
22+
1. **resources-subscribe** - Not implemented in SDK
23+
2. **resources-unsubscribe** - Not implemented in SDK
24+
3. **dns-rebinding-protection** - Missing Host/Origin validation (1/2 checks)
25+
26+
## Client Test Results
27+
28+
### Passing (3/4 scenarios, 9/10 checks)
29+
30+
- **initialize (1/1):** Protocol negotiation, clientInfo, capabilities
31+
- **tools_call (1/1):** Tool discovery and invocation
32+
- **elicitation-sep1034-client-defaults (5/5):** Default values for string, integer, number, enum, boolean
33+
34+
### Partially Passing (1/4 scenarios, 1/2 checks)
35+
36+
- **sse-retry (1/2 + 1 warning):**
37+
- ✅ Reconnects after stream closure
38+
- ❌ Does not respect retry timing
39+
- ⚠️ Does not send Last-Event-ID header (SHOULD requirement)
40+
41+
**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing.
42+
43+
## Known Limitations
44+
45+
1. **Resource Subscriptions:** SDK doesn't implement `resources/subscribe` and `resources/unsubscribe` handlers
46+
2. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header
47+
3. **DNS Rebinding Protection:** Missing Host/Origin header validation in server transport
48+
49+
## Running Tests
50+
51+
### Server
52+
```bash
53+
# Start server
54+
cd conformance-tests/server-servlet
55+
../../mvnw compile exec:java -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"
56+
57+
# Run tests (in another terminal)
58+
npx @modelcontextprotocol/conformance server --url http://localhost:8080/mcp --suite active
59+
```
60+
61+
### Client
62+
```bash
63+
# Build
64+
cd conformance-tests/client-jdk-http-client
65+
../../mvnw clean package -DskipTests
66+
67+
# Run all scenarios
68+
for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
69+
npx @modelcontextprotocol/conformance client \
70+
--command "java -jar target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
71+
--scenario $scenario
72+
done
73+
```
74+
75+
## Recommendations
76+
77+
### High Priority
78+
1. Fix client SSE retry field handling in `HttpClientStreamableHttpTransport`
79+
2. Implement resource subscription handlers in `McpStatelessAsyncServer`
80+
81+
### Medium Priority
82+
3. Add Host/Origin validation in `HttpServletStreamableServerTransportProvider` for DNS rebinding protection

‎conformance-tests/client-jdk-http-client/README.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# MCP Conformance Tests - JDK HTTP Client
2+
3+
This module provides a conformance test client implementation for the Java MCP SDK using the JDK HTTP Client with Streamable HTTP transport.
4+
5+
## Overview
6+
7+
The conformance test client is designed to work with the [MCP Conformance Test Framework](https://github.com/modelcontextprotocol/conformance). It validates that the Java MCP SDK client properly implements the MCP specification.
8+
9+
## Architecture
10+
11+
The client reads test scenarios from environment variables and accepts the server URL as a command-line argument, following the conformance framework's conventions:
12+
13+
- **MCP_CONFORMANCE_SCENARIO**: Environment variable specifying which test scenario to run
14+
- **Server URL**: Passed as the last command-line argument
15+
16+
## Supported Scenarios
17+
18+
Currently implemented scenarios:
19+
20+
- **initialize**: Tests the MCP client initialization handshake only
21+
- ✅ Validates protocol version negotiation
22+
- ✅ Validates clientInfo (name and version)
23+
- ✅ Validates proper handling of server capabilities
24+
- Does NOT call any tools or perform additional operations
25+
26+
- **tools_call**: Tests tool discovery and invocation
27+
- ✅ Initializes the client
28+
- ✅ Lists available tools from the server
29+
- ✅ Calls the `add_numbers` tool with test arguments (a=5, b=3)
30+
- ✅ Validates the tool result
31+
32+
- **elicitation-sep1034-client-defaults**: Tests client applies default values for omitted elicitation fields (SEP-1034)
33+
- ✅ Initializes the client
34+
- ✅ Lists available tools from the server
35+
- ✅ Calls the `test_client_elicitation_defaults` tool
36+
- ✅ Validates that the client properly applies default values from JSON schema to elicitation responses (5/5 checks pass)
37+
38+
- **sse-retry**: Tests client respects SSE retry field timing and reconnects properly (SEP-1699)
39+
- ⚠️ Initializes the client
40+
- ⚠️ Lists available tools from the server
41+
- ⚠️ Calls the `test_reconnection` tool which triggers SSE stream closure
42+
- ✅ Client reconnects after stream closure (PASSING)
43+
- ❌ Client does not respect retry timing (FAILING)
44+
- ⚠️ Client does not send Last-Event-ID header (WARNING - SHOULD requirement)
45+
46+
## Building
47+
48+
Build the executable JAR:
49+
50+
```bash
51+
cd conformance-tests/client-jdk-http-client
52+
../../mvnw clean package -DskipTests
53+
```
54+
55+
This creates an executable JAR at:
56+
```
57+
target/client-jdk-http-client-0.18.0-SNAPSHOT.jar
58+
```
59+
60+
## Running Tests
61+
62+
### Using the Conformance Framework
63+
64+
Run a single scenario:
65+
66+
```bash
67+
npx @modelcontextprotocol/conformance client \
68+
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
69+
--scenario initialize
70+
71+
npx @modelcontextprotocol/conformance client \
72+
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
73+
--scenario tools_call
74+
75+
npx @modelcontextprotocol/conformance client \
76+
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
77+
--scenario elicitation-sep1034-client-defaults
78+
79+
npx @modelcontextprotocol/conformance client \
80+
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
81+
--scenario sse-retry
82+
```
83+
84+
Run with verbose output:
85+
86+
```bash
87+
npx @modelcontextprotocol/conformance client \
88+
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
89+
--scenario initialize \
90+
--verbose
91+
```
92+
93+
### Manual Testing
94+
95+
You can also run the client manually if you have a test server:
96+
97+
```bash
98+
export MCP_CONFORMANCE_SCENARIO=initialize
99+
java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar http://localhost:3000/mcp
100+
```
101+
102+
## Test Results
103+
104+
The conformance framework generates test results showing:
105+
106+
**Current Status (3/4 scenarios passing):**
107+
- ✅ initialize: 1/1 checks passed
108+
- ✅ tools_call: 1/1 checks passed
109+
- ✅ elicitation-sep1034-client-defaults: 5/5 checks passed
110+
- ⚠️ sse-retry: 1/2 checks passed, 1 warning
111+
112+
Test result files are generated in `results/<scenario>-<timestamp>/`:
113+
- `checks.json`: Array of conformance check results with pass/fail status
114+
- `stdout.txt`: Client stdout output
115+
- `stderr.txt`: Client stderr output
116+
117+
### Known Issue: SSE Retry Handling
118+
119+
The `sse-retry` scenario currently fails because:
120+
1. The client treats the SSE `retry:` field as invalid instead of parsing it
121+
2. The client does not implement retry timing (reconnects immediately)
122+
3. The client does not send the Last-Event-ID header on reconnection
123+
124+
This is a known limitation in the `HttpClientStreamableHttpTransport` implementation.
125+
126+
## Next Steps
127+
128+
Future enhancements:
129+
130+
- Fix SSE retry field handling (SEP-1699) to properly parse and respect retry timing
131+
- Implement Last-Event-ID header on reconnection for resumability
132+
- Add auth scenarios (currently excluded as per requirements)
133+
- Implement a comprehensive "everything-client" pattern
134+
- Add to CI/CD pipeline
135+
- Create expected-failures baseline for known issues

‎conformance-tests/client-jdk-http-client/pom.xml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
<parent>
7+
<groupId>io.modelcontextprotocol.sdk</groupId>
8+
<artifactId>conformance-tests</artifactId>
9+
<version>0.18.0-SNAPSHOT</version>
10+
</parent>
11+
<artifactId>client-jdk-http-client</artifactId>
12+
<packaging>jar</packaging>
13+
<name>MCP Conformance Tests - JDK HTTP Client</name>
14+
<description>JDK HTTP Client conformance tests for the Java MCP SDK</description>
15+
<url>https://github.com/modelcontextprotocol/java-sdk</url>
16+
17+
<scm>
18+
<url>https://github.com/modelcontextprotocol/java-sdk</url>
19+
<connection>git://github.com/modelcontextprotocol/java-sdk.git</connection>
20+
<developerConnection>git@github.com/modelcontextprotocol/java-sdk.git</developerConnection>
21+
</scm>
22+
23+
<dependencies>
24+
<dependency>
25+
<groupId>io.modelcontextprotocol.sdk</groupId>
26+
<artifactId>mcp</artifactId>
27+
<version>0.18.0-SNAPSHOT</version>
28+
</dependency>
29+
30+
<!-- Logging -->
31+
<dependency>
32+
<groupId>ch.qos.logback</groupId>
33+
<artifactId>logback-classic</artifactId>
34+
<version>${logback.version}</version>
35+
<scope>runtime</scope>
36+
</dependency>
37+
</dependencies>
38+
39+
<build>
40+
<plugins>
41+
<!-- Maven Shade Plugin for creating executable JAR with dependencies -->
42+
<plugin>
43+
<groupId>org.apache.maven.plugins</groupId>
44+
<artifactId>maven-shade-plugin</artifactId>
45+
<version>3.5.1</version>
46+
<executions>
47+
<execution>
48+
<phase>package</phase>
49+
<goals>
50+
<goal>shade</goal>
51+
</goals>
52+
<configuration>
53+
<transformers>
54+
<transformer
55+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
56+
<mainClass>io.modelcontextprotocol.conformance.client.ConformanceJdkClientMcpClient</mainClass>
57+
</transformer>
58+
<transformer
59+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
60+
</transformers>
61+
<filters>
62+
<filter>
63+
<artifact>*:*</artifact>
64+
<excludes>
65+
<exclude>META-INF/*.SF</exclude>
66+
<exclude>META-INF/*.DSA</exclude>
67+
<exclude>META-INF/*.RSA</exclude>
68+
</excludes>
69+
</filter>
70+
</filters>
71+
</configuration>
72+
</execution>
73+
</executions>
74+
</plugin>
75+
</plugins>
76+
</build>
77+
78+
</project>

0 commit comments

Comments
 (0)