fix(openai): Token reporting

[alexander-alderman-webb]

Description

Start counting tokens in the case that the content field of an item in Completions or Responses input schemas contain a list of dictionaries, when the tiktoken fallback case is reached.

Issues

Closes #5390

Reminders

• Please add tests to validate your changes, and lint your code using tox -e linters.
• Add GH Issue ID & Linear ID (if applicable)
• PR title should use conventional commit style (feat:, fix:, ref:, meta:)
• For external contributors: CONTRIBUTING.md, Sentry SDK development docs, Discord community

[github-actions]

Semver Impact of This PR

🟢 Patch (bug fixes)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

---

New Features ✨

Span Streaming

• feat(span-streaming): Add spans to telemetry pipeline, add span name and attributes (3) by sentrivana in #5399
• feat(span-streaming): Add span batcher (2) by sentrivana in #5398
• feat(span-streaming): Add experimental trace_lifecycle switch (1) by sentrivana in #5397

Bug Fixes 🐛

Span Streaming

• fix(span-streaming): Always preserialize attributes by sentrivana in #5407
• fix(span-streaming): Warn about thread usage if any batcher is active by sentrivana in #5408

Other

• fix(arq): handle settings_cls passed as keyword argument by nc9 in #5393
• fix(dramatiq): cleanup isolated scope and transaction when message is skipped by frankie567 in #5346
• fix(google-genai): deactivate google genai when langchain is used by shellmayr in #5389

• fix(openai): Token reporting by alexander-alderman-webb in #5406

Internal Changes 🔧

Fastmcp

• test(fastmcp): Use AsyncClient for SSE by alexander-alderman-webb in #5400
• test(fastmcp): Use TestClient for Streamable HTTP by alexander-alderman-webb in #5384
• test(fastmcp): Simulate stdio transport with memory streams by alexander-alderman-webb in #5333

Mcp

• test(mcp): Use AsyncClient for SSE by alexander-alderman-webb in #5396
• test(mcp): Use TestClient for Streamable HTTP by alexander-alderman-webb in #5383
• test(mcp): Remove unused stdio helpers by alexander-alderman-webb in #5409
• test(mcp): Simulate stdio transport with memory streams by alexander-alderman-webb in #5329

Other

• ci: migration to the new codecov action by MathurAditya724 in #5392

---

_{🤖 This preview updates automatically when you update the PR.}

[github-actions]

Codecov Results 📊

✅ 1763 passed | ⏭️ 166 skipped | Total: 1929 | Pass Rate: 91.39% | Execution Time: 3m 20s

All tests are passing successfully.

❌ Patch coverage is 4.00%. Project has 11538 uncovered lines.

Files with missing lines (172)

File	Patch %	Lines
langchain.py	3.28%	⚠️ 590 Missing
openai.py	5.57%	⚠️ 492 Missing
utils.py	0.00%	⚠️ 479 Missing
__init__.py	5.08%	⚠️ 374 Missing
starlette.py	5.34%	⚠️ 337 Missing
mcp.py	4.85%	⚠️ 255 Missing
anthropic.py	9.43%	⚠️ 240 Missing
__init__.py	3.93%	⚠️ 220 Missing
langgraph.py	5.29%	⚠️ 215 Missing
span_processor.py	0.00%	⚠️ 205 Missing
__init__.py	5.14%	⚠️ 203 Missing
utils.py	83.90%	⚠️ 147 Missing and 47 partials
strawberry.py	7.54%	⚠️ 184 Missing
aws_lambda.py	16.50%	⚠️ 167 Missing
spark_driver.py	0.00%	⚠️ 166 Missing
huggingface_hub.py	8.89%	⚠️ 164 Missing
sanic.py	9.60%	⚠️ 160 Missing
aiohttp.py	10.84%	⚠️ 148 Missing
ai_client.py	0.00%	⚠️ 145 Missing
litellm.py	0.00%	⚠️ 138 Missing
tracing_utils.py	83.28%	⚠️ 98 Missing and 34 partials
litestar.py	9.59%	⚠️ 132 Missing
starlite.py	8.33%	⚠️ 132 Missing
scope.py	91.11%	⚠️ 73 Missing and 42 partials
falcon.py	8.94%	⚠️ 112 Missing
flask.py	12.50%	⚠️ 112 Missing
asgi.py	20.71%	⚠️ 111 Missing
cohere.py	12.70%	⚠️ 110 Missing
dramatiq.py	0.00%	⚠️ 110 Missing
agent_run.py	0.00%	⚠️ 110 Missing
arq.py	9.92%	⚠️ 109 Missing
asgi.py	0.00%	⚠️ 109 Missing
pymongo.py	10.17%	⚠️ 106 Missing
beat.py	8.62%	⚠️ 106 Missing
caching.py	0.00%	⚠️ 106 Missing
client.py	87.65%	⚠️ 62 Missing and 43 partials
transport.py	76.96%	⚠️ 91 Missing and 12 partials
utils.py	0.00%	⚠️ 103 Missing
templates.py	0.00%	⚠️ 100 Missing
quart.py	16.10%	⚠️ 99 Missing
gcp.py	0.00%	⚠️ 98 Missing
otlp.py	0.00%	⚠️ 97 Missing
utils.py	13.39%	⚠️ 97 Missing
pyramid.py	13.76%	⚠️ 94 Missing
tornado.py	14.55%	⚠️ 94 Missing
__init__.py	0.00%	⚠️ 93 Missing
models.py	5.15%	⚠️ 92 Missing
bottle.py	11.65%	⚠️ 91 Missing
middleware.py	0.00%	⚠️ 90 Missing
agent_run.py	0.00%	⚠️ 90 Missing
_wsgi_common.py	30.71%	⚠️ 88 Missing
tracing.py	87.37%	⚠️ 60 Missing and 27 partials
beam.py	0.00%	⚠️ 84 Missing
loguru.py	11.58%	⚠️ 84 Missing
ray.py	0.00%	⚠️ 84 Missing
invoke_agent.py	0.00%	⚠️ 79 Missing
runner.py	0.00%	⚠️ 77 Missing
clickhouse_driver.py	17.58%	⚠️ 75 Missing
pure_eval.py	0.00%	⚠️ 73 Missing
ariadne.py	14.46%	⚠️ 71 Missing
gql.py	10.13%	⚠️ 71 Missing
hub.py	67.36%	⚠️ 63 Missing and 7 partials
sqlalchemy.py	10.26%	⚠️ 70 Missing
fastapi.py	15.85%	⚠️ 69 Missing
graphene.py	12.82%	⚠️ 68 Missing
spotlight.py	55.56%	⚠️ 64 Missing and 3 partials
transactions.py	0.00%	⚠️ 67 Missing
rq.py	15.38%	⚠️ 66 Missing
streaming.py	0.00%	⚠️ 66 Missing
huey.py	17.72%	⚠️ 65 Missing
boto3.py	14.86%	⚠️ 63 Missing
utils.py	16.00%	⚠️ 63 Missing
httpx.py	16.44%	⚠️ 61 Missing
asyncio.py	27.50%	⚠️ 58 Missing
chalice.py	16.18%	⚠️ 57 Missing
propagator.py	0.00%	⚠️ 57 Missing
server.py	0.00%	⚠️ 56 Missing
spark_worker.py	0.00%	⚠️ 56 Missing
graph_nodes.py	0.00%	⚠️ 52 Missing
_async_common.py	0.00%	⚠️ 51 Missing
views.py	0.00%	⚠️ 50 Missing
tools.py	0.00%	⚠️ 49 Missing
caches.py	16.95%	⚠️ 49 Missing
transaction_profiler.py	90.15%	⚠️ 33 Missing and 16 partials
_asgi_common.py	16.67%	⚠️ 45 Missing
signals_handlers.py	0.00%	⚠️ 44 Missing
invoke_agent.py	0.00%	⚠️ 43 Missing
utils.py	85.99%	⚠️ 36 Missing and 5 partials
client.py	0.00%	⚠️ 40 Missing
_sync_common.py	22.00%	⚠️ 39 Missing
client.py	0.00%	⚠️ 38 Missing
_queue.py	68.89%	⚠️ 28 Missing and 8 partials
_span_batcher.py	35.71%	⚠️ 36 Missing
server.py	0.00%	⚠️ 35 Missing
continuous_profiler.py	92.65%	⚠️ 23 Missing and 9 partials
monitoring.py	62.20%	⚠️ 31 Missing
launchdarkly.py	0.00%	⚠️ 31 Missing
tools.py	0.00%	⚠️ 30 Missing
integration.py	0.00%	⚠️ 30 Missing
trytond.py	0.00%	⚠️ 29 Missing
error_tracing.py	0.00%	⚠️ 29 Missing
envelope.py	92.35%	⚠️ 15 Missing and 13 partials
_openai_completions_api.py	19.35%	⚠️ 25 Missing
redis_cluster.py	26.47%	⚠️ 25 Missing
typer.py	0.00%	⚠️ 24 Missing
tasks.py	0.00%	⚠️ 24 Missing
statsig.py	0.00%	⚠️ 23 Missing
execute_tool.py	0.00%	⚠️ 23 Missing
queries.py	25.81%	⚠️ 23 Missing
api.py	89.26%	⚠️ 16 Missing and 6 partials
session.py	91.11%	⚠️ 8 Missing and 14 partials
rust_tracing.py	90.21%	⚠️ 14 Missing and 8 partials
ai_client.py	15.38%	⚠️ 22 Missing
sessions.py	90.98%	⚠️ 12 Missing and 9 partials
worker.py	82.80%	⚠️ 16 Missing and 5 partials
logging.py	97.39%	⚠️ 4 Missing and 17 partials
unraisablehook.py	0.00%	⚠️ 21 Missing
__init__.py	8.70%	⚠️ 21 Missing
__init__.py	81.11%	⚠️ 17 Missing and 3 partials
openfeature.py	0.00%	⚠️ 20 Missing
execute_tool.py	0.00%	⚠️ 20 Missing
unleash.py	0.00%	⚠️ 19 Missing
_werkzeug.py	51.85%	⚠️ 13 Missing and 5 partials
logger.py	58.97%	⚠️ 16 Missing and 2 partials
wsgi.py	91.41%	⚠️ 11 Missing and 7 partials
model_request.py	0.00%	⚠️ 18 Missing
traces.py	41.38%	⚠️ 17 Missing
cloud_resource_context.py	91.03%	⚠️ 13 Missing and 4 partials
_batcher.py	86.25%	⚠️ 11 Missing and 5 partials
utils.py	0.00%	⚠️ 16 Missing
redis.py	25.00%	⚠️ 15 Missing
gnu_backtrace.py	88.24%	⚠️ 6 Missing and 7 partials
__init__.py	27.78%	⚠️ 13 Missing
serializer.py	95.16%	⚠️ 9 Missing and 3 partials
executing.py	81.58%	⚠️ 7 Missing and 5 partials
threading.py	92.63%	⚠️ 7 Missing and 4 partials
redis_py_cluster_legacy.py	26.67%	⚠️ 11 Missing
utils.py	41.18%	⚠️ 10 Missing
utils.py	85.19%	⚠️ 8 Missing and 2 partials
monitor.py	91.80%	⚠️ 5 Missing and 4 partials
atexit.py	78.13%	⚠️ 7 Missing and 2 partials
__init__.py	72.41%	⚠️ 8 Missing and 1 partials
_types.py	76.67%	⚠️ 7 Missing and 1 partials
excepthook.py	90.63%	⚠️ 3 Missing and 4 partials
handoff.py	0.00%	⚠️ 7 Missing
consts.py	0.00%	⚠️ 7 Missing
rb.py	30.00%	⚠️ 7 Missing
_openai_responses_api.py	33.33%	⚠️ 6 Missing
_log_batcher.py	90.48%	⚠️ 2 Missing and 2 partials
dedupe.py	95.00%	⚠️ 2 Missing and 2 partials
consts.py	0.00%	⚠️ 4 Missing
__init__.py	20.00%	⚠️ 4 Missing
__init__.py	0.00%	⚠️ 4 Missing
feature_flags.py	93.75%	⚠️ 2 Missing and 1 partials
serverless.py	94.44%	⚠️ 2 Missing and 1 partials
socket.py	96.00%	⚠️ 2 Missing and 1 partials
stdlib.py	100.00%	⚠️ 3 partials
__init__.py	0.00%	⚠️ 3 Missing
__init__.py	40.00%	⚠️ 3 Missing
__init__.py	0.00%	⚠️ 3 Missing
__init__.py	0.00%	⚠️ 3 Missing
__init__.py	0.00%	⚠️ 3 Missing
_compat.py	94.87%	⚠️ 2 Missing
consts.py	99.16%	⚠️ 2 Missing
debug.py	95.65%	⚠️ 1 Missing and 1 partials
scrubber.py	100.00%	⚠️ 2 partials
agent_workflow.py	71.43%	⚠️ 2 Missing
_init_implementation.py	100.00%	⚠️ 1 partials
_metrics_batcher.py	100.00%	⚠️ 1 partials
argv.py	100.00%	⚠️ 1 partials
consts.py	0.00%	⚠️ 1 Missing
consts.py	0.00%	⚠️ 1 Missing

---

Generated by Codecov Action

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}