fix: add zstd dependency for cloudflare compression

[lkstrp]

GitHub uses gzip compression, which works fine with httpx, also in streaming mode. But cloudflare supports zstd compression and uses it currently (see links below), which is not handled in httpx streaming, unless all packages are installed.

This PR adds the required packages.

https://raw.githubusercontent.com/electricitymaps/electricitymaps-contrib/refs/tags/v1.238.0/web/geo/world.geojson
vs
https://data.pypsa.org/workflows/eur/bidding_zones_electricitymaps/v1.238.0/world.geojson

[coroa]

From a very cursory glance at the httpx documentation, i understand, that they do support zstd?

[lkstrp]

I just tested this

# /// script
# requires-python = ">=3.10"
# dependencies = ["httpx"]
# ///
import json
import httpx

github_url = "https://raw.githubusercontent.com/electricitymaps/electricitymaps-contrib/refs/tags/v1.238.0/web/geo/world.geojson"
cloudflare_url = "https://data.pypsa.org/workflows/eur/bidding_zones_electricitymaps/v1.238.0/world.geojson"

headers = {"Accept-Encoding": "zstd, gzip, deflate, br"}


def is_valid_json(content: bytes) -> bool:
    try:
        json.loads(content)
        return True
    except (json.JSONDecodeError, UnicodeDecodeError):
        return False


print("=== GitHub (raw.githubusercontent.com) ===")
with httpx.stream("GET", github_url, headers=headers, follow_redirects=True) as r:
    content = b"".join(r.iter_bytes())
    print(f"Content-Encoding: {r.headers.get('content-encoding')}")
    print(f"Size: {len(content)} bytes")
    print(f"Valid JSON: {is_valid_json(content)}")

print("\n=== Cloudflare (data.pypsa.org) ===")
with httpx.stream("GET", cloudflare_url, headers=headers, follow_redirects=True) as r:
    content = b"".join(r.iter_bytes())
    print(f"Content-Encoding: {r.headers.get('content-encoding')}")
    print(f"Size: {len(content)} bytes")
    print(f"Valid JSON: {is_valid_json(content)}")

=== GitHub (raw.githubusercontent.com) ===
Content-Encoding: gzip
Size: 2568830 bytes
Valid JSON: True

=== Cloudflare (data.pypsa.org) ===
Content-Encoding: zstd
Size: 613275 bytes
Valid JSON: False

[coroa]

Me too,

In [14]: async def get(method, url):
    ...:     limits = httpx.Limits(
    ...:         max_keepalive_connections=3,
    ...:         max_connections=3,
    ...:     )
    ...:     timeout = httpx.Timeout(60, pool=None)
    ...:
    ...:     client = httpx.AsyncClient(
    ...:                 follow_redirects=True, limits=limits, timeout=timeout, headers={"Accept-Encoding": "zs
       ⋮ td"}
    ...:             )
    ...:
    ...:     async with client, client.stream(method, url) as response:
    ...:         async for chunk in response.aiter_bytes(chunk_size=8192):
    ...:             print(chunk)

In [14]: await get("get", "https://data.pypsa.org/workflows/eur/bidding_zones_electricitymaps/v1.238.0/world.ge
       ⋮ ojson")
[...]
b'     [-2.369633951292269, 49.419205668715762],\n              [-2.378574937391834, 49.431501421246644],\n              [-2.371422148512182, 49.434325685908163],\n              [-2.36707938383525, 49.449607120528675],\n              [-2.342555536247872, 49.429341579723243],\n              [-2.365802100106741, 49.420202736416293],\n              [-2.368101210818058, 49.410231147787357],\n              [-2.374743086206306, 49.408402803454798]\n            ]\n          ]\n        ]\n      }\n    },\n    {\n      "type": "Feature",\n      "properties": { "zoneName": "MV", "countryKey": "MV", "countryName": "Maldives" },\n      "geometry": {\n        "type": "MultiPolygon",\n        "coordinates": [\n          [\n            [\n              [73.075312912553684, -0.587802462766317],\n              [73.085378885095025, -0.580116119339015],\n              [73.115942838084194, -0.660090156064093],\n              [73.167187789203808, -0.693946063186671],\n              [73.165174594695529, -0.702364251585757],\n              [73.134610641706345, -0.69229902456802],\n              [73.112282484432811, -0.666312340415798],\n              [73.075312912553684, -0.587802462766317]\n            ]\n          ],\n          [\n            [\n              [73.27672036955785, 0.235225016532299],\n              [73.282614518444376, 0.235157268265302],\n              [73.285730964982065, 0.234208792492863],\n              [73.286543951035384, 0.230414888763887],\n              [73.283587035318263, 0.223321029736189],\n              [73.277736602124477, 0.215239263911558],\n              [73.269064750889157, 0.207990187716998],\n              [73.249824080960764, 0.196405208616789],\n              [73.241084480887665, 0.195185736657237],\n              [73.237832536674418, 0.196879447688173],\n              [73.235664573865591, 0.200402366079728],\n              [73.236477559918896, 0.202434818654149],\n              [73.250840313527405, 0.21158085204982],\n              [73.260121904302721, 0.2208623767236],\n              [73.262560862462664, 0.226011249241283],\n              [73.26655804389145, 0.231024623360328],\n              [73.271232713697998, 0.233463561483765],\n              [73.27672036955785, 0.235225016532299]\n            ]\n          ],\n          [\n            [\n              [73.540229474087326, 1.944324946182143],\n              [73.549307818349334, 1.939856091320918],\n              [73.553812643632796, 1.926670000185863],\n              [73.528556203438129, 1.868868136317417],\n              [73.521937287887809, 1.852778808797704],\n              [73.516923873892367, 1.853455942856341],\n              [73.498767185368408, 1.826505808364157],\n              [73.490230831808631, 1.821088646917177],\n              [73.486301399217623, 1.822984655278469],\n              [73.502832115634973, 1.840996634820411],\n              [73.524511743723323, 1.872077027660738],\n              [73.521192050672298, 1.876478346148723],\n              [73.528576673989889, 1.882369324323815],\n              [73.539171229027204, 1.913134294393391],\n              [73.540229474087326, 1.944324946182143]\n            ]\n          ],\n          [\n            [\n              [73.539811790915977, 4.239072004688713],\n              [73.552806046378464, 4.24016710592131],\n              [73.534138242756327, 4.179751713826575],\n              [73.51760136227017, 4.170421840325189],\n              [73.500596403988396, 4.169070450179473],\n              [73.50296761331056, 4.177516600442289],\n              [73.509674748250376, 4.179881506227228],\n              [73.522005036725616, 4.179881506227228],\n              [73.52645150008837, 4.192893829077663],\n              [73.528237929801008, 4.208732781191149],\n              [73.539811790915977, 4.239072004688713]\n            ]\n          ]\n        ]\n      }\n    },\n    {\n      "type": "Feature",\n      "properties": { "zoneName": "CW", "countryKey": "CW", "countryName": "Cura\xc3\xa7ao" },\n      "geometry": {\n        "type": "MultiPolygon",\n        "coordinates": [\n          [\n            [\n              [-69.151577303171862, 12.389546858796207],\n              [-69.057506214330914, 12.331623667507726],\n              [-68.978076540095557, 12.196062083734544],\n              [-68.83449916811945, 12.166275717998509],\n              [-68.732009265880308, 12.037431473445437],\n              [-68.825714319356109, 12.043159201497788],\n              [-68.982377455635969, 12.130491886315641],\n              [-69.029229982373863, 12.192037094564444],\n              [-69.073154226190624, 12.190605972580302],\n              [-69.109025691974352, 12.24999103497592],\n              [-69.155146147981952, 12.286474028380409],\n              [-69.151577303171862, 12.389546858796207]\n            ]\n          ]\n        ]\n      }\n    }\n  ]\n}\n'

[coroa]

Me, too

But i did it with: ❯ uvx --with "httpx[zstd]" ipython .

Ie we are missing a dependency.

[lkstrp]

But i did it with: ❯ uvx --with "httpx[zstd]" ipython .

Ah, that makes even more sense! Great, that's a better solution and resolves the mystery

I changed the PR