-
Notifications
You must be signed in to change notification settings - Fork 697
Description
Summary
Add a new OpenDAL backend that leverages the wasi:filesystem interface for file operations. This enables OpenDAL to access the host filesystem when running as a WebAssembly component, with the host runtime (e.g., wasmtime, wasmer) providing the actual filesystem implementation.
Motivation
Currently, the services-fs backend cannot compile to wasm32-wasip2 because it depends on tokio's filesystem features, which are not supported on WASM targets.
However, WASM components can access the host filesystem through the standardized wasi:filesystem interface. A wasi-fs backend would:
- Enable filesystem access in WASM components — Components could read/write files preopened by the host runtime
- Complement existing backends —
memoryands3work on WASM, but there's no local filesystem option - Leverage existing WASI ecosystem — Runtimes like wasmtime already implement
wasi:filesystem
Proposed Implementation
A new services-wasi-fs feature/backend that:
- Imports
wasi:filesystem/typesandwasi:filesystem/preopensinterfaces - Uses
wasi:io/streamsfor read/write operations - Works with preopened directories provided by the host
- Compiles to
wasm32-wasip2target only (or conditional compilation)
Use Case
We're building WASM components that expose storage capabilities. We want to:
- Create an OpenDAL-based storage component
- Export
wasi:filesysteminterface for downstream consumers - Support multiple backends (memory, S3, and host filesystem)
Without a wasi-fs backend, components cannot access files that the host runtime preopens for them.
Research Notes
| Backend | WASM-compatible | Notes |
|---|---|---|
services-memory |
✅ | Works out of the box |
services-s3 |
✅ | Works with reqwest/reqsign |
services-fs |
❌ | Requires tokio fs (not WASM-compatible) |
services-wasi-fs |
🆕 | Proposed — uses wasi:filesystem imports |
Related
- WASI filesystem spec: https://github.com/WebAssembly/wasi-filesystem
- Preview 2 (Component Model): https://github.com/WebAssembly/WASI/blob/main/preview2/README.md