Case study blog of DeepFabric using spin #163
Conversation
itowlson
left a comment
itowlson
left a comment
There was a problem hiding this comment.
Thanks for this @lukehinds - a good read and an intriguing use case! I noted a couple of possible typos or things to check, but otherwise looks good! If the things I've flagged are non-concerns then let me know and we get get this out there!
| author = "Luke Hinds" | ||
| --- | ||
|
|
||
| As the world exhausts its supply of original training data, synthetic data has become not just useful but necessary for continued AI training. Yet this shift to synthetics brings it's challenges - particularly for training models to effectively use Tools and conform with structured schema output. When both Tool calls and their responses are generated by an LLM, the resulting models consistently underperform against real systems. They struggle with error recovery, mishandle state dependencies, and often exhibit what we call "time travel" errors: acting on information they haven't actually received yet (e.g., skipping verification steps because they "know" a file exists). |
There was a problem hiding this comment.
typo fix
| As the world exhausts its supply of original training data, synthetic data has become not just useful but necessary for continued AI training. Yet this shift to synthetics brings it's challenges - particularly for training models to effectively use Tools and conform with structured schema output. When both Tool calls and their responses are generated by an LLM, the resulting models consistently underperform against real systems. They struggle with error recovery, mishandle state dependencies, and often exhibit what we call "time travel" errors: acting on information they haven't actually received yet (e.g., skipping verification steps because they "know" a file exists). | |
| As the world exhausts its supply of original training data, synthetic data has become not just useful but necessary for continued AI training. Yet this shift to synthetics brings its challenges - particularly for training models to effectively use tools and conform with structured schema output. When both tool calls and their responses are generated by an LLM, the resulting models consistently underperform against real systems. They struggle with error recovery, mishandle state dependencies, and often exhibit what we call "time travel" errors: acting on information they haven't actually received yet (e.g., skipping verification steps because they "know" a file exists). |
(I wasn't sure about the "tool" ones in case the capital was a reference to a specific role/model. But the rest of the post lower-cases it so I bet on typo grin - if that's wrong then please ignore.)
content/blog/deepfabric-spin.md
Outdated
|
|
||
| | Property | Docker | WebAssembly | | ||
| |----------|--------|-------------| | ||
| | Filesystem access | Must explicitly restrict | Denied by default | |
There was a problem hiding this comment.
Are the Docker filesystem and network lines in this correct? My vague impression of Docker is that guests do not have access to the host filesystem or network by default. But maybe I'm wrong? And/or misunderstanding what the table is saying?
There was a problem hiding this comment.
You're right lean into this on this. My position was thinking webassembly is equivalent to a container having the --read-only flag and then adding specific writable mounts can be added for persistence, but its bit more nuanced with that.
For the network, docker has outbound (egress) container traffic allowed by default. Egress connections initiated by containers are masqueraded/SNATed to an ephemeral port, but again - its kind of nuanced (I might be out of date in my knowledge of the domain) and I don't want to start some war on hackernews :)
I have reduced to the following:
Outbound network access | Allowed by default | Denied by default
content/blog/deepfabric-spin.md
Outdated
|
|
||
| The sandbox rejection isn't just a safety feature - it's valuable training data. Models learn that certain paths are off-limits and how to recover when access is denied and find a more appropriate method to achieve their goals. | ||
|
|
||
| This becomes more pressing as we start to witness SOTA frontier models exhibiting dangerous behaviors, such as attempting to delete a users entire home directory when given file system access. WebAssembly's default-deny posture ensures that any such attempts are safely blocked, while also providing informative error feedback for training. |
There was a problem hiding this comment.
SOTA = state-of-the-art? Consider spelling it out?
content/blog/deepfabric-spin.md
Outdated
|
|
||
| ### Polyglot Language Support | ||
|
|
||
| And of course, this being webassembly, you can build components in any language that compiles to Wasm, including Javascript, Go, and Python. The Spin framework handles the HTTP interface and capability restrictions uniformly across languages. |
There was a problem hiding this comment.
| And of course, this being webassembly, you can build components in any language that compiles to Wasm, including Javascript, Go, and Python. The Spin framework handles the HTTP interface and capability restrictions uniformly across languages. | |
| And of course, this being WebAssembly, you can build components in any language that compiles to Wasm, including Javascript, Go, and Python. The Spin framework handles the HTTP interface and capability restrictions uniformly across languages. |
content/blog/deepfabric-spin.md
Outdated
| @@ -0,0 +1,410 @@ | |||
| title = "DeepFabric and Spin: A Case Study in Building Better Agentic Training Data" | |||
| date = "2025-12-27T14:23:15Z" | |||
There was a problem hiding this comment.
As we've taken a while to review this (sorry - holidays), do you want to bump the date forward? (This will show as the article date now.)
|
Thanks @itowlson , solid review - let me fix these up and thank for investing your time to get it in order! |
|
@lukehinds just spotted this, sorry - all commits need to have verified GPG signatures - hope that's not a problem! |
|
@itowlson ahh, signed with my own key? I have not used gpg in quite ahile, let me see if I can get it set up. Does my key need to be on keyserver? |
|
It doesn't need to be on a key server but it does need to be added to your GH account: https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account. I hope it's not too much of a faff. |
lukehinds
force-pushed
the
deepfabric-post
branch
from
57eb0ba to
3edabda
Compare
|
should be good! 
|
|
GH is still squawking I'm afraid - you'll either need to amend the previous commits too, or to squash. Sorry about this. 
|
|
and squash and force push to the branch if that works ok? |
|
Ah, yep, you would need to force push the GPG-signed squashed commit. Thanks! |
As discussed, a blog providing an overview of how we use spin to allow real tool executions within dataset traces Signed-off-by: Luke Hinds <lukehinds@gmail.com>
lukehinds
force-pushed
the
deepfabric-post
branch
from
3edabda to
d3ef0a8
Compare
|
That should as good as gravy on chips now @itowlson , thanks for your assistance 👍 |
itowlson
left a comment
itowlson
left a comment
There was a problem hiding this comment.
"Good as gravy on chips" is my kind of simile. Thanks for persevering @lukehinds!
|
@lukehinds All right, the post is now up at https://spinframework.dev/blog/deepfabric-spin if you want to share it - thanks again for all your effort on this! |
2 participants
As discussed, a blog providing an overview of how we use spin to allow real tool executions within dataset traces.