|
1 | | -# Tiered Sparse Merkle Tree (TSMT) |
2 | | -A sparse Merkle tree (SMT) is a cryptographic data structure that allows authenticated manipulations of a key-value store, i.e. a dictionary. It is a Merkle tree because it is |
3 | | -a binary tree in which each leaf node represents a key-value pair, and each non-leaf node represents the hash of the concatenation of its two child nodes. The index of the |
4 | | -leaf node in the tree corresponds to the key and the data stored in this node corresponds to the value. This implies that the depth of such a tree is logarithmic in the size of the key space, which implies that, for most key spaces used in practice, such a tree is impractical if done naively. However, the tree is sparse, i.e. it is going to be made up of mostly empty leaves, which in turn implies that most sub-trees are empty with roots that can be pre-computed. |
5 | | -A major issue with SMTs is the fact that operations scale with the size of the key space, meaning, for example, that if our key space has size $~ 2^{256}$, then performing any operation on the key-value store, for example, an update, will require performing $256$ hashing operations. This is undesirable both inside as well as outside the STARK. |
| 1 | +A sparse Merkle tree (SMT) is a cryptographic data structure that allows authenticated manipulations of a key-value store, i.e. a dictionary. It is a Merkle tree because it is a binary tree in which each leaf node represents a key-value pair, and each non-leaf node represents the hash of the concatenation of its two child nodes. |
6 | 2 |
|
| 3 | +The index of the leaf node in the tree corresponds to the key and the data stored in this node corresponds to the value. This implies that the depth of such a tree is logarithmic in the size of the key space, which implies that, for most key spaces used in practice, such a tree is impractical if done naively. However, the tree is sparse, i.e. it is going to be made up of mostly empty leaves, which in turn implies that most sub-trees are empty with roots that can be pre-computed. |
7 | 4 |
|
8 | | -A common solution to this issue is to keep only non-empty leaves in the tree, which in practice means that sub-trees with only one non-empty child are replaced with the sub-tree rooted at that non-empty child. Here's an example of this: |
| 5 | +A major issue with SMTs is the fact that operations scale with the size of the key space, meaning, for example, that if our key space has size $~ 2^{256}$, then performing any operation on the key-value store, for example, an update, will require performing $256$ hashing operations. This is undesirable both inside as well as outside the STARK. |
9 | 6 |
|
10 | | -<p align="center"> |
11 | | - <img src="./diagram/../../diagrams/crypto-primitives/tsmt/tsmt_compaction_example.svg"> |
12 | | -</p> |
| 7 | +A common solution to this issue is to keep only non-empty leaves in the tree, which in practice means that sub-trees with only one non-empty child are replaced with the sub-tree rooted at that non-empty child. Here's an example of this: |
13 | 8 |
|
| 9 | +<center> |
| 10 | +{ width="80%" } |
| 11 | +</center> |
14 | 12 |
|
15 | 13 | Compaction implies then that the tree depth is logarithmic in the size of the dictionary entries as opposed to the size of the key space. This, in turn, means that the number of hashing required for executing operations on the key-value map scales logarithmically in the size of the map. |
16 | 14 |
|
17 | | -The first documented implementation of a compact SMT was described [here](https://github.com/proofchains/python-proofmarshal/blob/master/proofmarshal/merbinnertree.py). This implementation relies on augmenting nodes with |
18 | | -a field called `prefix` which is the longest common prefix of all childs of a given node. This means that internal nodes are hashes of a left and right child together with `prefix`, i.e. `(LEFT || RIGHT || PREFIX)`. Leaf nodes are hashes of just the key-value pair. In addition, domain separation is used to distinguish |
19 | | -between an internal and a leaf node by appending a `0` or `1` before hashing. |
| 15 | +The first documented implementation of a compact SMT was described [here](https://github.com/proofchains/python-proofmarshal/blob/master/proofmarshal/merbinnertree.py). This implementation relies on augmenting nodes with a field called `prefix` which is the longest common prefix of all childs of a given node. This means that internal nodes are hashes of a left and right child together with `prefix`, i.e. `(LEFT || RIGHT || PREFIX)`. Leaf nodes are hashes of just the key-value pair. In addition, domain separation is used to distinguish between an internal and a leaf node by appending a `0` or `1` before hashing. |
| 16 | + |
20 | 17 | A disadvantage of this implementation, as well as all known implementations of compact SMTs, is the reliance of their operations on bit-wise manipulations. This makes them very expensive to implement inside Miden VM. |
21 | 18 |
|
22 | 19 | Tiered sparse Merkle trees (TSMT) are a result of the idea that one can trade hashing for more efficient bit-wise operations. More precisely, in TSMT, compaction happens only at certain pre-specified tiers and this leads to simpler bit-wise manipulations. In the case of Miden VM, compaction happens at depths `16`, `32`, `48` and `64`. To illustrate this, consider the following example where the keys have length 4, i.e. the SMT has depth 4. |
23 | 20 |
|
24 | | -<p align="center"> |
25 | | - <img src="./diagram/../../diagrams/crypto-primitives/tsmt/tsmt_non_compact.svg"> |
26 | | -</p> |
| 21 | +<center> |
| 22 | +{ width="80%" } |
| 23 | +</center> |
27 | 24 |
|
28 | 25 | If we choose the tiers to be at depths 2 and 4 then the compact version of the above tree will look like |
29 | 26 |
|
30 | | -<p align="center"> |
31 | | - <img src="./diagram/../../diagrams/crypto-primitives/tsmt/tsmt_compact_2.svg"> |
32 | | -</p> |
| 27 | +<center> |
| 28 | +{ width="80%" } |
| 29 | +</center> |
33 | 30 |
|
34 | 31 | If, however, we choose compaction at all possible depths we get a compact SMT that is similar to the [Jellyfish SMT](https://developers.diem.com/papers/jellyfish-merkle-tree/2021-01-14.pdf) |
35 | 32 |
|
36 | | -<p align="center"> |
37 | | - <img src="./diagram/../../diagrams/crypto-primitives/tsmt/tsmt_compact_1.svg"> |
38 | | -</p> |
| 33 | +<center> |
| 34 | +{ width="80%" } |
| 35 | +</center> |
39 | 36 |
|
40 | 37 | We can see that a TSMT is a parametrized SMT, where the parameter is the number of tiers, that interpolates between a plain SMT with no compaction and a (fully) compact SMT with compaction allowed at evey depth. |
41 | 38 |
|
42 | | - |
43 | 39 | The Miden VM TSMT is a map from a key-space of size approximately `256`-bit to a value-space of the same size and are represented by four $64$-bit field elements in the Miden VM native field, i.e. a `Word`. Athough keys are `256`-bit long the maximal tree depth that is allowed is `64`. This has several consequences: |
44 | 40 |
|
45 | 41 | 1. Limiting the tree depth to `64` means that we can use one of the four field elements making up the (full) key as the (compact) key in the TSMT. We choose the 4th field element to act as the (compact) key since when loading the (full) key on the operand stack, the 4th field element will be at the tip of the operand stack and thus will be easier to access. Using a single field element to act as the (compact) key implies that we can levarage the native instructions of the Miden VM to manipulate it during the course of operations on the TSMT inside the VM. |
46 | 42 | 2. In order to deal with keys sharing a common prefix of length greater than `64`, we introduce a special type of leaf nodes at depth `64`. These leaf nodes are a sorted list of `(key, value)` pairs where sorting is done using `key`. Notice that all pairs in the sorted list have the same first part of the key, namely they share a common `64`-bit prefix , and that all pairs with the same first part of the key belong in the same sorted list. The decision to go with a sorted list of pairs instead of a standard sub-tree node with multiple childs at depth `80` sharing the same `64`-bit prefix is partially motivated by the fact that, for most scenarios related to Miden VM, the probability that two distinct keys will end up sharing a common prefix of length greater than or equal to `64` is very low, and even feasible grinding attacks can only make the sorted list on slightly longer than a dozen of elements and with very little clear benefits. Thus putting all pairs with the same first part of the key in a sorted list is a good tradeoff for dealing with prefixes of length greater than `64`. |
47 | 43 |
|
48 | 44 | The Miden VM TSMT has three types of nodes: |
49 | | -1. Internal nodes with a left and right child, and value computed as: `Hash(left || right)` |
| 45 | + |
| 46 | +1. Internal nodes with a left and right child, and value computed as: `Hash(left || right)`. |
50 | 47 | 2. Leaf nodes at depths `16`, `32` and `48` with value computed as: `Hash(rem_key || value; domain=depth)` where: |
51 | 48 | 1. `depth` is the depth of the node, i.e. either `16`, `32` or `48`. |
52 | 49 | 2. `rem_key` is the the remaining key after either the `16`, `32` or `48`-bit prefix is removed depending on the depth of the node. |
|
0 commit comments