You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This gives further details on the client-server protocol messages.
1
+
This document gives further details on the client-server protocol messages.
2
2
3
3
The stream client-server protocol messages are $\texttt{Start}$, $\texttt{StartBookmark}$, $\texttt{Stop}$, $\texttt{Header}$, $\texttt{Entry}$ and $\texttt{Bookmark}$.
4
4
@@ -8,7 +8,7 @@ The stream client-server protocol messages are $\texttt{Start}$, $\texttt{StartB
8
8
9
9
If the stream client knows the entry number at which the stream should start, it sends a $\texttt{Start}$ message with that particular entry number. That is, $\mathtt{entryNumber} \not= \texttt{0}$.
10
10
11
-
- $\texttt{StartBookmark}$ message is the type of message the stream client can send if the stream client does not the know entry number, but knows something more meaningful to the application, like a bookmark.
11
+
- $\texttt{StartBookmark}$ message is the type of message the stream client can send if the stream client does not know the entry number, but knows something more meaningful to the application, like a bookmark.
12
12
13
13
In the case of the Polygon zkEVM, if the stream client wants to receive information from a certain L2 block number, then it provides the appropriate bookmark by sending a $\texttt{StartBookmark}$ message.
14
14
@@ -31,37 +31,3 @@ The stream client-server protocol messages are $\texttt{Start}$, $\texttt{StartB
31
31
- $\texttt{Bookmark}$ message is a message in which a stream client sends a bookmark to the stream server, and the stream server in turn tells the stream client what entry number is linked to the bookmark.
32
32
33
33
The above six messages are all messages used in the client-server protocol. And they can be found [here](https://github.com/0xPolygonHermez/zkevm-data-streamer#stream-tcp-commands).
34
-
35
-
## Stream server-source library
36
-
37
-
Interaction between the stream source and each stream server is enabled by the Server-source library, which is a Go library with six main functions for modifying or adding entries to operations.
38
-
39
-
### Send data functions
40
-
41
-
When each of these functions is called, a corresponding message is generated and sent to the stream server:
42
-
43
-
1. $\texttt{StartAtomicOp}(\ )$ starts an atomic operation. When called, a message that amounts to saying: "start an atomic operation," is generated and sent from the stream source to the stream server.
44
-
2. $\texttt{AddStreamEntry(u32 entryType, u8[] data)}$ adds an entry to the atomic operation and returns an $\texttt{u64 entryNumber}$. When called, a message equivalent to saying: "Add an entry of this type, with this data, to the current atomic operation," is generated and sent to the stream server.
45
-
3. $\texttt{AddStreamBookmark(u8[] bookmark)}$ adds an entry to the atomic operation and returns an $\texttt{u64}$ $\texttt{entryNumber}$.
46
-
4. $\texttt{CommitAtomicOp}(\ )$ commits an operation $\texttt{Op}$ so that its entries can be sent to stream clients. When called, a message which is tantamount to saying: "All entries associated with the current operation have been sent, the operation ends with the last sent entry," is generated and sent to the stream server.
47
-
5. $\texttt{RollbackAtomicOp}(\ )$ rolls back an atomic operation.
48
-
6. $\texttt{UpdateEntryData(u64 entryNumber, u32 entryType, u8[] newData)}$ updates an existing entry. This function only applies to entries for which the atomic operation has not been committed.
49
-
50
-
51
-
52
-
### Query data functions
53
-
54
-
The stream source can use a few more functions of the stream server-source library, to get information from the stream server.
55
-
56
-
It uses the following functions:
57
-
58
-
- $\texttt{GetHeader()}$: The stream source uses this function to query the header of a particular entry. The function returns, $\texttt{struct HeaderEntry}$.
59
-
- $\texttt{GetEntry(u64 entryNumber)}$: This function is used to get an entry that corresponds to a given entry number. It returns, $\texttt{struct FileEntry}$.
60
-
- $\texttt{GetBookmark(u8[] bookmark)}$: The stream source uses this function to get a bookmark. The function returns, $\texttt{u64 entryNumber}$.
61
-
- $\texttt{GetFirstEventAfterBookmark(u8[] bookmark)}$: This function is used to get the first entry after a given bookmark. It returns, $\texttt{struct FileEntry}$.
62
-
63
-
64
-
65
-
The complete stream source-server library is described, but referred to as the DATA STREAMER INTERFACE (API), [here](https://github.com/0xPolygonHermez/zkevm-data-streamer#data-streamer-interface-api).
66
-
67
-
It's possible to create, using the stream source-server library, a stream source that connects with a server, opens and commits operations.
Copy file name to clipboardExpand all lines: docs/zkEVM/architecture/data-streamer/how-rollbacks-work.md
+2-4Lines changed: 2 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,13 +10,11 @@ Once all entries have been sent, the stream source calls the $\texttt{CommitAtom
10
10
11
11
But if the $\texttt{RollbackAtomicOp}(\ )$ is triggered instead of the $\texttt{CommitAtomicOp}(\ )$, the header is not updated.
12
12
13
-
In other words, the header of the stream file only changes when the $\texttt{CommitAtomicOp}(\ )$ function is called. So, although some entries related to the atomic operation have already been added to the stream file, the header of the stream file is updated only with information of entries related to committed atomic operations.
13
+
In other words, the header of the stream file is updated only when the $\texttt{CommitAtomicOp}(\ )$ function is called. So, although some entries related to the atomic operation have already been added to the stream file, the header of the stream file is updated only with information of entries related to committed atomic operations.
14
14
15
15
Since the $\texttt{RollbackAtomicOp}(\ )$ function can only be executed before a given atomic operation is committed, the header is not updated because the added entries (of the uncommitted atomic operation) will be overwritten with entries of the next atomic operation(s).
16
16
17
-
This means a rollback amounts to overwriting entries in the strem file that are related to an uncommitted atomic operation.
18
-
19
-
17
+
This means a rollback amounts to overwriting entries in the stream file that are related to an uncommitted atomic operation.
Interaction between the stream source and each stream server is enabled by the [`server-source`](https://github.com/0xPolygonHermez/zkevm-data-streamer#data-streamer-interface-api) library, which is a Go library with six main functions for modifying or adding entries to operations.
2
+
3
+
### Send data functions
4
+
5
+
When each of these functions is called, a corresponding message is generated and sent to the stream server:
6
+
7
+
1. $\texttt{StartAtomicOp}(\ )$ starts an atomic operation. When called, a message that amounts to saying: "start an atomic operation," is generated and sent from the stream source to the stream server.
8
+
2. $\texttt{AddStreamEntry(u32 entryType, u8[] data)}$ adds an entry to the atomic operation and returns an $\texttt{u64 entryNumber}$. When called, a message equivalent to saying: "Add an entry of this type, with this data, to the current atomic operation," is generated and sent to the stream server.
9
+
3. $\texttt{AddStreamBookmark(u8[] bookmark)}$ adds an entry to the atomic operation and returns an $\texttt{u64}$ $\texttt{entryNumber}$.
10
+
4. $\texttt{CommitAtomicOp}(\ )$ commits an operation $\texttt{Op}$ so that its entries can be sent to stream clients. When called, a message which is tantamount to saying: "All entries associated with the current operation have been sent, the operation ends with the last sent entry," is generated and sent to the stream server.
11
+
5. $\texttt{RollbackAtomicOp}(\ )$ rolls back an atomic operation.
12
+
6. $\texttt{UpdateEntryData(u64 entryNumber, u32 entryType, u8[] newData)}$ updates an existing entry. This function only applies to entries for which the atomic operation has not been committed.
13
+
14
+
### Query data functions
15
+
16
+
The stream source uses the following functions of the stream server-source library to get information from the stream server.
17
+
18
+
19
+
- $\texttt{GetHeader()}$: The stream source uses this function to query the header of a particular entry. The function returns, $\texttt{struct HeaderEntry}$.
20
+
- $\texttt{GetEntry(u64 entryNumber)}$: This function is used to get an entry that corresponds to a given entry number. It returns, $\texttt{struct FileEntry}$.
21
+
- $\texttt{GetBookmark(u8[] bookmark)}$: The stream source uses this function to get a bookmark. The function returns, $\texttt{u64 entryNumber}$.
22
+
- $\texttt{GetFirstEventAfterBookmark(u8[] bookmark)}$: This function is used to get the first entry after a given bookmark. It returns, $\texttt{struct FileEntry}$.
23
+
24
+
!!! tip
25
+
26
+
Find out more about the [DATA STREAMER INTERFACE (API)](https://github.com/0xPolygonHermez/zkevm-data-streamer#data-streamer-interface-api).
27
+
28
+
It's possible to create, using the stream source-server library, a stream source that connects with a server, opens and commits operations.
Copy file name to clipboardExpand all lines: docs/zkEVM/architecture/data-streamer/stream-file.md
+10-12Lines changed: 10 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,13 +23,11 @@ The $\texttt{HeaderEntry}$ consists of the following data; $\texttt{magicNumbers
23
23
\texttt{u8[16] magicNumbers}
24
24
$$
25
25
26
-
The $\texttt{magicNumbers}$ identify which application to which the data in the stream file belongs.
26
+
The $\texttt{magicNumbers}$ identify the application to which the data in the stream file belongs.
27
27
28
-
2. The $\texttt{magicNumbers}$ identify which application to which the data in the stream file belongs.
29
-
30
28
In the Polygon zkEVM case, the $\texttt{magicNumbers}$ is the ASCII-encoding of these sixteen (16) characters: $\texttt{polygonDATSTREAM}$.
31
29
32
-
3. After the $\texttt{magicNumbers}$ comes the $\texttt{packetType}$, which indicates whether the current page is a $\texttt{Header}$ page or a $\texttt{Data}$ page.
30
+
2. After the $\texttt{magicNumbers}$ comes the $\texttt{packetType}$, which indicates whether the current page is a $\texttt{Header}$ page or a $\texttt{Data}$ page.
33
31
34
32
$$
35
33
\texttt{u8 packetType = 1 // 1: Header entry}
@@ -39,21 +37,21 @@ The $\texttt{HeaderEntry}$ consists of the following data; $\texttt{magicNumbers
4. Included in the $\texttt{Header}$ page is the $\texttt{streamType}$, which has the same meaning as seen in the Server-source protocol: It indicates the application, or in particular, the stream source node to which the stream server should connect.
40
+
3. Included in the $\texttt{Header}$ page is the $\texttt{streamType}$, which has the same meaning as seen in the Server-source protocol: It indicates the application, or in particular, the stream source node to which the stream server should connect.
43
41
44
42
$$
45
43
\texttt{u64 streamType // 1: zkEVM Sequencer}
46
44
$$
47
45
48
46
As mentioned in the above line of code, $\texttt{streamType = 1}$ means the stream source node is the zkEVM Sequencer.
49
47
50
-
5. The $\texttt{streamType}$ is then followed by the $\texttt{TotalLength}$ , which is the total number of bytes used in the stream file.
48
+
4. The $\texttt{streamType}$ is then followed by the $\texttt{TotalLength}$ , which is the total number of bytes used in the stream file.
51
49
52
50
$$
53
51
\texttt{u64 TotalLength // Total bytes used in the file}
54
52
$$
55
53
56
-
6. After the $\texttt{TotalLength}$ is the $\texttt{TotalEntries}$, which is the total number of entries used in the file.
54
+
5. After the $\texttt{TotalLength}$ is the $\texttt{TotalEntries}$, which is the total number of entries used in the file.
57
55
58
56
$$
59
57
\texttt{u64 TotalEntries // Total number of data entries}
4.\quad &\texttt{u64 entryNumber // Entry number (sequence from 0)} \\
76
74
5.\quad &\texttt{u8[] data}
77
75
\end{aligned}
78
76
$$
79
77
80
-
The $\texttt{packetType}$ is followed by the $\texttt{Length}$ of the data entry, then the $\texttt{Type}$ of the entry. That is, whether it is a bookmark or an event entry.
78
+
The $\texttt{packetType}$ is followed by the $\texttt{Length}$ of the data entry, then the $\texttt{entryType}$ of the entry. That is, whether it is a bookmark or an event entry.
81
79
82
-
A bookmark $\texttt{Type}$ is indicated by $\texttt{0xb0}$, while each event's $\texttt{Type}$ is its position among a sequence of events. That is, each $i$-th event is of $\mathtt{Type = i}$.
80
+
A bookmark $\texttt{entryType}$ is indicated by $\texttt{0xb0}$, while each event's $\texttt{entryType}$ is its position among a sequence of events. That is, each $i$-th event is of $\mathtt{entryType = i}$.
83
81
84
-
The next value after the $\texttt{Type}$ is the entry number, denoted by $\texttt{Number}$. The next values in a data page are $\texttt{data}$.
82
+
The next value after the $\texttt{entryType}$ is the entry number, denoted by $\texttt{entryNumber}$. The next values in a data page are $\texttt{data}$.
85
83
86
84
After the last entry in a data page, is the $\texttt{packetType = 0}$ and some padding for any unused space.
0 commit comments