async generators

iliakan · iliakan · commit 136933266106 · 2019-03-02T23:36:53.000+03:00
diff --git a/1-js/06-advanced-functions/13-generators/article.md b/1-js/06-advanced-functions/13-generators/article.md
@@ -152,7 +152,7 @@ alert(sequence); // 0, 1, 2, 3
 
 In the code above, `...generateSequence()` turns the iterable into array of items (read more about the spread operator in the chapter [](info:rest-parameters-spread-operator#spread-operator))
 
-## Converting iterables to generators
+## Using generators instead of iterables
 
 Some time ago, in the chapter [](info:iterable) we created an iterable `range` object that returns values `from..to`.
 
@@ -203,6 +203,8 @@ alert(sequence); // 1, 2, 3, 4, 5
 
 ...But what if we'd like to keep a custom `range` object?
 
+## Converting Symbol.iterator to generator
+
 We can get the best from both worlds by providing a generator as `Symbol.iterator`:
 
 ```js run
@@ -220,7 +222,16 @@ let range = {
 alert( [...range] ); // 1,2,3,4,5
 ```
 
-The `range` object is now iterable. The last variant with a generator is much more concise than the original iterable code, and keeps the same functionality.
+The `range` object is now iterable.
+
+That works pretty well, because when `range[Symbol.iterator]` is called:
+- it returns an object (now a generator)
+- that has `.next()` method (yep, a generator has it)
+- that returns values in the form `{value: ..., done: true/false}` (check, exactly what generator does).
+
+That's not a coincidence, of course. Generators aim to make iterables easier, so we can see that.
+
+The last variant with a generator is much more concise than the original iterable code, and keeps the same functionality.
 
 ```smart header="Generators may continue forever"
 In the examples above we generated finite sequences, but we can also make a generator that yields values forever. For instance, an unending sequence of pseudo-random numbers.
diff --git a/1-js/09-async/07-async-iteration-generators/article.md b/1-js/09-async/07-async-iteration-generators/article.md
@@ -1,20 +1,297 @@
 
 # Async iteration and generators
 
-In web-programming, we often need to work with fragmented data that comes piece-by-piece.
+Asynchronous iterators allow to iterate over data that comes asynchronously, on-demand.
 
-That happens when we upload or download a file. Or we need to fetch paginated data.
+For instance, when we download something chunk-by-chunk, or just expect a events to come asynchronously and would like to iterate over that -- async iterators and generators may come in handy.
 
+Let's see a simple example first, to grasp the syntax, and then build a real-life example.
 
-For
-Either we need to download or upload something, or we need
+## Async iterators
 
-In the previous chapter we saw how `async/await` allows to write asynchronous code.
+Asynchronous iterators are totally similar to regular iterators, with a few syntactic differences.
 
-But they don't solve
+Here's a small cheatsheet:
 
+|       | Iterators | Async iterators |
+|-------|-----------|-----------------|
+| Object method to provide iteraterable | `Symbol.iterator` | `Symbol.asyncIterator` |
+| `next()` return value is              | any value         | `Promise`  |
+| to loop, use                          | `for..of`         | `for await..of` |
 
+Let's make an iterable `range` object, like we did in the chapter [](info:iterable), but now it will return values asynchronously, one per second:
 
-Regular iterators work fine with the data that doesn't take time to generate.
+```js run
+let range = {
+  from: 1,
+  to: 5,
 
-A `for..of` loop assumes that we
+  // for await..of calls this method once in the very beginning
+*!*
+  [Symbol.asyncIterator]() { // (1)
+*/!*
+    // ...it returns the iterator object:
+    // onward, for await..of works only with that object, asking it for next values
+    return {
+      current: this.from,
+      last: this.to,
+
+      // next() is called on each iteration by the for..of loop
+*!*
+      async next() { // (2)
+        // it should return the value as an object {done:.., value :...}
+        // (automatically wrapped into a promise by async)
+*/!*
+
+        // can use await inside, do async stuff:
+        await new Promise(resolve => setTimeout(resolve, 1000)); // (4)
+
+        if (this.current <= this.last) {
+          return { done: false, value: this.current++ };
+        } else {
+          return { done: true };
+        }
+      }
+    };
+  }
+};
+
+(async () => {
+
+*!*
+  for await (let value of range) { // (3)
+    alert(value); // 1,2,3,4,5
+  }
+*/!*
+
+})()
+```
+
+As we can see, the components are similar to regular iterators:
+
+1. To make an object asynchronously iterable, it must have a method `Symbol.asyncIterator` `(1)`.
+2. To iterate, we use `for await(let value of range)` `(3)`, namely add "await" after "for".
+3. It calls `range[Symbol.asyncIterator]` and expects it to return an object with `next` `(3)` method returning a promise.
+4. Then `next()` is called to obtain values. In that example values come with a delay of 1 second `(4)`.
+
+````warn header="A spread operator doesn't work asynchronously"
+Features that require regular, synchronous iterators, don't work with asynchronous ones.
+
+For instance, a spread operator won't work:
+```js
+alert( [...range] ); // Error, no Symbol.iterator
+```
+
+That's natural, as it expects to find `Symbol.iterator`, and there's none.
+````
+
+## Async generators
+
+Javascript also provides generators, that are also iterable.
+
+Let's recall a sequence generator from the chapter [](info:generators):
+
+```js run
+function* generateSequence(start, end) {
+  for (let i = start; i <= end; i++) {
+    yield i;
+  }
+}
+
+for(let value of generateSequence(1, 5)) {
+  alert(value); // 1, then 2, then 3, then 4, then 5
+}
+```
+
+Normally, we can't use `await` in generators. But what if we need to?
+
+No problem, let's make an async generator, like this:
+
+```js run
+*!*async*/!* function* generateSequence(start, end) {
+
+  for (let i = start; i <= end; i++) {
+
+*!*
+    // yay, can use await!
+    await new Promise(resolve => setTimeout(resolve, 1000));
+*/!*
+
+    yield i;
+  }
+
+}
+
+(async () => {
+
+  let generator = generateSequence(1, 5);
+  for *!*await*/!* (let value of generator) {
+    alert(value); // 1, then 2, then 3, then 4, then 5
+  }
+
+})();
+```
+
+So simple, a little bit magical. We add the `async` keyword, and the generator now can use `await` inside of it, rely on promises and other async functions.
+
+Technically, the difference of such generator is that `generator.next()` method is now asynchronous.
+
+Instead of `result = generator.next()` for a regular, non-async generator, values can be obtained like this:
+
+```js
+result = await generator.next(); // result = {value: ..., done: true/false}
+```
+
+## Iterables via async generators
+
+When we'd like to make an object iterable, we should add `Symbol.iterator` to it. A common practice is to implement it via a generator, just because that's simple.
+
+Let's recall an example from the chapter [](info:generators):
+
+```js run
+let range = {
+  from: 1,
+  to: 5,
+
+  *[Symbol.iterator]() { // a shorthand for [Symbol.iterator]: function*()
+    for(let value = this.from; value <= this.to; value++) {
+      yield value;
+    }
+  }
+};
+
+for(let value of range) {
+  alert(value); // 1, then 2, then 3, then 4, then 5
+}
+```
+
+Here a custom object `range` was made iterable, the generator `*[Symbol.iterator]` implements the logic behind listing values.
+
+We can make it iterable asynchronously by replacing `Symbol.iterator` with `Symbol.asyncIterator` and marking it `async`:
+
+```js run
+let range = {
+  from: 1,
+  to: 5,
+
+*!*
+  async *[Symbol.asyncIterator]() { // same as [Symbol.asyncIterator]: async function*()
+*/!*
+    for(let value = this.from; value <= this.to; value++) {
+
+      // make a pause between values, wait for something  
+      await new Promise(resolve => setTimeout(resolve, 1000));
+
+      yield value;
+    }
+  }
+};
+
+(async () => {
+
+  for *!*await*/!* (let value of range) {
+    alert(value); // 1, then 2, then 3, then 4, then 5
+  }
+
+})();
+```
+
+## Real-life example
+
+There are many online APIs that deliver paginated data. For instance, when we need a list of users, then we can fetch it page-by-page: a request returns a pre-defined count (e.g. 100 users), and provides an URL to the next page.
+
+The pattern is very common, it's not about users, but just about anything. For instance, Github allows to retrieve commits in the same, paginated fasion:
+
+- We should make a request to URL in the form `https://api.github.com/repos/(repo)/commits`.
+- It responds with a JSON of 30 commits, and also provides a link to the next page in the `Link` header.
+- Then we can use it for the next request, if need more commits, and so on.
+
+What we'd like to have is an iterable source of commits, so that we could do it like this:
+
+```js
+let repo = 'iliakan/javascript-tutorial-en'; // Github repository to get commits from
+
+for await (let commit of fetchCommits(repo)) {
+  // process commit
+}
+```
+
+We'd like `fetchCommits` to get commits for us, making requests whenever needed. And let it care about all pagination stuff.
+
+With async generators that's pretty easy to implement:
+
+```js
+async function* fetchCommits(repo) {
+  let url = `https://api.github.com/repos/${repo}/commits`;
+
+  while (url) {
+    const response = await fetch(url, { // (1)
+      headers: {'User-Agent': 'Our script'}, // github requires user-agent header
+    });
+
+    const body = await response.json(); // (2) parses response as JSON (array of commits)
+
+    // (3) the URL of the next page is in the headers, extract it
+    let nextPage = response.headers.get('Link').match(/<(.*?)>; rel="next"/);
+    nextPage = nextPage && nextPage[1];
+
+    url = nextPage;
+
+    for(let commit of body) { // (4) yield commits one by one, until the page ends
+      yield commit;
+    }
+  }
+}
+```
+
+1. We use the browser `fetch` method to download from a remote URL. It allows to supply authorization and other headers if needed (Github requires User-Agent).
+2. The result is parsed as JSON.
+3. ...And we extract the next page URL from the `Link` header of the response. It has a special format, so we use a regexp for that. The next page URL may look like this: `https://api.github.com/repositories/93253246/commits?page=2`, it's generatd by Github itself.
+4. Then we yield all commits received, and when they finish -- the next `while(url)` iteration will trigger, making one more request.
+
+An example of use (shows commit authors in console):
+
+```js run
+(async () => {
+
+  let count = 0;
+
+  for await (const commit of fetchCommits('iliakan/javascript-tutorial-en')) {
+
+    console.log(commit.author.login);
+
+    if (++count == 100) { // let's stop at 100 commits
+      break;
+    }
+  }
+
+})();
+```
+
+The internal mechanics is invisible from the outside. What we see -- is an async generator that returns commits, just what we need.
+
+## Summary
+
+Regular iterators and generators work fine with the data that doesn't take time to generate.
+
+When we expect the data to come asynchronously, with delays, their async counterparts can be used, and `for await..of` instead of `for..of`.
+
+Syntax differences between async and regular iterators:
+
+|       | Iterators | Async iterators |
+|-------|-----------|-----------------|
+| Object method to provide iteraterable | `Symbol.iterator` | `Symbol.asyncIterator` |
+| `next()` return value is              | any value         | `Promise`  |
+
+Syntax differences between async and regular generators:
+
+|       | Generators | Async generators |
+|-------|-----------|-----------------|
+| Declaration | `function*` | `async function*` |
+| `generator.next()` returns              | `{value:…, done: true/false}`         | `Promise` that resolves to `{value:…, done: true/false}`  |
+
+In web-development we often meet streams of data, when it flows chunk-by-chunk. For instance, downloading or uploading a big file.
+
+We could use async generators to process such data, but there's also another API called Streams, that may be more convenient. It's not a part of Javascript language standard though.
+
+Streams and async generators complement each other, both are great ways to handle async data flows.
diff --git a/1-js/09-async/07-async-iteration-generators/head.html b/1-js/09-async/07-async-iteration-generators/head.html
@@ -0,0 +1,24 @@
+<script>
+  async function* fetchCommits(repo) {
+    let url = `https://api.github.com/repos/${repo}/commits`;
+
+    while (url) {
+      const response = await fetch(url, {
+        headers: {'User-Agent': 'Our script'}, // github requires user-agent header
+      });
+
+      const body = await response.json(); // parses response as JSON (array of commits)
+
+      // the URL of the next page is in the headers, extract it
+      let nextPage = response.headers.get('Link').match(/<(.*?)>; rel="next"/);
+      nextPage = nextPage && nextPage[1];
+
+      url = nextPage;
+
+      // yield commits one by one, when they finish - fetch a new page url
+      for(let commit of body) {
+        yield commit;
+      }
+    }
+  }
+</script>
