Skip to content

Commit ad71f2c

Browse files
authored
Update README.md
1 parent 4d5504f commit ad71f2c

File tree

1 file changed

+27
-7
lines changed

1 file changed

+27
-7
lines changed

examples/RemoteJobs/README.md

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,32 @@
1-
## Communication with remote Jobs using Redis
2-
This example demonstrates how HyperFlow can communicate with remote job executors using Redis.
1+
## Distributed execution of workflow Jobs using Redis
2+
This example explains the distributed execution model of Hyperflow. It demonstrates how HyperFlow can communicate with remote job executors using Redis. It is also useful for testing the implementation of the [Hyperflow job executor](https://github.com/hyperflow-wms/hyperflow-job-executor).
33

4-
- The workflow invokes function `submitRemoteJob` from `functions.js` 100 times. This function simulates submission of jobs by starting 100 parallel processes of `node handler.js <taskId> <redis_url>`.
5-
- `handler.js` represents a remote job executor which is passed two parameters: `taskId` and `redis_url`.
4+
The distributed execution architecture consists of:
5+
1. Master components:
6+
- **The Hyperflow engine** - executes the workflow graph; for each workflow task it invokes the **Job invoker** function
7+
- **Job invoker** - Javascript function which creates jobs on a (remote) infrastructure to execute workflow tasks
8+
- **Redis server** - used for communication between the Hyperflow engine and Job executors on remote workers
9+
1. Worker components:
10+
- **Hyperflow job executor** - receives the job command from the Hyperflow engine and spawns application software
11+
- **Application software** - software that actually performs workflow tasks
12+
13+
In this example:
14+
- The workflow invokes function `submitRemoteJob` (Job invoker) from `functions.js` 100 times. This function simulates submission of jobs by starting 100 parallel processes of `node handler.js <taskId> <redis_url>`.
15+
- `../../../hyperflow-job-executor/handler.js` represents a remote job executor which is passed two parameters: `taskId` and `redis_url`.
616
- `handler.js` gets a `jobMessage` from HyperFlow, and then sends back a notification that the job has completed; `taskId` is used to construct appropriate Redis keys for this communication.
7-
- On the HyperFlow engine side, the Process Function can use two functions: `context.sendMsgToJob` to send a message to the job, and `context.jobResult` to wait for the notification. These functions return a [`Promise`](https://javascript.info/promise-basics), so the async/await syntax can be used as shown in the example.
17+
- On the HyperFlow engine side, the Job invoker can use two functions (provided by Hyperflow): `context.sendMsgToJob` to send a message to the job, and `context.jobResult` to wait for the notification. These functions return a [`Promise`](https://javascript.info/promise-basics), so the async/await syntax can be used as shown in the example.
818
- The parameter to the `context.jobResult` function is a timeout in seconds (0 denotes infinity). One can use a retry library, such as [promise-retry](https://www.npmjs.com/package/promise-retry), to implement an exponential retry strategy.
9-
- The Process Function also gets the Redis URL in `context.redis_url` which can be passed to the remote job executors.
19+
- The Job invoker also gets the Redis URL in `context.redis_url` which can be passed to the remote job executors.
1020

11-
To run the workflow, simply do `hflow run .` in this directory. You might need to run once `npm install` to install dependencies.
21+
To run the workflow, execute the following commands:
22+
1. First, clone the Hyperflow engine and the Hyperflow job executor:
23+
- `git clone https://github.com/hyperflow`
24+
- `git clone https://github.com/hyperflow-wms/hyperflow-job-executor`
25+
- `cd hyperflow; npm install`
26+
- `cd ../hyperflow-job-executor; npm install`
27+
1. Start the redis server
28+
1. To run the workflow:
29+
- `cd ../hyperflow/examples/RemoteJobs`
30+
- `npm install` (once)
31+
- `hflow run .`
1232

0 commit comments

Comments
 (0)