Skip to content
This repository was archived by the owner on Mar 25, 2023. It is now read-only.

Commit 3efc5d0

Browse files
committed
Update docs to use a newer mount handler syntax + link to the HNFDW example.
1 parent e8c9dab commit 3efc5d0

File tree

5 files changed

+21
-78
lines changed

5 files changed

+21
-78
lines changed

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0200_load-mongo-collections.mdx

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,8 @@ $ sgr mount mongo_fdw example/mongo \\
1010
-c username:password@mongo_host:27017 -o @- <<EOF
1111
{
1212
"example_table": {
13-
"db": "example_db",
14-
"coll": "example_collection",
15-
"schema": {"column_1": "text", "column_2": "numeric"}
13+
"schema": {"column_1": "text", "column_2": "numeric"},
14+
"options": {"db": "example_db", "coll": "example_collection"}
1615
}
1716
}
1817
EOF

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0300_load-postgres-tables.mdx

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,10 @@ You can also pass a dictionary of tables and their schema as `tables`. This will
2424
{
2525
"tables": {
2626
"table_1": {
27-
"col_1": "integer",
28-
"col_2": "text"
27+
"schema": {
28+
"col_1": "integer",
29+
"col_2": "text"
30+
}
2931
}
3032
}
3133
}

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0400_load-mysql-tables.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,8 @@ To mount a MySQL database into the Splitgraph engine using [`mysql_fdw`](https:/
99
$ sgr mount mysql_fdw local_schema -c username:password@host:port -o@- <<EOF
1010
{
1111
"remote_schema": "remote_schema",
12-
"tables": ["table_1"],
12+
"tables": ["table_1"]
1313
}
14-
1514
```
1615

1716
This will mount a MySQL schema `remote_schema` into a local schema `schema_name` on the
@@ -26,8 +25,10 @@ You can also pass a dictionary of tables and their schema as `tables`. This will
2625
{
2726
"tables": {
2827
"table_1": {
29-
"col_1": "integer",
30-
"col_2": "text"
28+
"schema": {
29+
"col_1": "integer",
30+
"col_2": "text"
31+
}
3132
}
3233
}
3334
}

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0450_load-elasticsearch-index.mdx

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,11 @@ $ sgr mount elasticsearch -c elasticsearch:9200 -o@- <<EOF
1717
"col_1": "text",
1818
"col_2": "boolean",
1919
},
20-
"index": "index-pattern*",
21-
"rowid_column": "id",
22-
"query_column": "query",
20+
"options": {
21+
"index": "index-pattern*",
22+
"rowid_column": "id",
23+
"query_column": "query"
24+
}
2325
}
2426
}
2527
}

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0500_load-mount-handlers.mdx

Lines changed: 5 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -6,76 +6,15 @@ export const meta = {
66
To mount a custom database into your Splitgraph engine, you have to do three things:
77

88
- Install the foreign data wrapper into the engine (either using PGXN or compiling the wrapper by yourself)
9-
- Write a Python function that, when invoked, will create the required mountpoint on the engine and initialize
10-
the remote foreign data wrapper. For an example, see `mount_postgres` in `splitgraph.hooks.mount_handlers`.
9+
- Write a Python class deriving from `ForeignDataWrapperDataSource` that defines the FDW's class
10+
and how command line options map to the FDW's own options. For an example, see `PostgreSQLDataSource` in `splitgraph.hooks.data_source`.
1111
- Register the handler in your `.sgconfig` file:
1212

1313
```ini
1414
[mount_handlers]
15-
handler_name=your.handler.module.handler_function
15+
handler_name=your.handler.module.HandlerClass
1616
```
1717

18-
Registering the handler in such a way will also parse its function signature and docstring, adding the handler automatically to the `sgr` client as a subcommand, as well as making it available to be used in Splitfiles.
18+
Registering the handler in such a way will also use make it available in the `sgr` client as an `sgr mount` subcommand, as well as make it available to be used in Splitfiles.
1919

20-
For example, the `mount_postgres` function:
21-
22-
```py
23-
def mount_postgres(mountpoint, server, port, username, password, dbname, remote_schema, tables=[]):
24-
"""
25-
Mount a Postgres database.
26-
27-
Mounts a schema on a remote Postgres database as a set of foreign tables locally.
28-
\b
29-
30-
:param mountpoint: Schema to mount the remote into.
31-
:param server: Database hostname.
32-
:param port: Port the Postgres server is running on.
33-
:param username: A read-only user that the database will be accessed as.
34-
:param password: Password for the read-only user.
35-
:param dbname: Remote database name.
36-
:param remote_schema: Remote schema name.
37-
:param tables: Tables to mount (default all).
38-
"""
39-
engine = get_engine()
40-
logging.info("Importing foreign Postgres schema...")
41-
42-
# Name foreign servers based on their targets so that we can reuse them.
43-
server_id = '%s_%s_%s_server' % (server, str(port), dbname)
44-
init_fdw(engine, server_id, "postgres_fdw", {'host': server, 'port': str(port), 'dbname': dbname},
45-
{'user': username, 'password': password}, overwrite=False)
46-
47-
engine.run_sql(SQL("CREATE SCHEMA {}").format(Identifier(mountpoint)))
48-
49-
# Construct a query: import schema limit to (%s, %s, ...) from server mountpoint_server into mountpoint
50-
query = "IMPORT FOREIGN SCHEMA {} "
51-
if tables:
52-
query += "LIMIT TO (" + ",".join("%s" for _ in tables) + ") "
53-
query += "FROM SERVER {} INTO {}"
54-
engine.run_sql(SQL(query).format(Identifier(remote_schema), Identifier(server_id),
55-
Identifier(mountpoint)), tables)
56-
```
57-
58-
gets made available in the `sgr` client with this help message:
59-
60-
```bash
61-
$ sgr mount postgres_fdw --help
62-
Usage: sgr mount postgres_fdw [OPTIONS] SCHEMA
63-
64-
Mount a Postgres database.
65-
66-
Mounts a schema on a remote Postgres database as a set of foreign
67-
tables locally.
68-
69-
Options:
70-
-c, --connection TEXT Connection string in the form
71-
username:password@server:port
72-
-o, --handler-options TEXT JSON-encoded dictionary of handler options:
73-
dbname: Remote database name.
74-
remote_schema:
75-
Remote schema name.
76-
tables: Tables to mount
77-
(default all).
78-
--help Show this message and exit.
79-
```
80-
81-
The connection string gets parsed and passed to the mount handler as `server`, `port`, `username` and `password` parameters. The remaining options are converted from a JSON dictionary and passed to the handler as extra kwargs.
20+
For an end-to-end example of a custom mount handler querying the Firebase Hacker News API, see [Splitgraph's GitHub repository](https://github.com/splitgraph/splitgraph/tree/master/examples/custom_fdw).

0 commit comments

Comments
 (0)