Skip to content
This repository was archived by the owner on Mar 25, 2023. It is now read-only.

Commit 6705e2d

Browse files
committed
Update docs to include new FDW features + elasticsearch FDW
1 parent 61ae642 commit 6705e2d

File tree

6 files changed

+77
-11
lines changed

6 files changed

+77
-11
lines changed

content/docs/0000_getting-started/0100_introduction.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ You do not need to download the full Splitgraph image to query it. Instead, you
2525

2626
Splitgraph does not limit your data sources to Postgres databases. It includes first-class support for importing and querying data from other databases using Postgres [foreign data wrappers](../ingesting-data/foreign-data-wrappers/introduction). You can create Splitgraph images or query data in [MongoDB](../ingesting-data/foreign-data-wrappers/load-mongo-collections),
2727
[MySQL](../ingesting-data/foreign-data-wrappers/load-mysql-tables),
28-
[CSV files](../ingesting-data/load-csv-files) or [other Postgres databases](../ingesting-data/foreign-data-wrappers/load-postgres-tables) using the same interface.
28+
[CSV files](../ingesting-data/load-csv-files), [other Postgres databases](../ingesting-data/foreign-data-wrappers/load-postgres-tables) or even [Elasticsearch clusters](../ingesting-data/foreign-data-wrappers/load-elasticsearch-index) using the same interface.
2929

3030
Finally, Splitgraph is peer-to-peer. You can push and pull data images between other Splitgraph installations and use it as a standalone tool to supercharge your data workflows.
3131

content/docs/0000_getting-started/0150_frequently-asked-questions.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ which will let you ingest data from existing databases without installing Splitg
7171

7272
### Does my data have to be in PostgreSQL to use Splitgraph?
7373

74-
With [mounting](../concepts/mounting), you can query data in other databases (including MongoDB, MySQL or PostgreSQL) directly through Spligraph with
74+
With [mounting](../concepts/mounting), you can query data in other databases (including MongoDB, MySQL, PostgreSQL or Elasticsearch) directly through Spligraph with
7575
[any PostgreSQL client](../integrating-splitgraph/other-clients). You do not need to copy your data into PostgreSQL to use Splitgraph.
7676

7777
### Why PostgreSQL? Why not write your own database?

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0100_introduction.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@ Foreign tables act similarly to normal tables and can accept the same SQL querie
99

1010
In Splitgraph terminology, this is called "mounting", kind of like mounting a filesystem to mount points in the directory tree. You can snapshot these mounted tables, turning them into Splitgraph images, with the [`sgr import`](../../sgr/image-management-creation/import) command.
1111

12-
The Splitgraph engine ships with several open-source foreign data wrappers: [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html), [`mongo_fdw`](https://github.com/EnterpriseDB/mongo_fdw) and [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw).
12+
The Splitgraph engine ships with several open-source foreign data wrappers: [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html), [`mongo_fdw`](https://github.com/EnterpriseDB/mongo_fdw), [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw) and [`pg_es_fdw`](https://github.com/splitgraph/postgres-elasticsearch-fdw).

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0300_load-postgres-tables.mdx

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,29 @@ export const meta = {
66
To use [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html) and mount remote PostgreSQL databases, run:
77

88
```
9-
$ sgr mount postgres_fdw \\
10-
-c username:password@server:port \\
11-
-o '{"dbname": remote_db_name, "remote_schema": remote_schema_name, \\
12-
"tables_to_mount": ["table_1", "table_2"]}' \\
13-
local_schema
9+
$ sgr mount postgres_fdw local_schema -c username:password@server:port -o@- <<EOF
10+
{
11+
"dbname": remote_db_name,
12+
"remote_schema": remote_schema_name,
13+
"tables": ["table_1", "table_2"],
14+
"extra_server_args": [optional, e.g. {"use_remote_estimate": "true"}]
15+
}
16+
EOF
1417
```
1518

16-
`"tables_to_mount"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).
19+
`"tables"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).
20+
21+
You can also pass a dictionary of tables and their schema as `tables`. This will override the schema that's inferred by `IMPORT FOREIGN SCHEMA`. For example:
22+
23+
```json
24+
{
25+
"tables": {
26+
"table_1": {
27+
"col_1": "integer",
28+
"col_2": "text"
29+
}
30+
}
31+
}
32+
```
1733

1834
To snapshot the data, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.

content/docs/0300_ingesting-data/0300_foreign-data-wrappers/0400_load-mysql-tables.mdx

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,31 @@ export const meta = {
66
To mount a MySQL database into the Splitgraph engine using [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw), run:
77

88
```
9-
$ sgr mount mysql_fdw -c username:password@host:port \\
10-
-o '{"remote_schema": "remote_schema"}' schema_name
9+
$ sgr mount mysql_fdw local_schema -c username:password@host:port -o@- <<EOF
10+
{
11+
"remote_schema": "remote_schema",
12+
"tables": ["table_1"],
13+
}
14+
1115
```
1216

1317
This will mount a MySQL schema `remote_schema` into a local schema `schema_name` on the
1418
Splitgraph engine. Note that if your MySQL server is running on localhost and you wish to
1519
access it over a TCP connection, you should use `127.0.0.1` instead of `localhost` as the `host`.
1620

21+
`"tables"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).
22+
23+
You can also pass a dictionary of tables and their schema as `tables`. This will override the schema that's inferred by `IMPORT FOREIGN SCHEMA`. For example:
24+
25+
```json
26+
{
27+
"tables": {
28+
"table_1": {
29+
"col_1": "integer",
30+
"col_2": "text"
31+
}
32+
}
33+
}
34+
```
35+
1736
To snapshot the table, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
export const meta = {
2+
id: "load-elasticsearch-index",
3+
title: "Loading Elasticsearch indexes",
4+
};
5+
6+
To mount a remote Elasticsearch index into the Splitgraph engine using Splitgraph's fork of [`pg_es_fdw`](https://github.com/splitgraph/postgres-elasticsearch-fdw), run:
7+
8+
```
9+
$ sgr mount elasticsearch -c elasticsearch:9200 -o@- <<EOF
10+
{
11+
"table_spec": {
12+
"table_1": {
13+
"schema": {
14+
"id": "text",
15+
"@timestamp": "timestamp",
16+
"query": "text",
17+
"col_1": "text",
18+
"col_2": "boolean",
19+
},
20+
"index": "index-pattern*",
21+
"rowid_column": "id",
22+
"query_column": "query",
23+
}
24+
}
25+
}
26+
EOF
27+
```
28+
29+
The FDW translates local queries into the [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html). You can also pass Lucene query strings as `query_column` equality clauses. For example, if `query_column` is set to `query`, you can use `WHERE query='text_body:chess'`).
30+
31+
To snapshot the table, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.

0 commit comments

Comments
 (0)