Skip to content

Commit 01d5633

Browse files
milesrichardsonmilesrichardson
committed
Add files and documentation for developing example in isolated repo
Migrate pull request from: splitgraph/madatdata#21 into its own repo, using `git-filter-repo` to include only commits from subdirectory `examples/nextjs-import-airbyte-github-export-seafowl/` ref: https://github.com/newren/git-filter-repo This commit adds the Yarn files necessary for running the example in an isolated repo (as opposed to as one of multiple examples in a shared multi-workspace `examples`), points the dependencies to `canary` versions (which reflect versions in splitgraph/madatdata#20), and also updates the readme with information for running in local development.
1 parent cf40fa9 commit 01d5633

File tree

6 files changed

+1948
-14
lines changed

6 files changed

+1948
-14
lines changed

‎.gitignore‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,24 @@
1+
# yarn ignore patterns (provided by yarn
2+
.yarn/*
3+
!.yarn/patches
4+
!.yarn/plugins
5+
!.yarn/releases
6+
!.yarn/sdks
7+
!.yarn/versions
8+
9+
# Swap the comments on the following lines if you don't wish to use zero-installs
10+
# Documentation here: https://yarnpkg.com/features/zero-installs
11+
#!.yarn/cache
12+
.pnp.*
13+
14+
# Next.js ignore patterns (don't check in secrets, but check in test vars)
115
.next
216
.env.local
317
.env.*.local
418
!.env.test.local
19+
20+
# We're using nodeLinker: node-modules ("traditional" setup rather than PnP/zero-install)
21+
node_modules
22+
23+
# Misc
24+
.DS_Store

‎.yarn/releases/yarn-3.2.1.cjs‎

Lines changed: 786 additions & 0 deletions
Large diffs are not rendered by default.

‎.yarnrc.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
enableTelemetry: false
2+
3+
nodeLinker: node-modules
4+
5+
npmPublishAccess: private
6+
7+
yarnPath: .yarn/releases/yarn-3.2.1.cjs

‎README.md‎

Lines changed: 111 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,125 @@
1-
# End-to-End Example: Use `airbyte-github` to import GitHub repository into Splitgraph, then export it to Seafowl, via Next.js API routes
1+
# End-to-End Example of Splitgraph, Seafowl and Madatdata
22

3-
This is a full end-to-end example demonstrating importing data to Splitgraph
4-
(using the `airbyte-github` plugin), exporting it to Seafowl (using the
5-
`export-to-seafowl` plugin), and then querying it (with `DbSeafowl` and React
6-
hooks from `@madatdata/react`). The importers and exporting of data is triggered
7-
by backend API routes (e.g. the Vecel runtime), which execute in an environment
8-
with secrets (an `API_SECRET` for Splitgraph, and a GitHub access token for
9-
`airbyte-github`). The client side queries Seafowl directly by sending raw SQL
10-
queries in HTP requests, which is what Seafowl is ultimately designed for.
3+
This is a full, end-to-end example built with Next.js that uses [madatdata][madatdata-repo],
4+
[Splitgraph][splitgraph-home] and [Seafowl][seafowl-home]. It renders a website
5+
where users can import GitHub repository data into Splitgraph, and then export
6+
that data to Seafowl, and query it directly from the client. In doing so,
7+
it demonstrates:
8+
9+
- Importing data to Splitgraph (using the `airbyte-github` madatdata plugin)
10+
- Exporting it to Seafowl (using the `export-to-seafowl` madatdata plugin)
11+
- And then querying it, at both Splitgraph DDN and `demo.seafowl.cloud`, sending
12+
queries directly from the client with `@madatdata/react` and `useSql`
13+
14+
The example uses Next.js with backend routes that call the Splitgraph API to trigger
15+
data import (from GitHub) and data export (to Seafowl). The client side pages query
16+
Seafowl and Splitgraph DDN directly by sending raw SQL queries in HTTP requests,
17+
which is what both Splitgraph and Seafowl are ultimately designed for.
1118

1219
## Try Now
1320

1421
### Preview Immediately
1522

1623
_No signup required, just click the button!_
1724

18-
[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/splitgraph/madatdata/tree/main/examples/nextjs-import-airbyte-github-export-seafowl?file=pages/index.tsx)
25+
[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/splitgraph/demo-github-analytics/tree/main?file=pages/index.tsx)
1926

20-
[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/splitgraph/madatdata/main/examples/nextjs-import-airbyte-github-export-seafowl?file=pages/index.tsx&hardReloadOnChange=true&startScript=dev&node=16&port=3000)
27+
[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/splitgraph/demo-github-analytics/main?file=pages/index.tsx&hardReloadOnChange=true&startScript=dev&node=16&port=3000)
2128

2229
### Or, deploy to Vercel (signup required)
2330

2431
_Signup, fork the repo, and import it_
2532

26-
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/git/external?repository-url=https://github.com/splitgraph/madatdata/tree/main/examples/nextjs-import-airbyte-github-export-seafowl&project-name=madatdata-basic-hooks&repository-name=madatdata-nextjs-basic-hooks)
33+
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/git/external?repository-url=https://github.com/splitgraph/demo-github-analytics/tree/main&project-name=splitgraph-seafowl-madatdata-demo&repository-name=splitgraph-seafowl-madatdata-demo)
34+
35+
## Local Development
36+
37+
### Make sure you're using Yarn Berry
38+
39+
Make sure your local Yarn executable is set to Yarn Berry, so that it will read
40+
from `.yarnrc.yml` which will direct it to use the executable in `.yarn/releases`,
41+
which at the moment of writing is version `3.2.1`
42+
43+
```bash
44+
yarn set version berry
45+
```
46+
47+
Sanity check, `yarn --version` should be `3.2.1`:
48+
49+
```bash
50+
yarn --version
51+
# 3.2.1
52+
```
53+
54+
Also make sure you're using Node v18+:
55+
56+
```bash
57+
node --version
58+
# v18.10.0
59+
```
60+
61+
<details>
62+
63+
<summary>Install latest `node` with `nvm`, and then install `yarn` as global package</summary>
64+
65+
You can use `nvm` to install the latest Node version (assuming you've setup `nvm`):
66+
67+
```bash
68+
nvm install
69+
nvm use
70+
npm install -g yarn
71+
```
72+
73+
Note: When calling `nvm install`, you can tell it to [migrate global packages][github-nvm-migrate-global-packages]
74+
from your current version to the new version, which could save you from
75+
running `npm install -g yarn`, if for some reason this is easier for you:
76+
77+
```bash
78+
nvm install --reinstall-packages-from=current
79+
nvm use
80+
```
81+
82+
</details>
83+
84+
### Install
85+
86+
To install using the same versions of all packages as listed in the lockfile:
87+
88+
```bash
89+
yarn install --immutable
90+
```
91+
92+
### Configure
93+
94+
You will need:
95+
96+
- Splitgraph API key and secret ([manage credentials][splitgraph-manage-credentials])
97+
- Splitgraph namespace to use (it should be your username)
98+
- GitHub Personal Access Token ([get one][create-github-pat])
99+
100+
The specific environment variable keys are documented via comments in the `.env.test.local`
101+
file, which you should copy to the (ignored) path of `.env.local`:
102+
103+
```bash
104+
cp .env.test.local .env.local
105+
```
106+
107+
Then, edit the `.env.local` file to insert the appropriate values for the required
108+
variables.
109+
110+
### Run
111+
112+
To run in development mode:
113+
114+
```bash
115+
yarn dev
116+
```
117+
118+
---
119+
120+
[madatdata-repo]: https://github.com/splitgraph/madatdata
121+
[splitgraph-home]: https://www.splitgraph.com
122+
[seafowl-home]: https://seafowl.io
123+
[github-nvm-migrate-global-packages]: https://github.com/nvm-sh/nvm#migrating-global-packages-while-installing
124+
[splitgraph-manage-credentials]: https://www.splitgraph.com/connect/query
125+
[create-github-pat]: https://github.com/settings/personal-access-tokens/new

‎package.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,10 @@
66
"build": "yarn next build",
77
"start": "yarn next start"
88
},
9+
"packageManager": "yarn@3.2.1",
910
"dependencies": {
10-
"@madatdata/core": "latest",
11-
"@madatdata/react": "latest",
11+
"@madatdata/core": "canary",
12+
"@madatdata/react": "canary",
1213
"@observablehq/plot": "0.6.7",
1314
"next": "latest",
1415
"react": "18.2.0",

0 commit comments

Comments
 (0)