Bring back local development docs (#2491)

* Bring back docs on local development

* Improve local dev docs

Author: ZIJ

docs/ce/local-development/backend.mdx

@@ -1,5 +1,5 @@
 ---
-title: Backend (orchestrator) local setup
+title: Orchestrator local setup
 ---
 
 The backend serves orchestration APIs, GitHub app endpoints, and internal APIs the UI relies on.
@@ -51,7 +51,7 @@ The backend serves orchestration APIs, GitHub app endpoints, and internal APIs t
 
 ## GitHub app integration
 
-- For a quick install link, set `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local` to your app’s install URL (`https://github.com/apps/<app>/installations/new`).
+- For a quick install link, set `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local` to your app's install URL (`https://github.com/apps/<app>/installations/new`).
 - To create a new app via the backend, open `http://localhost:3000/github/setup` (requires `HOSTNAME` set to a reachable URL for callbacks).
 
 ## Troubleshooting

docs/ce/local-development/github-app.mdx

@@ -0,0 +1,33 @@
+---
+title: GitHub App settings for local dev
+---
+
+Use these settings when connecting a GitHub App to your local stack (tunneled via the UI domain).
+
+## Required URLs
+
+- **Callback URL** (OAuth/web): `https://<your-public-ui-domain>/orchestrator/github/callback`
+- **Webhook URL**: `https://<your-public-ui-domain>/orchestrator/github/webhook`
+- **Setup URL (optional)**: `https://<your-public-ui-domain>/dashboard/onboarding?step=github`
+
+> The UI forwards these to the backend. Ensure `ORCHESTRATOR_BACKEND_URL`/`SECRET` are set in UI and the backend is reachable from the UI host.
+
+## Permissions & events (recommended)
+
+- Permissions: `contents:read`, `pull_requests:write`, `issues:write`, `statuses:write`, `checks:write`, `metadata:read`, `administration:read`, `workflows:write`, `repository_hooks:write`, `members:read`.
+- Events: `issue_comment`, `pull_request`, `pull_request_review`, `pull_request_review_comment`, `push`, `check_run`, `status`.
+
+## Install URL
+
+After creating the app, use its install URL (e.g., `https://github.com/apps/<app-name>/installations/new`) as `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local`. This drives the "Connect with GitHub" button.
+
+## Creating an app via the backend wizard
+
+- Open `http://localhost:3000/github/setup` (or the backend's public URL) to generate a manifest and create the app in GitHub. Needed envs on backend: `HOSTNAME` set to a reachable URL, and optional `GITHUB_ORG` if you want to scope to an org.
+- Once created, copy the install URL into `ORCHESTRATOR_GITHUB_APP_URL` and restart the UI.
+
+## Troubleshooting
+
+- **404 on connect**: `ORCHESTRATOR_GITHUB_APP_URL` not set or points to a non-existent path.
+- **Callbacks fail**: UI not exposed publicly; tunnel the UI port and update callback/webhook URLs to that domain.
+- **Backend rejects /api/github/link**: ensure `DIGGER_ENABLE_API_ENDPOINTS=true` and `DIGGER_INTERNAL_SECRET` matches the UI `ORCHESTRATOR_BACKEND_SECRET`.

docs/ce/local-development/overview.mdx

@@ -6,7 +6,7 @@ This section explains how to run the three core services locally:
 
 - **Backend** (`backend/`, port 3000 by default) – orchestrator + REST APIs for repos/orgs/jobs.
 - **Statesman** (`taco/cmd/statesman`, port 8080) – state storage API and Terraform Cloud-compatible endpoints.
-- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS.
+- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS. When tunneling (e.g., ngrok), expose the UI host; WorkOS and GitHub callbacks should point to the UI domain.
 
 ## Prerequisites
 
@@ -23,17 +23,17 @@ This section explains how to run the three core services locally:
 
 ## High-level workflow
 
-1) **Start backend** with internal + API endpoints enabled (so UI can call `/api/*` and `/github/*`).  
-2) **Start statesman** with internal endpoints enabled; use memory storage for quick start.  
-3) **Configure UI** `.env.local` with URLs + secrets + WorkOS creds; run `pnpm dev --host --port 3030`.  
-4) **Sync org/user** into backend and statesman (WorkOS org id and user id/email) via the provided curl snippets in each page.  
-5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend.
+1) **Start backend** with internal + API endpoints enabled (so UI can call `/api/*` and `/github/*`).
+2) **Start statesman** with internal endpoints enabled; use memory storage for quick start.
+3) **Configure UI** `.env.local` with URLs + secrets + WorkOS creds; run `pnpm dev --host --port 3030`.
+4) **Sync org/user** into backend and statesman (WorkOS org id and user id/email) via the provided curl snippets in each page.
+5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend. Use the UI domain for app callback/webhook URLs (see GitHub App settings page).
 
 ## Troubleshooting cheatsheet
 
-- **Backend /api/* returns 404**: `DIGGER_ENABLE_API_ENDPOINTS` not `true` or org not upserted.  
-- **Statesman 403**: webhook secret mismatch. **Statesman 404/500 resolving org**: org not synced (missing `external_org_id`).  
-- **UI WorkOS auth succeeds but org is empty**: add membership in WorkOS and resync org/user to services.  
+- **Backend /api/* returns 404**: `DIGGER_ENABLE_API_ENDPOINTS` not `true` or org not upserted.
+- **Statesman 403**: webhook secret mismatch. **Statesman 404/500 resolving org**: org not synced (missing `external_org_id`).
+- **UI WorkOS auth succeeds but org is empty**: add membership in WorkOS and resync org/user to services.
 - **GitHub connect opens 404**: set `ORCHESTRATOR_GITHUB_APP_URL` to a valid install/setup URL.
 
 Continue with the per-service pages for commands and env examples.

docs/ce/local-development/statesman.mdx

@@ -4,6 +4,22 @@ title: Statesman local setup
 
 Statesman serves state storage + Terraform Cloud-compatible APIs. The UI uses its internal endpoints, so enable webhook auth and sync your org/user.
 
+## First-time setup (SQLite migrations)
+
+If using SQLite with persistence, run migrations before starting:
+
+```bash
+cd taco
+
+# If you get checksum errors, regenerate the hash first:
+atlas migrate hash --dir file://migrations/sqlite
+
+# Apply migrations (adjust path to match your OPENTACO_SQLITE_DB_PATH)
+atlas migrate apply --dir file://migrations/sqlite --url "sqlite://cmd/statesman/data/taco.db"
+```
+
+If you don't have Atlas installed: `make atlas-install`
+
 ## Quick start
 
 1. Set env vars:
@@ -24,6 +40,15 @@ Statesman serves state storage + Terraform Cloud-compatible APIs. The UI uses it
 
 Statesman resolves orgs by `external_org_id` (your WorkOS org id). If it cannot resolve, `/internal/api/units` will 500.
 
+A helper script is available at `taco/cmd/statesman/sync-org.sh` - edit the values before running:
+
+```bash
+chmod +x taco/cmd/statesman/sync-org.sh
+./taco/cmd/statesman/sync-org.sh
+```
+
+Or run manually:
+
 ```bash
 SECRET=$OPENTACO_ENABLE_INTERNAL_ENDPOINTS
 ORG_ID=org_xxx                        # WorkOS org id
@@ -49,6 +74,8 @@ curl -s -X POST http://localhost:8080/internal/api/users \
 
 ## Troubleshooting
 
+- **"no such table: organizations"**: Run migrations first (see First-time setup above).
+- **Atlas checksum mismatch**: Run `atlas migrate hash --dir file://migrations/sqlite` then retry apply.
 - **403**: webhook secret mismatch (`OPENTACO_ENABLE_INTERNAL_ENDPOINTS` vs UI `STATESMAN_BACKEND_WEBHOOK_SECRET`).
-- **404/500 resolving org**: org not synced; rerun the `orgs/sync` call above.
+- **404/500 resolving org**: org not synced; rerun the sync script or `orgs/sync` call above.
 - **SQLite quirks**: defaults to SQLite in-process; no config needed for local. For Postgres/MySQL, set `TACO_QUERY_BACKEND` and related envs (see `docs/ce/state-management/query-backend`).

docs/ce/local-development/ui.mdx

@@ -2,21 +2,21 @@
 title: UI local setup
 ---
 
-The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman.
+The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman. It also acts as the public gateway when tunneling (e.g., ngrok): expose the UI port, and point external callbacks to the UI domain.
 
 ## Quick start
 
 1. Copy `.env.example` to `.env.local` in `ui/` and fill the essentials:
    ```bash
    # URLs
-   PUBLIC_URL=http://localhost:3030
-   ALLOWED_HOSTS=localhost,127.0.0.1
+   PUBLIC_URL=http://localhost:3030                            # replace host with your public tunnel when exposing UI
+   ALLOWED_HOSTS=localhost,127.0.0.1                           # include your public tunnel host here
 
    # WorkOS (User Management)
    WORKOS_CLIENT_ID=<your_workos_client_id>
    WORKOS_API_KEY=<your_workos_api_key>
    WORKOS_COOKIE_PASSWORD=<32+ random chars>
-   WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback
+   WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback # replace host with your public tunnel; must match WorkOS config
    WORKOS_WEBHOOK_SECRET=<if using webhooks locally via tunnel>
 
    # Backend
@@ -40,7 +40,7 @@ The UI is a TanStack Start app that authenticates via WorkOS and calls both back
    pnpm install   # or npm install
    pnpm dev --host --port 3030
    ```
-   Open `http://localhost:3030` and sign in with a WorkOS user that belongs to at least one org.
+   Open `http://localhost:3030` (or your tunnel URL) and sign in with a WorkOS user that belongs to at least one org. Ensure the WorkOS redirect URI matches the public URL you configured.
 3. Ensure backend + statesman were started and the same secrets are in place (see [Backend](./backend) and [Statesman](./statesman)).
 4. Sync the WorkOS org/user to both services using the curl snippets on those pages (required for repos/units to load).
 
@@ -49,3 +49,4 @@ The UI is a TanStack Start app that authenticates via WorkOS and calls both back
 - **NotFound/Forbidden listing units**: statesman org/user not synced or webhook secret mismatch.
 - **404 on repos or GitHub connect**: backend missing org/user, `DIGGER_ENABLE_API_ENDPOINTS` not set, or `ORCHESTRATOR_GITHUB_APP_URL` points to a non-existent path.
 - **WorkOS login succeeds but dashboard redirects to / or errors**: the signed-in user has no WorkOS org membership; add to an org and resync to services.
+- **WorkOS redirect blocked**: public URL not whitelisted; add your tunnel host to `ALLOWED_HOSTS` and to the WorkOS redirect URI list.

docs/docs.json

@@ -171,7 +171,8 @@
               "ce/local-development/overview",
               "ce/local-development/backend",
               "ce/local-development/statesman",
-              "ce/local-development/ui"
+              "ce/local-development/ui",
+              "ce/local-development/github-app"
             ]
           },
           {

taco/cmd/statesman/sync-org.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+SECRET=topsecret                      # must match OPENTACO_ENABLE_INTERNAL_ENDPOINTS
+ORG_ID=org_xxx                        # your WorkOS org id
+ORG_NAME=my-org                       # internal org name/slug
+ORG_DISPLAY="My Org"                  # display name
+USER_ID=user_xxx                      # your WorkOS user id
+USER_EMAIL=you@example.com            # your email
+
+# create/sync org
+curl -s -X POST http://localhost:8080/internal/api/orgs/sync \
+  -H "Authorization: Bearer $SECRET" \
+  -H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"'"$ORG_NAME"'","display_name":"'"$ORG_DISPLAY"'","external_org_id":"'"$ORG_ID"'"}'
+
+echo ""
+echo "Org synced. Now syncing user..."
+
+# ensure user exists in that org
+curl -s -X POST http://localhost:8080/internal/api/users \
+  -H "Authorization: Bearer $SECRET" \
+  -H "X-Org-ID: $ORG_ID" -H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \
+  -H "Content-Type: application/json" \
+  -d '{"subject":"'"$USER_ID"'","email":"'"$USER_EMAIL"'"}'
+
+echo ""
+echo "Done!"