Skip to content

Commit 09a85b2

Browse files
author
Jan Tychtl
committed
docs: update documentation for microservices docker-compose
- Replace AIO/GD.CN terminology with GoodData throughout - Add detailed docker-compose setup instructions to CONTRIBUTING.md - Update test overview and comments to reflect new architecture - Fix license key instructions (use build/license file) jira: DX-319 risk: low
1 parent 34be502 commit 09a85b2

File tree

11 files changed

+117
-33
lines changed

11 files changed

+117
-33
lines changed

.openapi-generator/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ if no such exists or updates existing client code.
1515
Use prepared `make` targets from repository root:
1616
- `make api-client`
1717

18-
Predefined targets use `http://gooddata-cn-ce:3000` URL to fetch OpenAPI schema and connect generator container to
19-
the `gooddata-python-sdk_default` network. It is prepared to be executed against running `docker-compose.yaml`
20-
services.
18+
Predefined targets use `http://localhost:3000` URL to fetch OpenAPI schema. If you need to connect to the GoodData
19+
services running in docker-compose, you can use the `gooddata-python-sdk_default` network. The docker-compose.yaml
20+
in the repository root starts all required GoodData microservices for testing and development.
2121

2222
When you need to point generator to different schema source, use `generate_client.sh` script directly.
2323
Below is the example for `gooddata-scan-client` and schema on URL `https://my-gd-cn`, execute it from

CONTRIBUTING.md

Lines changed: 94 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -182,23 +182,104 @@ Tests triggered by `make` can be controlled via these environment variables:
182182
```
183183
184184
### How to update vcrpy tests
185-
Some tests include HTTP call(s) to GD.CN instance. That tests are executed through
186-
[vcrpy](https://vcrpy.readthedocs.io/) so that GD.CN instance is needed either first time or when request is changed.
185+
Some tests include HTTP call(s) to GoodData instance. Those tests are executed through
186+
[vcrpy](https://vcrpy.readthedocs.io/) so that a GoodData instance is needed either the first time or when a request is changed.
187187
It has clear benefits:
188-
- ability to run the tests without GD.CN
188+
- ability to run the tests without a GoodData instance
189189
- request and response snapshot - it makes debugging of HTTP calls simple
190190
191-
But there is one disadvantage. One needs GD.CN instance with the original setup to change tests.
192-
`docker-compose.yaml` in root of the repository is here to help. It starts:
193-
- GD.CN AIO in selected version
194-
- postgres with gooddata-fdw extension
195-
- service which setups GD.CN AIO demo project including PDM, LDM, metrics and visualizations
191+
But there is one disadvantage. One needs a GoodData instance with the original setup to change tests.
192+
`docker-compose.yaml` in the root of the repository is here to help.
193+
194+
#### Prerequisites for Running Tests Locally
195+
196+
1. **AWS ECR Login** - The docker-compose uses ECR images:
197+
```bash
198+
aws ecr get-login-password | docker login --username AWS --password-stdin 020413372491.dkr.ecr.us-east-1.amazonaws.com
199+
```
200+
201+
2. **GoodData License Key** - Get from the GoodData team and place it in the `./build/license` file:
202+
```bash
203+
mkdir -p build
204+
echo "<your_license_key>" > build/license
205+
```
206+
The auth-service reads the license from this mounted location.
207+
208+
#### What docker-compose starts
209+
210+
The docker-compose starts a full GoodData microservices stack:
211+
212+
**Infrastructure services:**
213+
- PostgreSQL (with demo databases: `md`, `dex`, `automation`, `gw`, `tiger`)
214+
- Redis (caching)
215+
- Apache Pulsar (messaging)
216+
- Traefik (routing)
217+
- Dex (OIDC authentication)
218+
219+
**Core GoodData services:**
220+
- metadata-api, auth-service, calcique, sql-executor, result-cache
221+
- afm-exec-api, scan-model, api-gateway, api-gw
222+
- automation, export-controller, tabular-exporter
223+
- quiver (data processing engine)
224+
225+
**Bootstrap services (run once):**
226+
- `metadata-organization-bootstrap` - Creates organization + admin user
227+
- `data-loader` - Loads demo data into PostgreSQL (with `--no-schema-versioning`)
228+
- `create-ds` - Registers data sources in metadata-api
229+
- `layout-uploader` - Uploads workspace hierarchy, analytics model, users, permissions
230+
231+
#### The `--no-schema-versioning` Flag
232+
233+
The data-loader uses `--no-schema-versioning` flag to ensure:
234+
- Schema names are consistent (e.g., `demo` not `demo_abc123`)
235+
- Fixture names don't have hash suffixes
236+
- VCR cassette tests produce reproducible results
237+
238+
#### Starting GoodData for Tests
239+
240+
```bash
241+
# Start all services
242+
docker compose up -d
243+
244+
# Wait for bootstrap to complete (watch for "Layout upload completed successfully!")
245+
docker compose logs -f metadata-organization-bootstrap data-loader create-ds layout-uploader
246+
247+
# Check service status
248+
docker compose ps
249+
250+
# The GoodData API is available at http://localhost:3000
251+
# Default credentials: demo@example.com / demo123
252+
# API token: YWRtaW46Ym9vdHN0cmFwOmFkbWluMTIz
253+
```
254+
255+
#### Updating vcrpy Cassettes
196256
197257
When a vcrpy supported test needs to be updated:
198-
- start GD.CN using above `docker-compose.yaml`
199-
- delete original vcrpy cassette with `make remove-cassettes`
200-
- execute test
201-
- update a newly generated cassette to the git
258+
- Start GoodData using the above `docker-compose.yaml`
259+
- Wait for all bootstrap services to complete
260+
- Delete original vcrpy cassette with `make remove-cassettes`
261+
- Execute test
262+
- Commit the newly generated cassette to git
263+
264+
#### Stopping and Cleanup
265+
266+
```bash
267+
# Stop all services
268+
docker compose down
269+
270+
# Full cleanup (remove volumes - required for fresh start)
271+
docker compose down -v
272+
```
273+
274+
#### Running gooddata-fdw Tests
275+
276+
The FDW (Foreign Data Wrapper) tests require an additional service. Start it with:
277+
278+
```bash
279+
docker compose --profile fdw up -d
280+
```
281+
282+
This starts a PostgreSQL instance with the gooddata-fdw extension on port 2543.
202283
203284
## Run continuous integration tests
204285
Tests in pull request (PR) are executed using docker. The following is done to make test environment as close
@@ -229,7 +310,7 @@ venv automatically. So when docker tox tests are executed after localhost tests
229310
```bash
230311
TEST_ENVS=py311,py310 ADD_ARGS="-k http_headers" make test-ci
231312
```
232-
- run tests on localhost against all-in-one image started with docker-compose
313+
- run tests on localhost against microservices started with docker-compose
233314
```bash
234315
RECREATE_ENVS=1 HOST_NETWORK=1 make test-ci
235316
```

clients_README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Generated API Clients
22

3-
The generated client provide Python classes that you can use to call GoodData.CN REST APIs. The client contains
3+
The generated client provides Python classes that you can use to call GoodData REST APIs. The client contains
44
models for API requests and responses and controllers to actually call the APIs.
55

66
We recommend using the gooddata_sdk package where possible because:

docker-compose.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# aws ecr get-login-password | docker login --username AWS --password-stdin 020413372491.dkr.ecr.us-east-1.amazonaws.com
1212
#
1313
# 2. GoodData license key (get from GoodData team):
14-
# export GDCN_LICENSE_KEY=<your_license_key>
14+
# mkdir -p build && echo "<your_license_key>" > build/license
1515
#
1616
# ============================================================================
1717
# WORKFLOW
@@ -43,10 +43,10 @@
4343
# USAGE
4444
# ============================================================================
4545
# # Start all services:
46-
# GDCN_LICENSE_KEY=<key> docker compose up -d
46+
# docker compose up -d
4747
#
4848
# # Wait for bootstrap to complete:
49-
# docker compose logs -f metadata-organization-bootstrap data-loader create-ds
49+
# docker compose logs -f metadata-organization-bootstrap data-loader create-ds layout-uploader
5050
#
5151
# # Check service status:
5252
# docker compose ps

packages/gooddata-dbt/src/gooddata_dbt/dbt_plugin.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
from gooddata_dbt.sdk_wrapper import GoodDataSdkWrapper
2222
from gooddata_dbt.utils import get_duration, report_message_to_git_vendor
2323

24-
# TODO - upgrade AIO, cleanup, start from scratch, test everything
24+
# TODO - cleanup, start from scratch, test everything
2525

2626
GOODDATA_LAYOUTS_DIR = Path("gooddata_layouts")
2727

packages/gooddata-dbt/src/gooddata_dbt/sdk_wrapper.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010

1111
class GoodDataSdkWrapper:
12-
# Timeout=600 because supporting waiting for All-in-one image starts
12+
# Timeout=600 because supporting waiting for GoodData services to start
1313
def __init__(
1414
self, args: argparse.Namespace, logger: Logger, profile: Optional[str] = None, timeout: int = 600
1515
) -> None:

packages/gooddata-pandas/tests/dataframe/test_dataframe_for_insight.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
@gd_vcr.use_cassette(str(_fixtures_dir / "dataframe_for_visualization_date.yaml"))
1414
def test_dataframe_for_visualization_date(gdf: DataFrameFactory):
1515
# 2 metrics grouped by date dimension with data for last 12 months
16-
# exact numbers cannot be checked as date data are changed each AIO build
16+
# exact numbers cannot be checked as date data are changed each build
1717
df = gdf.for_visualization(visualization_id="customers_trend")
1818

1919
assert len(df) == 12

packages/gooddata-sdk/tests/catalog/test_catalog_data_source.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -652,7 +652,7 @@ def test_scan_model(test_config):
652652
with pytest.raises(ValueError):
653653
CatalogScanModelRequest(scan_tables=False, scan_views=False)
654654

655-
# TODO - how to simulate warnings in AIO?
655+
# TODO - how to simulate warnings in the test environment?
656656
assert len(scan_result.warnings) == 0
657657

658658

packages/gooddata-sdk/tests/compute/test_compute_service.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
# Skip all tests in this module
1010
pytest.skip(
11-
"Skipping all tests in this module because it requires gen-ai which is not available in AIO.",
11+
"Skipping all tests in this module because it requires gen-ai which is not available in the test environment.",
1212
allow_module_level=True,
1313
)
1414

packages/gooddata-sdk/tests/overview.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,12 @@ will find that the data in snapshots does not match the conventions of the API.
3838

3939
## Component tests with vcrpy
4040

41-
The vcrpy simplifies mocking HTTP endpoints. The component tests use the AIO image environment and the demo
42-
workspace. The connection details are stored in [gd_test_config.yaml](gd_test_config.yaml). Tests accept command
43-
line parameter `--gd-test-config` to pass custom test configuration.
41+
The vcrpy simplifies mocking HTTP endpoints. The component tests use the GoodData microservices environment
42+
(started via `docker-compose.yaml` in the repository root) and the demo workspace. The connection details are
43+
stored in [gd_test_config.yaml](gd_test_config.yaml). Tests accept command line parameter `--gd-test-config`
44+
to pass custom test configuration.
45+
46+
See [CONTRIBUTING.md](../../../CONTRIBUTING.md) for detailed instructions on setting up the test environment.
4447

4548
Additionally, make sure that every time you use vcrpy, you configure the tool:
4649
- to not store the 'Authorization' token and 'user-agent' in the captured data

0 commit comments

Comments
 (0)