Skip to content

Commit 058dca1

Browse files
NormBNormB
authored
fix: codebase improvements from analysis (#50)
* refactor: align implementations and improve error messages - Fix Go debug default from true to false for security - Align API-first FastAPI logging with code-first (structured JSON) - Update API-first version to 1.1.0 to match code-first - Add rate limiting to API-first metrics endpoint - Add structured logging to API-first middleware - Remove outdated pytest version comment - Improve VAULT_TOKEN error message in generate-certificates.sh - Run go mod tidy to fix missing go.sum entries * feat: add test suites and improve documentation Node.js reference app: - Add Jest configuration with coverage thresholds - Add comprehensive health check tests with mocked dependencies - Add configuration tests - Add application entry point tests TypeScript API-first reference app: - Add Jest configuration for TypeScript with ts-jest - Add comprehensive health check tests - Add configuration and entry point tests - Add supertest and @types/supertest dependencies Documentation: - Document AppRole secret_id renewal process with commands - Add automated renewal instructions (crontab) - Add renewal timeline table Configuration: - Fix .env.example network IPs to match 4-tier architecture - Organize IPs by network segment (vault/data/app/observability)
1 parent 33a2818 commit 058dca1

File tree

19 files changed

+1367
-82
lines changed

19 files changed

+1367
-82
lines changed

‎.env.example‎

Lines changed: 28 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,26 +26,37 @@ VAULT_TOKEN=
2626
# Or leave empty and it will be read from ~/.config/vault/root-token
2727

2828
# ===========================================================================
29-
# Docker Network IP Addresses (172.20.0.0/16)
29+
# Docker Network IP Addresses (4-Tier Segmentation)
3030
# ===========================================================================
31-
# Static IP addresses for services in the dev-services network
31+
# Static IP addresses for services across 4 network segments:
32+
# - vault-network: 172.20.1.0/24 (secrets management)
33+
# - data-network: 172.20.2.0/24 (databases, redis, rabbitmq)
34+
# - app-network: 172.20.3.0/24 (forgejo, reference APIs)
35+
# - observability-network: 172.20.4.0/24 (prometheus, grafana, loki)
36+
#
3237
# Change these if you need custom IP assignments or have conflicts
3338
#
34-
# Default assignments:
35-
POSTGRES_IP=172.20.0.10
36-
PGBOUNCER_IP=172.20.0.11
37-
MYSQL_IP=172.20.0.12
38-
REDIS_1_IP=172.20.0.13
39-
RABBITMQ_IP=172.20.0.14
40-
MONGODB_IP=172.20.0.15
41-
REDIS_2_IP=172.20.0.16
42-
REDIS_3_IP=172.20.0.17
43-
FORGEJO_IP=172.20.0.20
44-
VAULT_IP=172.20.0.21
45-
REFERENCE_API_IP=172.20.0.100
46-
PROMETHEUS_IP=172.20.0.101
47-
GRAFANA_IP=172.20.0.102
48-
LOKI_IP=172.20.0.103
39+
# Vault Network (172.20.1.x):
40+
VAULT_IP=172.20.1.5
41+
42+
# Data Network (172.20.2.x):
43+
POSTGRES_IP=172.20.2.10
44+
PGBOUNCER_IP=172.20.2.11
45+
MYSQL_IP=172.20.2.12
46+
REDIS_1_IP=172.20.2.13
47+
RABBITMQ_IP=172.20.2.14
48+
MONGODB_IP=172.20.2.15
49+
REDIS_2_IP=172.20.2.16
50+
REDIS_3_IP=172.20.2.17
51+
52+
# App Network (172.20.3.x):
53+
FORGEJO_IP=172.20.3.20
54+
REFERENCE_API_IP=172.20.3.100
55+
56+
# Observability Network (172.20.4.x):
57+
PROMETHEUS_IP=172.20.4.10
58+
GRAFANA_IP=172.20.4.20
59+
LOKI_IP=172.20.4.30
4960

5061
# ===========================================================================
5162
# TLS Configuration (Enabled by Default)

‎docs/VAULT.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,57 @@ vault write -f -field=secret_id auth/approle/role/reference-api/secret-id > ~/.c
303303
- AppRole implementation: `reference-apps/fastapi/app/services/vault.py`
304304
- Docker configuration: `docker-compose.yml` (line 879)
305305

306+
#### AppRole Secret ID Renewal
307+
308+
**⚠️ IMPORTANT:** AppRole secret_ids have a **30-day TTL**. Services will fail to authenticate after expiry.
309+
310+
**Check Secret ID Expiry:**
311+
```bash
312+
export VAULT_ADDR=http://localhost:8200
313+
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)
314+
315+
# Check remaining TTL for a service (e.g., reference-api)
316+
vault write auth/approle/role/reference-api/secret-id-accessor/lookup \
317+
secret_id_accessor=$(cat ~/.config/vault/approles/reference-api/secret-id-accessor 2>/dev/null)
318+
```
319+
320+
**Renew Secret IDs (Before Expiry):**
321+
```bash
322+
# Renew all AppRole secret_ids
323+
./scripts/vault-approle-bootstrap.sh --renew-secrets
324+
325+
# Or renew for a specific service
326+
SERVICE=reference-api
327+
vault write -f -field=secret_id auth/approle/role/${SERVICE}/secret-id \
328+
> ~/.config/vault/approles/${SERVICE}/secret-id
329+
330+
# Save the accessor for future lookups
331+
vault write -f -field=secret_id_accessor auth/approle/role/${SERVICE}/secret-id \
332+
> ~/.config/vault/approles/${SERVICE}/secret-id-accessor
333+
334+
# Restart the service to pick up new credentials
335+
docker compose restart ${SERVICE}
336+
```
337+
338+
**Automated Renewal (Recommended for Production):**
339+
340+
Add to crontab to renew secret_ids weekly:
341+
```bash
342+
# Edit crontab
343+
crontab -e
344+
345+
# Add this line (runs every Sunday at 3 AM)
346+
0 3 * * 0 cd /path/to/devstack-core && ./scripts/vault-approle-bootstrap.sh --renew-secrets >> /var/log/vault-approle-renewal.log 2>&1
347+
```
348+
349+
**Renewal Timeline:**
350+
| Event | TTL Remaining | Action |
351+
|-------|---------------|--------|
352+
| Created | 30 days | Normal operation |
353+
| Week 3 | 7-14 days | Consider renewal |
354+
| Week 4 | < 7 days | **Renew immediately** |
355+
| Expired | 0 days | Service authentication fails |
356+
306357
### SSL/TLS Certificate Management
307358

308359
**TLS Implementation: Pre-Generated Certificates with Vault-Based Configuration**

‎reference-apps/fastapi-api-first/app/main.py‎

Lines changed: 66 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,10 @@
1414
from slowapi.errors import RateLimitExceeded
1515
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
1616
import logging
17+
import sys
1718
import time
1819
import uuid
20+
from pythonjsonlogger import jsonlogger
1921

2022
from app.routers import (
2123
health_checks,
@@ -30,12 +32,19 @@
3032
from app.middleware.cache import cache_manager
3133
from app.services.vault import vault_client
3234

33-
# Configure logging
34-
logging.basicConfig(
35-
level=logging.INFO,
36-
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
35+
# Configure structured JSON logging (matches code-first implementation)
36+
logHandler = logging.StreamHandler(sys.stdout)
37+
formatter = jsonlogger.JsonFormatter(
38+
'%(asctime)s %(name)s %(levelname)s %(message)s %(request_id)s %(method)s %(path)s %(status_code)s %(duration_ms)s'
3739
)
40+
logHandler.setFormatter(formatter)
3841
logger = logging.getLogger(__name__)
42+
logger.addHandler(logHandler)
43+
logger.setLevel(logging.INFO)
44+
45+
# Disable default basicConfig
46+
logging.getLogger().handlers.clear()
47+
logging.getLogger().addHandler(logHandler)
3948

4049
# Prometheus metrics
4150
http_requests_total = Counter(
@@ -68,7 +77,7 @@
6877
# Create FastAPI app
6978
app = FastAPI(
7079
title="DevStack Core - Reference API (API-First)",
71-
version="1.0.0",
80+
version="1.1.0",
7281
description="API-First implementation generated from OpenAPI specification",
7382
docs_url="/docs",
7483
redoc_url="/redoc",
@@ -112,10 +121,10 @@ async def metrics_middleware(request: Request, call_next):
112121
request.state.request_id = request_id
113122

114123
method = request.method
115-
path = request.url.path
124+
endpoint = request.url.path
116125

117126
# Track in-progress requests
118-
http_requests_in_progress.labels(method=method, endpoint=path).inc()
127+
http_requests_in_progress.labels(method=method, endpoint=endpoint).inc()
119128

120129
start_time = time.time()
121130

@@ -126,23 +135,58 @@ async def metrics_middleware(request: Request, call_next):
126135
# Record metrics
127136
http_requests_total.labels(
128137
method=method,
129-
endpoint=path,
138+
endpoint=endpoint,
130139
status=response.status_code
131140
).inc()
132141

133142
http_request_duration_seconds.labels(
134143
method=method,
135-
endpoint=path
144+
endpoint=endpoint
136145
).observe(duration)
137146

147+
# Log request with structured data (matches code-first implementation)
148+
logger.info(
149+
"HTTP request completed",
150+
extra={
151+
"request_id": request_id,
152+
"method": method,
153+
"path": endpoint,
154+
"status_code": response.status_code,
155+
"duration_ms": round(duration * 1000, 2)
156+
}
157+
)
158+
138159
# Add headers
139160
response.headers["X-Request-ID"] = request_id
140161
response.headers["X-Response-Time"] = f"{duration:.3f}s"
141162

142163
return response
143164

165+
except Exception as e:
166+
# Record error metrics
167+
duration = time.time() - start_time
168+
http_requests_total.labels(
169+
method=method,
170+
endpoint=endpoint,
171+
status=500
172+
).inc()
173+
174+
# Log error with structured data
175+
logger.error(
176+
f"Request failed: {str(e)}",
177+
extra={
178+
"request_id": request_id,
179+
"method": method,
180+
"path": endpoint,
181+
"status_code": 500,
182+
"duration_ms": round(duration * 1000, 2)
183+
},
184+
exc_info=True
185+
)
186+
raise
187+
144188
finally:
145-
http_requests_in_progress.labels(method=method, endpoint=path).dec()
189+
http_requests_in_progress.labels(method=method, endpoint=endpoint).dec()
146190

147191

148192
# Include routers
@@ -158,7 +202,7 @@ async def metrics_middleware(request: Request, call_next):
158202
async def startup_event():
159203
"""Application startup event handler."""
160204
# Set app info metric
161-
app_info.labels(version="1.0.0", name="api-first").set(1)
205+
app_info.labels(version="1.1.0", name="api-first").set(1)
162206

163207
# Initialize response caching with Redis
164208
try:
@@ -171,10 +215,14 @@ async def startup_event():
171215
logger.error(f"Failed to initialize cache: {e}")
172216
logger.warning("Application will continue without caching")
173217

174-
logger.info("Starting API-First FastAPI application...")
175-
logger.info(f"Debug mode: {settings.DEBUG}")
176-
logger.info(f"Vault address: {settings.VAULT_ADDR}")
177-
logger.info(f"Redis cache enabled: {cache_manager.enabled}")
218+
logger.info(
219+
"Starting DevStack Core Reference API (API-First)",
220+
extra={
221+
"vault_address": settings.VAULT_ADDR,
222+
"redis_cache_enabled": cache_manager.enabled,
223+
"version": "1.1.0"
224+
}
225+
)
178226
logger.info("Application ready")
179227

180228

@@ -195,7 +243,7 @@ async def root(request: Request):
195243
"""
196244
return {
197245
"name": "DevStack Core Reference API",
198-
"version": "1.0.0",
246+
"version": "1.1.0",
199247
"description": "Reference implementation for infrastructure integration",
200248
"docs": "/docs",
201249
"health": "/health/all",
@@ -253,7 +301,8 @@ async def root(request: Request):
253301

254302

255303
@app.get("/metrics")
256-
async def metrics():
304+
@limiter.limit("1000/minute") # High limit for metrics scraping
305+
async def metrics(request: Request):
257306
"""Prometheus metrics endpoint"""
258307
return Response(
259308
content=generate_latest(),

‎reference-apps/fastapi/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ python-json-logger==4.0.0
2828
prometheus-client==0.23.1
2929

3030
# Testing
31-
pytest==9.0.1 # pytest-asyncio 1.2.0 requires pytest<9
31+
pytest==9.0.1
3232
pytest-asyncio==1.3.0
3333
pytest-cov==7.0.0
3434
pytest-mock==3.15.1

‎reference-apps/golang/go.mod‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ require (
88
github.com/google/uuid v1.6.0
99
github.com/hashicorp/vault/api v1.22.0
1010
github.com/jackc/pgx/v5 v5.7.6
11+
github.com/prometheus/client_golang v1.19.1
1112
github.com/rabbitmq/amqp091-go v1.10.0
1213
github.com/redis/go-redis/v9 v9.17.2
1314
github.com/sirupsen/logrus v1.9.3
@@ -16,6 +17,7 @@ require (
1617

1718
require (
1819
filippo.io/edwards25519 v1.1.0 // indirect
20+
github.com/beorn7/perks v1.0.1 // indirect
1921
github.com/bytedance/sonic v1.14.0 // indirect
2022
github.com/bytedance/sonic/loader v0.3.0 // indirect
2123
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
@@ -53,6 +55,9 @@ require (
5355
github.com/modern-go/reflect2 v1.0.2 // indirect
5456
github.com/montanaflynn/stats v0.7.1 // indirect
5557
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
58+
github.com/prometheus/client_model v0.5.0 // indirect
59+
github.com/prometheus/common v0.48.0 // indirect
60+
github.com/prometheus/procfs v0.12.0 // indirect
5661
github.com/quic-go/qpack v0.5.1 // indirect
5762
github.com/quic-go/quic-go v0.54.1 // indirect
5863
github.com/ryanuber/go-glob v1.0.0 // indirect

‎reference-apps/golang/go.sum‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
22
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
3+
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
4+
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
35
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
46
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
57
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
@@ -110,6 +112,14 @@ github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0
110112
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
111113
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
112114
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
115+
github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE=
116+
github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho=
117+
github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw=
118+
github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI=
119+
github.com/prometheus/common v0.48.0 h1:QO8U2CdOzSn1BBsmXJXduaaW+dY/5QLjfB8svtSzKKE=
120+
github.com/prometheus/common v0.48.0/go.mod h1:0/KsvlIEfPQCQ5I2iNSAWKPZziNCvRs5EC6ILDTlAPc=
121+
github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo=
122+
github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo=
113123
github.com/quic-go/qpack v0.5.1 h1:giqksBPnT/HDtZ6VhtFKgoLOWmlyo9Ei6u9PqzIMbhI=
114124
github.com/quic-go/qpack v0.5.1/go.mod h1:+PC4XFrEskIVkcLzpEkbLqq1uCoxPhQuvK5rH1ZgaEg=
115125
github.com/quic-go/quic-go v0.54.1 h1:4ZAWm0AhCb6+hE+l5Q1NAL0iRn/ZrMwqHRGQiFwj2eg=

‎reference-apps/golang/internal/config/config.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ func Load() *Config {
5757
return &Config{
5858
// Application
5959
Environment: getEnv("ENVIRONMENT", "development"),
60-
Debug: getEnvBool("DEBUG", true),
60+
Debug: getEnvBool("DEBUG", false),
6161
HTTPPort: getEnv("HTTP_PORT", "8002"),
6262
HTTPSPort: getEnv("HTTPS_PORT", "8445"),
6363
EnableTLS: getEnvBool("GOLANG_API_ENABLE_TLS", false),

‎reference-apps/nodejs/jest.config.js‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,25 @@
1+
/**
2+
* Jest Configuration
3+
*/
4+
15
module.exports = {
26
testEnvironment: 'node',
3-
coverageDirectory: 'coverage',
4-
collectCoverageFrom: ['src/**/*.js'],
57
testMatch: ['**/tests/**/*.test.js'],
8+
collectCoverageFrom: [
9+
'src/**/*.js',
10+
'!src/index.js' // Exclude main entry point from coverage
11+
],
12+
coverageDirectory: 'coverage',
13+
coverageReporters: ['text', 'lcov', 'html'],
14+
coverageThreshold: {
15+
global: {
16+
branches: 50,
17+
functions: 50,
18+
lines: 50,
19+
statements: 50
20+
}
21+
},
22+
setupFilesAfterEnv: ['./tests/setup.js'],
23+
testTimeout: 10000,
624
verbose: true
725
};

0 commit comments

Comments
 (0)