Devportal alpha release commit

fabriziofiorucci · fabriziofiorucci · commit 1c0c254e2cbe · 2023-12-08T14:01:15.000Z
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,5 @@
 /.idea/
 /src/__pycache__/
+/src/Contrib/__pycache__/
+/contrib/devportal/src/__pycache__/
 /venv/
diff --git a/README.md b/README.md
@@ -88,8 +88,9 @@ NGINX Instance Manager ->> NGINX: Publish config to NGINX instances
 | Cookie-based stickiness    | X                                                                                       |                                                                                                  |
 | Maps                       | X                                                                                       |                                                                                                  |
 | NGINX Plus REST API access | X                                                                                       |                                                                                                  |
-| NGINX App Protect WAF      | Per-policy CRUD at `server` and `location` level with dataplane-based bundle compilation | Security policies can be dynamically fetched from source of truth                               | 
-| API Gateway                | Swagger and OpenAPI YAML and JSON schema support                                        | Automated configuration, HTTP methods and rate limiting enforcement                              | 
+| NGINX App Protect WAF      | Per-policy CRUD at `server` and `location` level with dataplane-based bundle compilation | Security policies can be dynamically fetched from source of truth                                | 
+| API Gateway                | Swagger and OpenAPI YAML and JSON schema support                                        | Automated configuration, HTTP methods and rate limiting enforcement                              |
+| API Developer Portal       | Swagger and OpenAPI YAML and JSON schema support                                        | Based on Redocly                                                                                 |
 
 ## How to use
 
@@ -105,7 +106,7 @@ This is the recommended method to run NGINX Declarative API on a Linux virtual m
 
 ### As a Python application
 
-This repository has been tested with and requires Python 3.9 or newer.
+This repository requires Python 3.10+
 A running instance of [redis](https://redis.io/) is required: redis host and port can be configured in the `config.toml` file.
 
 Run NGINX Declarative API using:
@@ -124,8 +125,8 @@ The docker image can be built and run using:
 ```
 $ git clone https://github.com/fabriziofiorucci/NGINX-Declarative-API
 $ cd NGINX-Declarative-API
-$ docker build -t nginx-declarative-api:latest -f contrib/docker/Dockerfile .
-$ docker run --name nginx-declarative-api -d -p 5000:5000 nginx-declarative-api:latest
+$ docker build --no-cache -t nginx-declarative-api -f ./Dockerfile .
+$ docker run --name nginx-declarative-api -d -p 5000:5000 nginx-declarative-api
 ```
 
 Pre-built docker images are available on Docker Hub at https://hub.docker.com/repository/docker/fiorucci/nginx-declarative-api/general and can be run using:
diff --git a/USAGE-v3.md b/USAGE-v3.md
@@ -53,7 +53,7 @@ Locations `.declaration.http.servers[].locations[].uri` match modifiers in `.dec
 
 ### API Gateway ###
 
-Swagger files and OpenAPI schemas can be used to automatically configure NGINX as an API Gateway.
+Swagger files and OpenAPI schemas can be used to automatically configure NGINX as an API Gateway. Developer portal creation is supported through [Redocly](https://redocly.com/)
 
 Declaration path `.declaration.http.servers[].locations[].apigateway` defines the API Gateway configuration:
 
@@ -62,9 +62,10 @@ Declaration path `.declaration.http.servers[].locations[].apigateway` defines th
 - `api_gateway.strip_uri` - removes the `.declaration.http.servers[].locations[].uri` part of the URI before forwarding requests to the upstream
 - `api_gateway.server_url` - the base URL of the upstream server
 - `developer_portal.enabled` - enable/disable Developer portal provisioning
+- `developer_portal.uri` - the trailing part of the Developer portal URI, this is appended to `.declaration.http.servers[].locations[].uri`. If omitted it defaults to `devportal.html`
 - `rate_limit` - optional, used to enforce rate limiting at the API Gateway level
 
-A sample API Gateway declaration to publish the `https://petstore.swagger.io` REST API and enforce:
+A sample API Gateway declaration to publish the `https://petstore.swagger.io` REST API DevPortal, API Gateway and enforce:
 
 - REST API endpoint URIs
 - HTTP Methods
@@ -116,7 +117,8 @@ is:
                                     "server_url": "https://petstore.swagger.io/v2"
                                 },
                                 "developer_portal": {
-                                    "enabled": false
+                                    "enabled": false,
+                                    "uri": "/petstore-devportal.html"
                                 },
                                 "rate_limit": {
                                     "profile": "petstore_ratelimit",
@@ -150,6 +152,10 @@ It can be tested using:
 
     curl -iH "Host: apigw.nginx.lab" http://<NGINX_INSTANCE_IP_ADDRESS>/petstore/store/inventory
 
+The API Developer portal can be accessed at:
+
+    http://<NGINX_INSTANCE_IP_ADDRESS>/petstore/petstore-devportal.html
+
 ### Maps ###
 
 Map entries `.declaration.maps[].entries.keymatch` can be:
diff --git a/contrib/docker-compose/docker-compose.yaml b/contrib/docker-compose/docker-compose.yaml
@@ -14,6 +14,14 @@ services:
     volumes:
       - redis_data:/data:rw
 
+  devportal:
+    image: fiorucci/nginx-declarative-api-devportal:latest
+    user: "${USERID}:${USERGROUP}"
+    container_name: "devportal"
+    restart: always
+    ports:
+      - "5001:5000"
+
   nginx-dapi:
     image: fiorucci/nginx-declarative-api:latest
     user: "${USERID}:${USERGROUP}"
diff --git a/contrib/docker-compose/nginx-dapi.sh b/contrib/docker-compose/nginx-dapi.sh
@@ -30,6 +30,9 @@ USERNAME=`whoami`
 export USERID=`id -u $USERNAME`
 export USERGROUP=`id -g $USERNAME`
 
+echo "-> Updating docker images"
+docker-compose -f $DOCKER_COMPOSE_YAML pull
+
 echo "-> Deploying NGINX Declarative API"
 COMPOSE_HTTP_TIMEOUT=240 docker-compose -p $PROJECT_NAME -f $DOCKER_COMPOSE_YAML up -d --remove-orphans
 }
diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile
diff --git a/contrib/docker/start.sh b/contrib/docker/start.sh
diff --git a/etc/config.toml b/etc/config.toml
@@ -24,13 +24,20 @@ port = 5000
 
 # Redis backend for CI/CD automation
 [redis]
-host = "127.0.0.1"
+host = redis
 port = 6379
 
+# Devportal helper
+[devportal]
+host = devportal
+port = 5001
+uri = "/v1/devportal"
+
 # Staged configuration for NGINX Management Suite
 [nms]
 config_dir = '/etc/nginx'
 certs_dir = '/etc/nginx/ssl'
+devportal_dir = '/etc/nginx/devportal'
 
 # Time to wait to get status after committing a staged config
 staged_config_publish_waittime = 2
diff --git a/src/Contrib/DevPortal.py b/src/Contrib/DevPortal.py
@@ -0,0 +1,37 @@
+"""
+API Gateway Developer Portal support functions
+"""
+
+import json
+import requests
+import base64
+
+# NGINX Declarative API modules
+from NcgConfig import NcgConfig
+import Contrib.GitOps
+import Contrib.MiscUtils
+
+
+def buildDevPortal(openapischema):
+    response = requests.post(f"http://{NcgConfig.config['devportal']['host']}:"
+                             f"{NcgConfig.config['devportal']['port']}{NcgConfig.config['devportal']['uri']}",
+                             headers={'Content-Type': 'application/json'}, data=openapischema)
+
+    return response.status_code, json.loads(response.text)
+
+
+# Builds the declarative JSON for the API Gateway configuration
+# Return a tuple: status, description. If status = 200 things were successful
+def createDevPortal(locationDeclaration: dict):
+    if locationDeclaration['apigateway']['openapi_schema']:
+        status, apiSchemaString = Contrib.GitOps.getObjectFromRepo(
+            content=locationDeclaration['apigateway']['openapi_schema'], base64Encode=False)
+
+        if Contrib.MiscUtils.yaml_or_json(apiSchemaString) == 'yaml':
+            # YAML to JSON conversion
+            apiSchemaString = Contrib.MiscUtils.yaml_to_json(apiSchemaString)
+
+        status, devportalJSON = buildDevPortal(openapischema=apiSchemaString)
+        devportalHTML = base64.b64encode(bytes(devportalJSON['devportal'], 'utf-8')).decode('utf-8')
+
+    return 200, devportalHTML
diff --git a/src/Contrib/GitOps.py b/src/Contrib/GitOps.py
@@ -35,4 +35,4 @@ def getObjectFromRepo(content: str, base64Encode: bool=True):
             else:
                 content = bytes(content, 'utf-8').decode("utf-8")
 
-    return status_code, content
+    return status_code, content
diff --git a/src/V3_CreateConfig.py b/src/V3_CreateConfig.py
@@ -18,6 +18,7 @@
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
 
 import Contrib.APIGateway
+import Contrib.DevPortal
 import Contrib.DeclarationPatcher
 import Contrib.GitOps
 import Contrib.MiscUtils
@@ -60,6 +61,8 @@ def configautosync(configUid):
 def createconfig(declaration: ConfigDeclaration, apiversion: str, runfromautosync: bool = False, configUid: str = ""):
     # Building NGINX configuration for the given declaration
 
+    auxFiles = {'files': [], 'rootDir': NcgConfig.config['nms']['config_dir']}
+
     try:
         # Pydantic JSON validation
         ConfigDeclaration(**declaration.model_dump())
@@ -137,12 +140,26 @@ def createconfig(declaration: ConfigDeclaration, apiversion: str, runfromautosyn
                                 "message": {"status_code": status, "message":
                                     {"code": status, "content": f"invalid HTTP upstream [{loc['upstream']}]"}}}
 
+                    # API Gateway provisioning
                     if loc['apigateway'] and loc['apigateway']['api_gateway']['enabled'] == True:
                         status, apiGatewayConfigDeclaration = (
                             Contrib.APIGateway.createAPIGateway(locationDeclaration=loc))
                     else:
                         apiGatewayConfigDeclaration = ''
 
+                    # API Gateway Developer portal provisioning
+                    if loc['apigateway'] and loc['apigateway']['developer_portal']['enabled'] == True:
+                        status, devPortalHTML = (
+                            Contrib.DevPortal.createDevPortal(locationDeclaration=loc))
+
+                        ### Add optional API Developer portal HTML files
+                        # devPortalHTML
+                        newAuxFile = {'contents': devPortalHTML, 'name': NcgConfig.config['nms']['devportal_dir'] +
+                                                                           loc['apigateway']['developer_portal']['uri']}
+                        auxFiles['files'].append(newAuxFile)
+
+                        ### / Add optional API Developer portal HTML files
+
                     if loc['rate_limit'] is not None:
                         if 'profile' in loc['rate_limit'] and loc['rate_limit']['profile'] and loc['rate_limit'][
                             'profile'] not in all_ratelimits:
@@ -278,8 +295,6 @@ def createconfig(declaration: ConfigDeclaration, apiversion: str, runfromautosyn
                     "message": {"status_code": 400, "message": {"code": 400, "content": "synctime must be >= 0"}},
                     "headers": {'Content-Type': 'application/json'}}
 
-        auxFiles = {'files': [], 'rootDir': NcgConfig.config['nms']['config_dir']}
-
         # Fetch NGINX App Protect WAF policies from source of truth if needed
         d_policies = Contrib.MiscUtils.getDictKey(d, 'output.nms.policies')
         if d_policies is not None:
@@ -360,7 +375,7 @@ def createconfig(declaration: ConfigDeclaration, apiversion: str, runfromautosyn
                                                         server['names'])}
                                     }}
 
-        # Adds optional certificates specified under output.nms.certificates
+        # Add optional certificates specified under output.nms.certificates
         extensions_map = {'certificate': '.crt', 'key': '.key'}
 
         d_certificates = Contrib.MiscUtils.getDictKey(d, 'output.nms.certificates')
@@ -376,7 +391,7 @@ def createconfig(declaration: ConfigDeclaration, apiversion: str, runfromautosyn
                                                                '/' + c['name'] + extensions_map[c['type']]}
                 auxFiles['files'].append(newAuxFile)
 
-        ### / Adds optional certificates specified under output.nms.certificates
+        ### / Add optional certificates specified under output.nms.certificates
 
         # NGINX main configuration file through template
         j2_env = Environment(loader=FileSystemLoader(NcgConfig.config['templates']['root_dir'] + '/' + apiversion),
diff --git a/src/V3_NginxConfigDeclaration.py b/src/V3_NginxConfigDeclaration.py
@@ -403,7 +403,7 @@ class API_Gateway(BaseModel, extra=Extra.forbid):
 
 class DeveloperPortal(BaseModel, extra=Extra.forbid):
     enabled: Optional[bool] = False
-
+    uri: Optional[str] = "/devportal.html"
 
 class APIGateway(BaseModel, extra=Extra.forbid):
     openapi_schema: Optional[str] = ""
diff --git a/src/requirements.txt b/src/requirements.txt
@@ -12,4 +12,4 @@ redis
 uuid
 schedule
 future
-pyyaml
+pyyaml
diff --git a/templates/v3/apigateway.tmpl b/templates/v3/apigateway.tmpl
@@ -53,4 +53,11 @@ location ~ {{ declaration.location.uri }}{{ path.path | regex_replace('{(.*?)}',
 }
 
 {% endfor %}
+
+{% if declaration.location.apigateway.developer_portal.enabled == True -%}
+location = {{ declaration.location.uri }}{{ declaration.location.apigateway.developer_portal.uri }} {
+    rewrite ^{{ declaration.location.uri }}/(.*)$ /$1 break;
+    root {{ ncgconfig.nms.devportal_dir }};
+}
+{% endif %}
 {% endif %}

Original file line number	Diff line number	Diff line change
@@ -30,6 +30,9 @@ USERNAME=`whoami`
`30`	`30`	export USERID=`id -u $USERNAME`
`31`	`31`	export USERGROUP=`id -g $USERNAME`
`32`	`32`
	`33`	`+echo "-> Updating docker images"`
	`34`	`+docker-compose -f $DOCKER_COMPOSE_YAML pull`
	`35`	`+`
`33`	`36`	`echo "-> Deploying NGINX Declarative API"`
`34`	`37`	`COMPOSE_HTTP_TIMEOUT=240 docker-compose -p $PROJECT_NAME -f $DOCKER_COMPOSE_YAML up -d --remove-orphans`
`35`	`38`	`}`