Version 1.0.2

.gitignore

@@ -1,5 +1,6 @@
 config.py
 run.py
+migrations/
 
 # PyCharm
 .idea/

README.md

@@ -38,12 +38,15 @@ See how is ExaFS integrated into the network in the picture below.
 
 The central part of the ExaFS is a web application, written in Python3.6 with Flask framework. It provides a user interface for ExaBGP rule CRUD operations. The application also provides the REST API with CRUD operations for the configuration rules. The web app uses Shibboleth authorization; the REST API is using token-based authorization. 
 
-The app creates the ExaBGP commands and forwards them to ExaAPI. All rules are carefully validated, and only valid rules are stored in the database and sent to the ExaBGP connector.
+The app creates the ExaBGP commands and forwards them to ExaBGP process. All rules are carefully validated, and only valid rules are stored in the database and sent to the ExaBGP connector.
 
-This second part of the system is another web application that replicates the received command to the stdout. The connection between ExaBGP daemon and stdout of ExaAPI is specified in the ExaBGP config. 
+This second part of the system is another application that replicates the received command to the stdout. The connection between ExaBGP daemon and stdout of ExaAPI (ExaBGP process) is specified in the ExaBGP config. 
+
+This API was a part of the project, but now has been moved to own repository. You can use [pip package exabgp-process](https://pypi.org/project/exabgp-process/) or clone the git repo. Or you can create your own version.
 
-Every time this API gets a command from ExaFS, it replicates this command to the ExaBGP daemon through the stdout. The registered daemon then updates the ExaBGP table – create, modify or remove the rule from command.
-Last part of the system is Guarda service. This systemctl service is running in the host system and gets a notification on each restart of ExaBGP service via systemctl WantedBy config option. For every restart of ExaBGP the Guarda service will put all the valid and active rules to the ExaBGP rules table again.
+Every time this process gets a command from ExaFS, it replicates this command to the ExaBGP service through the stdout. The registered service then updates the ExaBGP table – create, modify or remove the rule from command.
+
+You may also need to monitor the ExaBGP and renew the commands after restart / shutdown. In docs you can find and example of system service named Guarda. This systemctl service is running in the host system and gets a notification on each restart of ExaBGP service via systemctl WantedBy config option. For every restart of ExaBGP the Guarda service will put all the valid and active rules to the ExaBGP rules table again.
 
 ## DOCS
 * [Install notes](./docs/INSTALL.md)
@@ -52,9 +55,19 @@ Last part of the system is Guarda service. This systemctl service is running in
 * [Local database instalation notes](./docs/DB_LOCAL.md)
 
 ## Change Log
-- 0.8.1 application is using Flask-Session stored in DB using SQL Alchemy driver. This can be configured for other
-drivers, however server side session is required for the application proper function.
-- 0.8.0 - API keys update.  **Run migration scripts to update your DB**.  Keys can now have expiration date and readonly flag. Admin can create special keys for certain machines.
+- 1.0.2 - fixed bug in IPv6 Flowspec messages
+- 1.0.1 . minor bug fixes
+- 1.0.0 . Major changes
+    - Limits for nuber of rules in the system introduced. There are now limits for rules for organization and overall limit for the instalation. Database changed / migration is required. Migrating the database to version 1.0.x is a bit more complicated, you need to link existing rules to organizations. [A more detailed description is in a separate document](./docs/DB_MIGRATIONS.md).
+    - Rules are now tied to organization. If the user belongs to more than one organization, the organization for the session must be selected after login.
+    - Bulk import for users enabled for admin.
+    - Introduced Swagger docs for API on the local system. Just open /apidocs url. 
+    - New format of message for ExaAPI - now sends information about author of rule (user) for logging purposes.
+    - ExaAPI and Guarda modules moved outside of the project.
+    - ExaAPI is now available as a [pip package exabgp-process](https://pypi.org/project/exabgp-process/), with own [github repostiory](https://github.com/CESNET/exabgp-process).
+    - Watch of exabgp restart can be still done by guarda service - see docs. Or it can be done by override of the exabgp service settings.  
+- 0.8.1 application is using Flask-Session stored in DB using SQL Alchemy driver. This can be configured for other drivers, however server side session is required for the application proper function.
+- 0.8.0 - API keys update.  **Run migration scripts to update your DB**.  Keys can now have expiration date and readonly flag. Admin can create special keys for certain machinnes.
 - 0.7.3 - New possibility of external auth proxy. 
 - 0.7.2 - Dashboard and Main menu are now customizable in config. App is ready to be packaged using setup.py.
 - 0.7.0 - ExaAPI now have two options - HTTP or RabbitMQ. ExaAPI process has been renamed, update of ExaBGP process value is needed for this version.

config.example.py

@@ -1,30 +1,39 @@
-class Config():
+class Config:
     """
     Default config options
     """
 
+    # Limits
+    FLOWSPEC4_MAX_RULES = 9000
+    FLOWSPEC6_MAX_RULES = 9000
+    RTBH_MAX_RULES = 100000
+
     # Flask debugging
     DEBUG = True
     # Flask testing
     TESTING = False
-    # SSO auth enabled
 
-    SSO_AUTH = True
+    # Choose your authentication method and set it to True here or
+    # the production / development config
+    # SSO auth enabled
+    SSO_AUTH = False
     # Authentication is done outside the app, use HTTP header to get the user uuid.
     # If SSO_AUTH is set to True, this option is ignored and SSO auth is used.
     HEADER_AUTH = False
+    # Local authentication - used when SSO_AUTH and HEADER_AUTH are set to False
+    LOCAL_AUTH = False
 
     # Name of HTTP header containing the UUID of authenticated user.
     # Only used when HEADER_AUTH is set to True
-    AUTH_HEADER_NAME = 'X-Authenticated-User'
+    AUTH_HEADER_NAME = "X-Authenticated-User"
     # SSO LOGOUT
     LOGOUT_URL = "https://flowspec.example.com/Shibboleth.sso/Logout"
     # SQL Alchemy config
     SQLALCHEMY_TRACK_MODIFICATIONS = False
 
     # ExaApi configuration
     # possible values HTTP, RABBIT
-    EXA_API = "HTTP"
+    EXA_API = "RABBIT"
     # for HTTP EXA_API_URL must be specified
     EXA_API_URL = "http://localhost:5000/"
     # for RABBITMQ EXA_API_RABBIT_* must be specified
@@ -39,24 +48,6 @@ class Config():
     JWT_SECRET = "GenerateSomeLongRandomSequence"
     SECRET_KEY = "GenerateSomeLongRandomSequence"
 
-    # LOCAL user parameters - when the app is used without SSO_AUTH
-    # Defined in User model
-    LOCAL_USER_UUID = "admin@example.com"
-    # Defined in User model
-    LOCAL_USER_ID = 1
-    # Defined in Role model / default 1 - view, 2 - normal user, 3 - admin
-    LOCAL_USER_ROLES = ["admin"]
-    # Defined in Organization model
-    # List of organizations for the local user. There can be many of them.
-    # Define the name and the adress range. The range is then used for first data insert
-    # after the tables are created with db-init.py script.
-    LOCAL_USER_ORGS = [
-        {"name": "Example Org.", "arange": "192.168.0.0/16\n2121:414:1a0b::/48"},
-    ]
-    # Defined in Role model / default 1 - view, 2 - normal user, 3 - admin
-    LOCAL_USER_ROLE_IDS = [3]
-    # Defined in Organization model
-    LOCAL_USER_ORG_IDS = [1]
     # APP Name - display in main toolbar
     APP_NAME = "ExaFS"
     # Route Distinguisher for VRF
@@ -75,10 +66,26 @@ class ProductionConfig(Config):
     SQLALCHEMY_DATABASE_URI = "Your Productionl Database URI"
     # Public IP of the production machine
     LOCAL_IP = "127.0.0.1"
+    LOCAL_IP6 = "::ffff:127.0.0.1"
     # SSO AUTH enabled in produciion
     SSO_AUTH = True
+    SSO_ATTRIBUTE_MAP = {
+        "eppn": (False, "eppn"),
+        "HTTP_X_EPPN": (False, "eppn"),
+    }
+    SSO_LOGIN_URL = "/login"
+
     # Set true if you need debug in production
     DEBUG = False
+    DEVEL = False
+
+    # is production behind a reverse proxy?
+    BEHIND_PROXY = True
+
+    # Set cookie behavior
+    SESSION_COOKIE_SECURE = (True,)
+    SESSION_COOKIE_HTTPONLY = (True,)
+    SESSION_COOKIE_SAMESITE = ("Lax",)
 
 
 class DevelopmentConfig(Config):
@@ -88,7 +95,14 @@ class DevelopmentConfig(Config):
 
     SQLALCHEMY_DATABASE_URI = "Your Local Database URI"
     LOCAL_IP = "127.0.0.1"
+    LOCAL_IP6 = "::ffff:127.0.0.1"
     DEBUG = True
+    DEVEL = True
+
+    # LOCAL user parameters - when the app is used without SSO_AUTH
+    # Local User must be in the database
+    LOCAL_USER_UUID = "admin@example.com"
+    LOCAL_AUTH = True
 
 
 class TestingConfig(Config):

docs/API.md

@@ -1,4 +1,5 @@
-# ExaFS tool
-## API 
+# ExaFS API  
 
-ExaFS API documentation can be [found on Apiary.io](https://exafs.docs.apiary.io/#).
+Local Swagger API docs is available on https://your.server.name/apidocs
+
+ExaFS API documentation can be also [found on Apiary.io](https://exafs.docs.apiary.io/#).

docs/AUTH.md

@@ -1,5 +1,4 @@
-# ExaFS tool
-## Auth mechanism
+# ExaFS Auth mechanism
 
 Since version 0.7.3, the application supports three different forms of user authorization. 
 

docs/DB_MIGRATIONS.md

@@ -0,0 +1,35 @@
+# How to Upgrade the Database  
+
+## General Guidelines  
+Migrations can be inconsistent. To avoid issues, we removed migrations from git repostory. To start the migration on your server, it is recomended reset the migration state on the server and run the migration based on the updated database models when switching application versions via Git.  
+
+```bash
+rm -rf migrations/
+```
+
+```SQL
+DROP TABLE alembic_version;
+```
+
+```bash
+flask db init
+flask db migrate -m "Initial migration based on current DB state"
+flask db upgrade
+```
+
+## Steps for Upgrading to v1.0.x
+Limits for number of rules were introduced. Some database engines (Mariadb 10.x for example) have issue to set Non Null foreigin key to 0 and automatic migrations fail. The solution may be in diferent version (Mariadb 11.x works fine), or to set limits in db manually later.
+
+To set the limit to 0 for existing organizations run
+
+```SQL
+UPDATE organization 
+SET limit_flowspec4 = 0, limit_flowspec6 = 0, limit_rtbh = 0 
+WHERE limit_flowspec4 IS NULL OR limit_flowspec6 IS NULL OR limit_rtbh IS NULL;
+```
+
+In all cases we need later assign rules to organizations. There's an admin endpoint for this:  
+
+`https://yourexafs.url/admin/set-org-if-zero`
+
+Or you can start with clean database and manually migrate data by SQL dump later. Feel free to contact jiri.vrany@cesnet.cz if you need help with the DB migration to 1.0.x. 

docs/INSTALL.md

@@ -100,7 +100,8 @@ systemctl start httpd
 
 #### Supervisord - install as root
 
-Supervisord is used to run and manage application.
+Supervisord is used to run and manage applications, but it is not mandatory for deployment.
+You can skip this section if you are using a different deployment method, such as Docker.
 
 1. install:
    `pip install supervisor`
@@ -112,9 +113,9 @@ Supervisord is used to run and manage application.
 
 
 3. setup as service:
-    `cp supervisord.example.service /usr/lib/systemd/system/supervisord.service`
+    `cp docs/supervisor/supervisord.example.service /usr/lib/systemd/system/supervisord.service`
 4. copy exafs.supervisord.conf to /etc/supervisord/
-  `cp exafs.supervisord.conf /etc/supervisord/conf.d/`
+  `cp docs/supervisor/exafs.supervisord.conf /etc/supervisord/conf.d/`
 5. start service
    `systemctl start supervisord`
 6. view service status: