Skip to content

Support Async Queries via Celery #698

@maltesander

Description

@maltesander

Superset can use async queries to support long running queries.

This requires to add at least one new role, a worker.

This is a small spike where the rolegroups are used to provide the webserver (our standard superset), the worker and the beat which i found to not be required :

---
apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
  name: superset
spec:
  image:
    productVersion: 4.1.4
  clusterConfig:
    credentialsSecret: superset-credentials
    mapboxSecret: superset-mapbox-api-key
  nodes:
    envOverrides:
      FILTER_STATE_CACHE_CONFIG: |-
        {
          'CACHE_TYPE': 'RedisCache',
          'CACHE_DEFAULT_TIMEOUT': 86400,
          'CACHE_KEY_PREFIX': 'superset_filter_cache',
          'CACHE_REDIS_URL': 'redis://:{{ redisPassword }}@redis-superset-master:6379'
        }
    configOverrides:
      superset_config.py:
        EXPERIMENTAL_FILE_HEADER: |-
          from flask_caching.backends.rediscache import RedisCache
        EXPERIMENTAL_FILE_FOOTER: |-
          RESULTS_BACKEND = RedisCache(host='redis-superset-master', port=6379, key_prefix='superset_results', password='{{ redisPassword }}')

          class CeleryConfig(object):
            broker_url = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
            imports = (
                "superset.sql_lab",
                "superset.tasks.scheduler",
            )
            result_backend = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
            worker_prefetch_multiplier = 10
            task_acks_late = True
            task_annotations = {
                "sql_lab.get_sql_results": {
                    "rate_limit": "100/s",
                },
            }

          CELERY_CONFIG = CeleryConfig
    roleConfig:
      listenerClass: external-stable
    roleGroups:
      webserver:
        replicas: 1
      worker:
        replicas: 1
        podOverrides:
          spec:
            containers:
              - name: superset
                args:
                  - |2
                    mkdir --parents /stackable/app/pythonpath
                    cp /stackable/config/* /stackable/app/pythonpath
                    cp /stackable/log_config/log_config.py /stackable/app/pythonpath
                    celery --app=superset.tasks.celery_app:app worker --pool=prefork -O fair -c 4
                env:
                  - name: SUPERSET_CONFIG_PATH
                    value: /stackable/app/pythonpath/superset_config.py
      beat:
        replicas: 1
        podOverrides:
          spec:
            containers:
              - name: superset
                args:
                  - |2
                    mkdir --parents /stackable/app/pythonpath
                    cp /stackable/config/* /stackable/app/pythonpath
                    cp /stackable/log_config/log_config.py /stackable/app/pythonpath
                    celery --app=superset.tasks.celery_app:app beat
                env:
                  - name: SUPERSET_CONFIG_PATH
                    value: /stackable/app/pythonpath/superset_config.py

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions