The DoTS search API

Elasticsearch API to search the DoTS collections & resources.

Prerequisite - Install Elasticsearch

Install Elasticsearch and its ICU plugin

⚠️ Use an ES version compatible with requirements.txt ℹ️ Below commands are run independently/outside virtual environments (deactivate)

• Elasticsearch: refer to your organisation instructions or Elasticsearch guidelines

• ICU plugin: check if ICU is installed with uconv -V, otherwise:

  path/to/elasticsearch_folder/bin/elasticsearch-plugin install analysis-icu

• With docker (security disabled)

  
      docker run --name dots-es -d -p 9200:9200 -e "discovery.type=single-node" -e "xpack.security.enabled=false" -e "xpack.security.http.ssl.enabled=false" elasticsearch:8.12.1
      docker exec dots-es bash -c "bin/elasticsearch-plugin install analysis-icu"
      docker restart dots-es
    

Install

• Clone the GitHub repository in your projects' folder:

  cd path/to/projects_folder/
  git clone https://github.com/chartes/dots-cli-es.git

• Ensure you are running Python 3.12, for example with pyenv:

  pyenv shell 3.12

• Set up the virtual environment:

  
  cd path/to/projects_folder/dots-cli-es
  python3 -m venv your_venv_name
  source your_venv_name/bin/activate
  pip install -r requirements.txt
  

• For servers requiring uWSGI to run Python apps (remote Nginx servers):

  • check if uWSGI is installed pip list --local
  • install it in your virtual your_venv_name if it's not: pip install uwsgi. The WSGI application is located at flask_app:flask_app NB : this command may require wheel:
    • to check whether wheel is installed: pip show wheel
    • to install it if required: pip install wheel

Indexing

• Install Elasticsearch and create indices if they are not available:

• Follow the ES installation & initial indexing instructions above

⚠️ Below (re)indexing commands are run within the app virtual environment: reactivate the virtual environment if needed (source your_venv_name/bin/activate)

In below commands, options are indicated within brackets (option). Remove them as required.

With ES security enabled, theES_PASSWORD option is required in commands below.

Initial indexing (and reindexing without configuration changes):

(ES_PASSWORD=ELASTIC_PASSWORD) python3 manage.py (--config=local/staging/prod) index (--years "YYYY-YYYY")

When the index doesn't exist it is created according to the project ES configuration files.

Updating index configuration

This operation will delete the pre-existing index.

(ES_PASSWORD=ELASTIC_PASSWORD) python3 manage.py (--config=local/staging/prod) update-conf --rebuild --indexes=dots_document/dots_collection

The above command updates the indexes according to the project ES configuration files.

Check created indexes:

• with ES security disabled:

curl -X POST "http://localhost:9200/dots_document/_refresh?pretty"
curl http://localhost:9200/_cat/indices?v

• with ES security enabled:

curl -X POST "http://elastic:ELASTIC_PASSWORD@localhost:9200/dots_document/_refresh?pretty"
curl http://elastic:ELASTIC_PASSWORD@localhost:9200/_cat/indices?v

Launch the app:

⚠️ Below commands are mainly for local launch. For servers, apps may be started via processes management tools, refer to the servers documentation

• Reactivate the virtual environment if needed (source your_venv_name/bin/activate)
• Launch: from the subfolder containing flask_app.py (cd path/to/dots-cli-es) (ES_PASSWORD=ELASTIC_PASSWORD) python3 flask_app.py
• Then visit http://localhost:5003/api/1.0/search?query=*&index=dots_document to test it is running

Launch the front-end:

• Front-end's Readme

---

Additional details for offline commands:

python3 manage.py --help

Usage: manage.py [OPTIONS] COMMAND [ARGS]...

Options:
  --config [local|staging|prod]  select appropriate .env file to use
                                 [default: staging]
  --help                         Show this message and exit.

Commands:
  delete       Delete the indexes
  index        Rebuild the elasticsearch indexes
  search       Perform a search using the provided query.
  update-conf  Update the index configuration and mappings