Document Adding indexes (#3594)

Author: r4victor

contributing/MIGRATIONS.md

@@ -47,3 +47,29 @@ These steps apply to **renaming a column** or **changing the type of a column**
 ### Altering multiple tables
 
 Altering a table requires Postgres to [take an ACCESS EXCLUSIVE lock](https://www.postgresql.org/docs/current/sql-altertable.html). (This applies not only to statements that rewrite the tables but also to statements that modify tables metadata.) Altering multiple tables can cause deadlocks due to conflict with read operations since the `dstack` server does not define an order for read operations. Altering multiple tables should be done in separate transactions/migrations.
+
+### Adding indexes
+
+Use `CREATE INDEX CONCURRENTLY` to avoid tacking exclusive lock on the table for a long time.
+For migrations that create multiple indexes, failures can leave the schema in a partial state
+(some indexes already created, some missing). On Postgres, concurrent index creation can also fail
+midway and leave an invalid index object with the same name. Retrying the migration then fails
+with "already exists".
+
+For retry-safe migrations, pre-drop indexes with `if_exists=True` before creating them again:
+
+```python
+with op.get_context().autocommit_block():
+    op.drop_index(
+        "ix_table_col",
+        table_name="table",
+        if_exists=True,
+        postgresql_concurrently=True,
+    )
+    op.create_index(
+        "ix_table_col",
+        "table",
+        ["col"],
+        postgresql_concurrently=True,
+    )
+```

contributing/RELEASE.md

@@ -8,7 +8,10 @@ This is a `dstack` release guide and checklist for core maintainers.
     1. Compare changes to the previous release, e.g. [`https://github.com/dstackai/dstack/compare/0.19.39...master`](https://github.com/dstackai/dstack/compare/0.19.39...master).
     2. Test that `master` CLI works with the previous server release. PRs that add new model fields can potentially break client backward compatibility.
     3. Test that `master` server works with the previous CLI release.
-    4. Pay special attention to releases with DB migrations. Migrations should work with rolling deployments and avoid locking multiple tables. See [MIGRATIONS.md](MIGRATIONS.md).
+    4. Pay special attention to releases with DB migrations. See [MIGRATIONS.md](MIGRATIONS.md).
+      * Ensure migrations work with rolling deployments and do not lock multiple tables.
+      * Test applying migrations while old replicas do active processing.
+      * Test migrations can be retried if they fail. For example, concurrent index may fail and stay in invalid state.
 2. Create a tag, e.g. `git tag 0.19.40`.
 3. Push the tag to trigger the Release `workflow`, i.e. `git push --tags`.
 4. Generate GitHub release notes from the tag. Highlight major features, deprecations, breaking changes.