优化sqlite读写性能，优化对话数据页面和会话管理页面的交互与性能 #2952
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
嘿，你好——我已经审阅了你的修改——以下是一些反馈：
- 考虑将
AbortController + fetch逻辑提取到一个可重用的辅助函数或可组合项中，以避免在多个方法中重复请求中止模式。 - 将硬编码的 SQLite PRAGMA 设置移动到配置或环境变量中可能会更灵活，这样你就可以在不修改代码的情况下调整性能。
- 确保新的
cid列已正确索引并包含在你的后端过滤逻辑中，以防止在按该字段排序或搜索时进行全表扫描。
供 AI 代理使用的提示
请处理此代码审查中的评论：
## 总体评论
- 考虑将 `AbortController + fetch` 逻辑提取到一个可重用的辅助函数或可组合项中，以避免在多个方法中重复请求中止模式。
- 将硬编码的 SQLite PRAGMA 设置移动到配置或环境变量中可能会更灵活，这样你就可以在不修改代码的情况下调整性能。
- 确保新的 `cid` 列已正确索引并包含在你的后端过滤逻辑中，以防止在按该字段排序或搜索时进行全表扫描。
## 个人评论
### 评论 1
<location> `dashboard/src/views/ConversationPage.vue:81` </location>
<code_context>
</v-chip>
</template>
+ <template v-slot:item.cid="{ item }">
+ <span class="text-truncate">{{ item.cid || tm('status.unknown') }}</span>
+ </template>
</code_context>
<issue_to_address>
**建议：** 截断会话 ID 可能会影响某些工作流的可用性。
考虑添加工具提示或类似功能，允许用户查看或复制完整的 ID。
建议的实现：
```
<template v-slot:item.cid="{ item }">
<v-tooltip location="top" open-delay="300">
<template #activator="{ props }">
<span class="text-truncate" v-bind="props" style="max-width: 180px; display: inline-block;">
{{ item.cid || tm('status.unknown') }}
</span>
</template>
<div style="display: flex; align-items: center;">
<span style="word-break: break-all;">{{ item.cid || tm('status.unknown') }}</span>
<v-btn
v-if="item.cid"
icon="mdi-content-copy"
size="x-small"
variant="text"
@click.stop="copyToClipboard(item.cid)"
:aria-label="tm('actions.copy')"
class="ml-2"
></v-btn>
</div>
</v-tooltip>
</template>
```
```
<script>
```
如果你的 `<script>` 部分中尚不存在 `copyToClipboard` 方法，则需要实现它。例如：
```js
methods: {
copyToClipboard(text) {
navigator.clipboard.writeText(text);
this.$toast?.success?.(this.tm('actions.copied')) // or use your preferred notification method
},
// ...other methods
}
```
另外，请确保你的项目中可以从 Vuetify 获得 `v-tooltip` 和 `v-btn` 组件。
</issue_to_address>
### 评论 2
<location> `astrbot/core/db/sqlite.py:35-40` </location>
<code_context>
"""Initialize the database by creating tables if they do not exist."""
async with self.engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
+ await conn.execute(text("PRAGMA journal_mode=WAL"))
+ await conn.execute(text("PRAGMA synchronous=NORMAL"))
+ await conn.execute(text("PRAGMA cache_size=50000"))
+ await conn.execute(text("PRAGMA temp_store=MEMORY"))
+ await conn.execute(text("PRAGMA mmap_size=268435456"))
+ await conn.execute(text("PRAGMA optimize"))
await conn.commit()
</code_context>
<issue_to_address>
**建议（性能）：** 应用多个 PRAGMA 语句可能会产生意想不到的副作用。
这些 PRAGMA 设置可能会影响持久性、并发性和内存使用。请审查并记录它们，以确认它们符合部署需求。
```suggestion
# 将 SQLite 设置为使用预写式日志 (WAL) 以提高并发性。
# WAL 模式可能会提高性能，但在崩溃时可能会影响持久性。
await conn.execute(text("PRAGMA journal_mode=WAL"))
# 将同步模式设置为 NORMAL 以加快写入速度。
# 与 FULL 相比，这会降低持久性保证。
await conn.execute(text("PRAGMA synchronous=NORMAL"))
# 将缓存大小增加到 50000 页（如果页面大小为 1KB，则约为 50MB）。
# 更高的缓存大小可提高读取性能，但会增加内存使用量。
await conn.execute(text("PRAGMA cache_size=50000"))
# 将临时表和索引存储在内存中以加快访问速度。
# 可能会增加内存使用量。
await conn.execute(text("PRAGMA temp_store=MEMORY"))
# 将内存映射 I/O 大小设置为 256MB。
# 可以提高性能，但会增加内存使用量。
await conn.execute(text("PRAGMA mmap_size=268435456"))
# 运行 SQLite 的内部优化器。
# 可能会提高查询性能。
await conn.execute(text("PRAGMA optimize"))
# 注意：这些 PRAGMA 设置会影响持久性、并发性和内存使用。
# 请在生产环境中使用前审查并确认它们符合部署要求。
```
</issue_to_address>
### 评论 3
<location> `astrbot/core/db/sqlite.py:169` </location>
<code_context>
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
+ col(ConversationV2.conversation_id).ilike(f"%{search_query}%"),
)
)
</code_context>
<issue_to_address>
**建议（性能）：** 将 conversation_id 添加到搜索中可能会影响查询性能。
监控查询性能，如果需要，请在 conversation_id 上添加索引以防止速度变慢。
建议的实现：
```python
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
# 注意：按 conversation_id 搜索可能会影响查询性能。
# 监控查询性能，如果需要，请在 conversation_id 上添加索引。
col(ConversationV2.conversation_id).ilike(f"%{search_query}%"),
```
```python
)
# 记录查询性能以进行监控
import logging, time
logger = logging.getLogger("astrbot.db")
start_time = time.time()
```
```python
)
elapsed = time.time() - start_time
if elapsed > 1.0: # 如果查询时间超过 1 秒，则记录
logger.warning(f"使用 conversation_id 的搜索查询耗时 {elapsed:.2f}s。如果这种情况很常见，请考虑添加索引。")
```
</issue_to_address>帮助我更有用！请点击每个评论上的 👍 或 👎，我将使用反馈来改进你的评论。
Original comment in English
Hey there - I've reviewed your changes - here's some feedback:
- Consider extracting the AbortController + fetch logic into a reusable helper or composable to avoid duplicating the request-abort pattern across multiple methods.
- It might be more flexible to move the hardcoded SQLite PRAGMA settings into configuration or environment variables so you can tune performance without modifying code.
- Ensure the new
cidcolumn is properly indexed and included in your backend filtering logic to prevent full table scans when sorting or searching by that field.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider extracting the AbortController + fetch logic into a reusable helper or composable to avoid duplicating the request-abort pattern across multiple methods.
- It might be more flexible to move the hardcoded SQLite PRAGMA settings into configuration or environment variables so you can tune performance without modifying code.
- Ensure the new `cid` column is properly indexed and included in your backend filtering logic to prevent full table scans when sorting or searching by that field.
## Individual Comments
### Comment 1
<location> `dashboard/src/views/ConversationPage.vue:81` </location>
<code_context>
</v-chip>
</template>
+ <template v-slot:item.cid="{ item }">
+ <span class="text-truncate">{{ item.cid || tm('status.unknown') }}</span>
+ </template>
</code_context>
<issue_to_address>
**suggestion:** Truncating the conversation ID may hinder usability for some workflows.
Consider adding a tooltip or similar feature to allow users to view or copy the full ID.
Suggested implementation:
```
<template v-slot:item.cid="{ item }">
<v-tooltip location="top" open-delay="300">
<template #activator="{ props }">
<span class="text-truncate" v-bind="props" style="max-width: 180px; display: inline-block;">
{{ item.cid || tm('status.unknown') }}
</span>
</template>
<div style="display: flex; align-items: center;">
<span style="word-break: break-all;">{{ item.cid || tm('status.unknown') }}</span>
<v-btn
v-if="item.cid"
icon="mdi-content-copy"
size="x-small"
variant="text"
@click.stop="copyToClipboard(item.cid)"
:aria-label="tm('actions.copy')"
class="ml-2"
></v-btn>
</div>
</v-tooltip>
</template>
```
```
<script>
```
You will need to implement the `copyToClipboard` method in your `<script>` section if it does not already exist. For example:
```js
methods: {
copyToClipboard(text) {
navigator.clipboard.writeText(text);
this.$toast?.success?.(this.tm('actions.copied')) // or use your preferred notification method
},
// ...other methods
}
```
Also, ensure that the `v-tooltip` and `v-btn` components are available from Vuetify in your project.
</issue_to_address>
### Comment 2
<location> `astrbot/core/db/sqlite.py:35-40` </location>
<code_context>
"""Initialize the database by creating tables if they do not exist."""
async with self.engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
+ await conn.execute(text("PRAGMA journal_mode=WAL"))
+ await conn.execute(text("PRAGMA synchronous=NORMAL"))
+ await conn.execute(text("PRAGMA cache_size=50000"))
+ await conn.execute(text("PRAGMA temp_store=MEMORY"))
+ await conn.execute(text("PRAGMA mmap_size=268435456"))
+ await conn.execute(text("PRAGMA optimize"))
await conn.commit()
</code_context>
<issue_to_address>
**suggestion (performance):** Applying multiple PRAGMA statements may have unintended side effects.
These PRAGMA settings may impact durability, concurrency, and memory usage. Please review and document them to confirm they meet deployment needs.
```suggestion
# Set SQLite to use Write-Ahead Logging for improved concurrency.
# WAL mode may increase performance but can affect durability in case of crashes.
await conn.execute(text("PRAGMA journal_mode=WAL"))
# Set synchronous mode to NORMAL for faster writes.
# This reduces durability guarantees compared to FULL.
await conn.execute(text("PRAGMA synchronous=NORMAL"))
# Increase cache size to 50000 pages (~50MB if page size is 1KB).
# Higher cache size improves read performance but increases memory usage.
await conn.execute(text("PRAGMA cache_size=50000"))
# Store temporary tables and indices in memory for faster access.
# May increase memory usage.
await conn.execute(text("PRAGMA temp_store=MEMORY"))
# Set memory-mapped I/O size to 256MB.
# Can improve performance but increases memory usage.
await conn.execute(text("PRAGMA mmap_size=268435456"))
# Run SQLite's internal optimizer.
# May improve query performance.
await conn.execute(text("PRAGMA optimize"))
# NOTE: These PRAGMA settings impact durability, concurrency, and memory usage.
# Please review and confirm they meet deployment requirements before use in production.
```
</issue_to_address>
### Comment 3
<location> `astrbot/core/db/sqlite.py:169` </location>
<code_context>
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
+ col(ConversationV2.conversation_id).ilike(f"%{search_query}%"),
)
)
</code_context>
<issue_to_address>
**suggestion (performance):** Adding conversation_id to search may impact query performance.
Monitor query performance and add an index on conversation_id if needed to prevent slowdowns.
Suggested implementation:
```python
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
# NOTE: Searching by conversation_id may impact query performance.
# Monitor query performance and add an index on conversation_id if needed.
col(ConversationV2.conversation_id).ilike(f"%{search_query}%"),
```
```python
)
# Log query performance for monitoring
import logging, time
logger = logging.getLogger("astrbot.db")
start_time = time.time()
```
```python
)
elapsed = time.time() - start_time
if elapsed > 1.0: # Log if query takes longer than 1 second
logger.warning(f"Search query with conversation_id took {elapsed:.2f}s. Consider adding an index if this is common.")
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
astrbot/core/db/sqlite.py
Outdated
Comment on lines
35
to
40
| await conn.execute(text("PRAGMA journal_mode=WAL")) | ||
| await conn.execute(text("PRAGMA synchronous=NORMAL")) | ||
| await conn.execute(text("PRAGMA cache_size=50000")) | ||
| await conn.execute(text("PRAGMA temp_store=MEMORY")) | ||
| await conn.execute(text("PRAGMA mmap_size=268435456")) | ||
| await conn.execute(text("PRAGMA optimize")) |
Contributor
There was a problem hiding this comment.
建议（性能）： 应用多个 PRAGMA 语句可能会产生意想不到的副作用。
这些 PRAGMA 设置可能会影响持久性、并发性和内存使用。请审查并记录它们，以确认它们符合部署需求。
Suggested change
| await conn.execute(text("PRAGMA journal_mode=WAL")) | |
| await conn.execute(text("PRAGMA synchronous=NORMAL")) | |
| await conn.execute(text("PRAGMA cache_size=50000")) | |
| await conn.execute(text("PRAGMA temp_store=MEMORY")) | |
| await conn.execute(text("PRAGMA mmap_size=268435456")) | |
| await conn.execute(text("PRAGMA optimize")) | |
| # 将 SQLite 设置为使用预写式日志 (WAL) 以提高并发性。 | |
| # WAL 模式可能会提高性能，但在崩溃时可能会影响持久性。 | |
| await conn.execute(text("PRAGMA journal_mode=WAL")) | |
| # 将同步模式设置为 NORMAL 以加快写入速度。 | |
| # 与 FULL 相比，这会降低持久性保证。 | |
| await conn.execute(text("PRAGMA synchronous=NORMAL")) | |
| # 将缓存大小增加到 50000 页（如果页面大小为 1KB，则约为 50MB）。 | |
| # 更高的缓存大小可提高读取性能，但会增加内存使用量。 | |
| await conn.execute(text("PRAGMA cache_size=50000")) | |
| # 将临时表和索引存储在内存中以加快访问速度。 | |
| # 可能会增加内存使用量。 | |
| await conn.execute(text("PRAGMA temp_store=MEMORY")) | |
| # 将内存映射 I/O 大小设置为 256MB。 | |
| # 可以提高性能，但会增加内存使用量。 | |
| await conn.execute(text("PRAGMA mmap_size=268435456")) | |
| # 运行 SQLite 的内部优化器。 | |
| # 可能会提高查询性能。 | |
| await conn.execute(text("PRAGMA optimize")) | |
| # 注意：这些 PRAGMA 设置会影响持久性、并发性和内存使用。 | |
| # 请在生产环境中使用前审查并确认它们符合部署要求。 |
Original comment in English
suggestion (performance): Applying multiple PRAGMA statements may have unintended side effects.
These PRAGMA settings may impact durability, concurrency, and memory usage. Please review and document them to confirm they meet deployment needs.
Suggested change
| await conn.execute(text("PRAGMA journal_mode=WAL")) | |
| await conn.execute(text("PRAGMA synchronous=NORMAL")) | |
| await conn.execute(text("PRAGMA cache_size=50000")) | |
| await conn.execute(text("PRAGMA temp_store=MEMORY")) | |
| await conn.execute(text("PRAGMA mmap_size=268435456")) | |
| await conn.execute(text("PRAGMA optimize")) | |
| # Set SQLite to use Write-Ahead Logging for improved concurrency. | |
| # WAL mode may increase performance but can affect durability in case of crashes. | |
| await conn.execute(text("PRAGMA journal_mode=WAL")) | |
| # Set synchronous mode to NORMAL for faster writes. | |
| # This reduces durability guarantees compared to FULL. | |
| await conn.execute(text("PRAGMA synchronous=NORMAL")) | |
| # Increase cache size to 50000 pages (~50MB if page size is 1KB). | |
| # Higher cache size improves read performance but increases memory usage. | |
| await conn.execute(text("PRAGMA cache_size=50000")) | |
| # Store temporary tables and indices in memory for faster access. | |
| # May increase memory usage. | |
| await conn.execute(text("PRAGMA temp_store=MEMORY")) | |
| # Set memory-mapped I/O size to 256MB. | |
| # Can improve performance but increases memory usage. | |
| await conn.execute(text("PRAGMA mmap_size=268435456")) | |
| # Run SQLite's internal optimizer. | |
| # May improve query performance. | |
| await conn.execute(text("PRAGMA optimize")) | |
| # NOTE: These PRAGMA settings impact durability, concurrency, and memory usage. | |
| # Please review and confirm they meet deployment requirements before use in production. |
astrbot/core/db/sqlite.py
Outdated
| col(ConversationV2.title).ilike(f"%{search_query}%"), | ||
| col(ConversationV2.content).ilike(f"%{search_query}%"), | ||
| col(ConversationV2.user_id).ilike(f"%{search_query}%"), | ||
| col(ConversationV2.conversation_id).ilike(f"%{search_query}%"), |
Contributor
There was a problem hiding this comment.
建议（性能）： 将 conversation_id 添加到搜索中可能会影响查询性能。
监控查询性能，如果需要，请在 conversation_id 上添加索引以防止速度变慢。
建议的实现：
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
# 注意：按 conversation_id 搜索可能会影响查询性能。
# 监控查询性能，如果需要，请在 conversation_id 上添加索引。
col(ConversationV2.conversation_id).ilike(f"%{search_query}%"), )
# 记录查询性能以进行监控
import logging, time
logger = logging.getLogger("astrbot.db")
start_time = time.time() )
elapsed = time.time() - start_time
if elapsed > 1.0: # 如果查询时间超过 1 秒，则记录
logger.warning(f"使用 conversation_id 的搜索查询耗时 {elapsed:.2f}s。如果这种情况很常见，请考虑添加索引。")Original comment in English
suggestion (performance): Adding conversation_id to search may impact query performance.
Monitor query performance and add an index on conversation_id if needed to prevent slowdowns.
Suggested implementation:
col(ConversationV2.title).ilike(f"%{search_query}%"),
col(ConversationV2.content).ilike(f"%{search_query}%"),
col(ConversationV2.user_id).ilike(f"%{search_query}%"),
# NOTE: Searching by conversation_id may impact query performance.
# Monitor query performance and add an index on conversation_id if needed.
col(ConversationV2.conversation_id).ilike(f"%{search_query}%"), )
# Log query performance for monitoring
import logging, time
logger = logging.getLogger("astrbot.db")
start_time = time.time() )
elapsed = time.time() - start_time
if elapsed > 1.0: # Log if query takes longer than 1 second
logger.warning(f"Search query with conversation_id took {elapsed:.2f}s. Consider adding an index if this is common.")
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation / 动机
Modifications / 改动点
1、优化了sqlite配置，启用WAL模式和NORMAL，优化并发读写性能，同时增加了一些缓存策略。
2、对话数据，列表显示会话id字段，查询增加会话id条件（会话id指cid，原来的sessionId改为用户id）。
3、对话数据，增加请求打断功能，现在查询时仍然允许输入，如上一次请求没有完成将会被取消。
4、会话管理，搜索框增加输入防抖。
Verification Steps / 验证步骤
Screenshots or Test Results / 运行截图或测试结果
新的sqlite配置中普通查询平均快100ms左右
这是原来的配置
Compatibility & Breaking Changes / 兼容性与破坏性变更
Checklist / 检查清单
requirements.txt和pyproject.toml文件相应位置。/ I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations inrequirements.txtandpyproject.toml.Sourcery 总结
优化 SQLite 性能，并提升对话和会话管理页面的交互性和性能。
新功能：
改进：
Original summary in English
Summary by Sourcery
Optimize SQLite performance and enhance interactivity and performance on the conversation and session management pages.
New Features:
Enhancements: