[python] 修复进程重启机制不完善与Scheduler停止时的AttributeError by surpercodehang · Pull Request #359 · ModelEngine-Group/fit-framework

surpercodehang · 2025-10-30T06:46:04Z

🔗 相关问题 / Related Issue

Issue 链接 / Issue Link: #362 👈👈

我已经创建了相关 Issue 并进行了讨论 / I have created and discussed the related issue
这是一个微小的修改（如错别字），不需要 Issue / This is a trivial change (like typo fix) that doesn't need an issue

📋 变更类型 / Type of Change

🐛 Bug 修复 / Bug fix (non-breaking change which fixes an issue)
✨ 新功能 / New feature (non-breaking change which adds functionality)
💥 破坏性变更 / Breaking change (fix or feature that would cause existing functionality to not work as expected)
📚 文档更新 / Documentation update
🔧 重构 / Refactoring (no functional changes)
⚡ 性能优化 / Performance improvement
📦 依赖升级 / Dependency upgrade (update dependencies to newer versions)
🚀 功能增强 / Feature enhancement (improve existing functionality without breaking changes)
🧹 代码清理 / Code cleanup

📝 变更目的 / Purpose of the Change

主进程退出之后，心跳偶现不能将主进程拉起的问题，导致

📋 主要变更 / Brief Changelog

新增进程监控和健康检查工具
修复重启时Scheduler中的_timer因重启时间太长未赋值导致的进程下线无法重启的问题
增加心跳与主进程的日志检测

🧪 验证变更 / Verifying this Change

测试步骤 / Test Steps

部署环境，在环境上验证了两周，主进程下线之后可以正常重启

测试覆盖 / Test Coverage

我已经添加了单元测试 / I have added unit tests
所有现有测试都通过 / All existing tests pass
我已经进行了手动测试 / I have performed manual testing

📸 截图 / Screenshots

✅ 贡献者检查清单 / Contributor Checklist

请确保你的 Pull Request 符合以下要求 / Please ensure your Pull Request meets the following requirements:

基本要求 / Basic Requirements:

确保有 GitHub Issue 对应这个变更（微小变更如错别字除外）/ Make sure there is a Github issue filed for the change (trivial changes like typos excluded)
你的 Pull Request 只解决一个 Issue，没有包含其他不相关的变更 / Your PR addresses just this issue, without pulling in other changes - one PR resolves one issue
PR 中的每个 commit 都有有意义的主题行和描述 / Each commit in the PR has a meaningful subject line and body

代码质量 / Code Quality:

我的代码遵循项目的代码规范 / My code follows the project's coding standards
我已经进行了自我代码审查 / I have performed a self-review of my code
我已经为复杂的代码添加了必要的注释 / I have commented my code, particularly in hard-to-understand areas

测试要求 / Testing Requirements:

我已经编写了必要的单元测试来验证逻辑正确性 / I have written necessary unit-tests to verify the logic correction
当存在跨模块依赖时，我尽量使用了 mock / I have used mocks when cross-module dependencies exist
基础检查通过：mvn -B clean package -Dmaven.test.skip=true / Basic checks pass
单元测试通过：mvn clean install / Unit tests pass

文档和兼容性 / Documentation and Compatibility:

我已经更新了相应的文档 / I have made corresponding changes to the documentation
如果有破坏性变更，我已经在 PR 描述中详细说明 / If there are breaking changes, I have documented them in detail
我已经考虑了向后兼容性 / I have considered backward compatibility

📋 附加信息 / Additional Notes

审查者注意事项 / Reviewer Notes:

surpercodehang · 2025-11-10T02:33:46Z

根据当前的日志分析
1、主进程因空闲（缓存为空且休息时间超过 10 秒）正常退出
2、主进程调用 shutdown()，触发 FRAMEWORK_STOPPING 事件offline() 被调用，向心跳任务发送退出信号
3、心跳任务正常退出
4、HeartBeatWatchThread（守护线程）检测到心跳任务已退出
5、该线程无法区分正常退出与意外退出，再次调用 shutdown()导致重复关闭，可能引发竞态

修复方案
在 heart_beat_agent.py 中：
1、添加 _SHUTDOWN_IN_PROGRESS 标志，用于标记是否正在关闭
2、添加 _SHUTDOWN_LOCK 锁，保证线程安全
3、在 offline() 中设置该标志，表示正常关闭开始
4、在 _heart_beat_monitor() 中检查该标志：
如果正在关闭，心跳任务退出是预期的，直接返回，不再调用 shutdown()
如果未在关闭，则视为意外退出，调用 shutdown()

这样可避免正常关闭时 HeartBeatWatchThread 重复调用 shutdown()。

loveTsong · 2025-11-18T03:22:57Z

framework/fit/python/fitframework/__init__.py

+                time.sleep(restart_delay)
+
+            except Exception as e:
+                fit_logger.error(f"Error during process management: {e}")


可以简化逻辑，同时去除重复代码。两个地方只获取exit_code， try 外层统一处理exit_code

loveTsong · 2025-11-18T06:08:17Z

framework/fit/python/fitframework/utils/process_monitor.py

@@ -0,0 +1,88 @@
+# -- encoding: utf-8 --
+# Copyright (c) 2024 Huawei Technologies Co., Ltd. All Rights Reserved.


这里新增文件版权时间应为2025

loveTsong · 2025-11-18T06:16:19Z

framework/fit/python/fitframework/utils/process_monitor.py

+
+        # 检查CPU使用（需要两次采样）
+        cpu_percent = info['cpu_percent']
+        if cpu_percent > self.max_cpu_percent:


1. 注释和代码不匹配 2. 单次采集cpu超限就认为不健康需要重启策略不够合理，可能导致瞬时冲高触发重启，应该采集一段周期的平均使用率才合理

loveTsong · 2025-11-18T06:16:44Z

framework/fit/python/fitframework/utils/restart_policy.py

@@ -0,0 +1,107 @@
+# -- encoding: utf-8 --
+# Copyright (c) 2024 Huawei Technologies Co., Ltd. All Rights Reserved.


版权时间应该为2025

loveTsong · 2025-11-18T06:23:46Z

framework/fit/python/plugin/fit_py_health_monitor/conf/application.yml

+health-monitor:
+  check-interval: 30  # 健康检查间隔（秒）
+  memory-threshold: 80.0  # 内存使用率阈值（百分比）
+  cpu-threshold: 90.0  # CPU使用率阈值（百分比）


这里要关注下默认值是否合理，90%不到一个核心。另外，当前主要是靠docker控制可用的cpu，没有必要默认启用，这里是否可以默认不设置，不启用该配置。

loveTsong · 2025-11-18T07:10:51Z

framework/fit/python/plugin/fit_py_health_monitor/health_monitor.py

@@ -0,0 +1,176 @@
+# -- encoding: utf-8 --


看起来这个插件没有使用，可以删除

loveTsong · 2025-11-18T07:11:28Z

framework/fit/python/fitframework/utils/process_monitor.py

@@ -0,0 +1,88 @@
+# -- encoding: utf-8 --


没有用到的文件

[python] 修复进程重启机制不完善与Scheduler停止时的AttributeError

bc8bc11

surpercodehang self-assigned this Oct 30, 2025

surpercodehang linked an issue Oct 30, 2025 that may be closed by this pull request

Python启动日志显示HeartBeatThread报错 #276

Open

4 tasks

surpercodehang requested a review from CodeCasterX October 30, 2025 06:49

[python] 修复 HeartBeatWatchThread 重复调用 shutdown() 方法的问题

0134ef8

[python] 添加主进程异常退出的日志记录

58e74ad

surpercodehang linked an issue Nov 18, 2025 that may be closed by this pull request

[fit-python] fit-runtime-python 代码节点报错no fittable #362

Closed

4 tasks

surpercodehang removed a link to an issue Nov 18, 2025

Python启动日志显示HeartBeatThread报错 #276

Open

4 tasks

CodeCasterX added type: bug A general bug in: fit Issues in FIT modules labels Nov 18, 2025

CodeCasterX modified the milestones: 3.6.0, 3.6.1 Nov 18, 2025

surpercodehang modified the milestones: 3.6.1, 3.6.x Nov 18, 2025

surpercodehang added this to Nova Nov 18, 2025

loveTsong reviewed Nov 18, 2025

View reviewed changes

CodeCasterX modified the milestones: 3.6.x, 3.6.1 Nov 18, 2025

surpercodehang changed the base branch from main to 3.6.x November 18, 2025 06:35

loveTsong reviewed Nov 18, 2025

View reviewed changes

surpercodehang added 2 commits November 18, 2025 15:13

[python] 移除无用的代码

987b923

[python] 修改检视意见

3493a9f

surpercodehang force-pushed the bugfix-thread-error-exit branch from 5bd64ff to 3493a9f Compare November 18, 2025 07:41

loveTsong approved these changes Nov 19, 2025

View reviewed changes

Merge branch '3.6.x' into bugfix-thread-error-exit

e380015

loveTsong approved these changes Nov 19, 2025

View reviewed changes

CodeCasterX merged commit f108df0 into ModelEngine-Group:3.6.x Nov 19, 2025
1 check passed

github-project-automation bot moved this to Done in Nova Nov 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python] 修复进程重启机制不完善与Scheduler停止时的AttributeError#359

[python] 修复进程重启机制不完善与Scheduler停止时的AttributeError#359
CodeCasterX merged 6 commits intoModelEngine-Group:3.6.xfrom
surpercodehang:bugfix-thread-error-exit

surpercodehang commented Oct 30, 2025 •

edited

Loading

Uh oh!

surpercodehang commented Nov 10, 2025

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

loveTsong Nov 18, 2025 •

edited

Loading

Uh oh!

loveTsong Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,88 @@
		# -- encoding: utf-8 --
		# Copyright (c) 2024 Huawei Technologies Co., Ltd. All Rights Reserved.

		@@ -0,0 +1,107 @@
		# -- encoding: utf-8 --
		# Copyright (c) 2024 Huawei Technologies Co., Ltd. All Rights Reserved.

Conversation

surpercodehang commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 相关问题 / Related Issue

📋 变更类型 / Type of Change

📝 变更目的 / Purpose of the Change

📋 主要变更 / Brief Changelog

🧪 验证变更 / Verifying this Change

测试步骤 / Test Steps

测试覆盖 / Test Coverage

📸 截图 / Screenshots

✅ 贡献者检查清单 / Contributor Checklist

📋 附加信息 / Additional Notes

Uh oh!

surpercodehang commented Nov 10, 2025

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

loveTsong Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

surpercodehang commented Oct 30, 2025 •

edited

Loading

loveTsong Nov 18, 2025 •

edited

Loading