步骤4阶段1 D2执行方案(最终版)
制定日期: 2026-09-14
执行日期: 2026-09-15
前置条件: D1观察通过(90次运行,100%成功)
D2最终时间表
| 时间 |
任务 |
内容 |
状态 |
| 17:30-18:15 |
Dead-Man测试 |
暂停cron→等待30分钟→验证告警→恢复 |
⏳ 待执行 |
| 21:00-21:45 |
受控测试(前3项) |
告警路径+超时路径+通道健康 |
⏳ 待执行 |
| 21:45-22:15 |
影子cron集成 |
B3临时替换验证 |
⏳ 待执行 |
| 22:15-22:45 |
回归验证 |
清理数据+确认无异常 |
⏳ 待执行 |
任务1:Dead-Man测试(17:30-18:15)
执行步骤
| 时间 |
步骤 |
命令/操作 |
预期结果 |
| 17:30 |
准备 |
打印时间戳 |
[TEST] Dead-Man测试开始: 2026-09-15T17:30:00 |
| 17:31 |
备份 |
hermes cron list > D:/Hermes/reports/D2_deadman_backup.txt |
备份文件生成 |
| 17:32 |
暂停cron |
hermes cron pause 1f99cf7364a9 |
cron状态=paused |
| 17:33 |
验证暂停 |
hermes cron list \| grep 1f99cf7364a9 |
显示paused |
| 17:34-18:04 |
等待 |
不做任何操作 |
alert_channel_health每15分钟检测 |
| 18:04 |
验证告警 |
SELECT * FROM alerts WHERE task_id = 'ops_system_health' ORDER BY created_at DESC LIMIT 1 |
应有P1告警 |
| 18:06 |
恢复cron |
hermes cron resume 1f99cf7364a9 --run-now |
cron状态=active |
| 18:10 |
验证恢复 |
hermes cron list \| grep 1f99cf7364a9 |
显示active |
| 18:12 |
清理 |
DELETE FROM alerts WHERE created_at >= '2026-09-15T17:30:00' |
测试数据已清理 |
| 18:15 |
完成 |
打印总结 |
[TEST] Dead-Man测试完成 |
验收标准
✓ ops_system_health在暂停30分钟后被检测到超时
✓ alerts表有P1级别告警记录
✓ 恢复后alert_channel_health显示正常
✓ 无残留测试数据
任务2:受控测试前3项(21:00-21:45)
测试1:告警路径(21:00-21:15)
# 步骤:
1. 修改ops_system_health的last_run_at为3小时前
UPDATE task_runs SET start_time = datetime('now', '-3 hours')
WHERE task_id = 'ops_system_health' ORDER BY start_time DESC LIMIT 1
2. 手动触发alert_channel_health
hermes cron run 7e8ae2aeca81
3. 验证告警生成
SELECT * FROM alerts WHERE task_id = 'ops_system_health' ORDER BY created_at DESC LIMIT 1
4. 清理
DELETE FROM task_runs WHERE task_id = 'ops_system_health' AND start_time < '2026-09-15T21:00:00'
测试2:超时路径(21:15-21:30)
# 步骤:
1. 检查ops_system_health当前状态
SELECT status, start_time FROM task_runs WHERE task_id = 'ops_system_health' ORDER BY start_time DESC LIMIT 1
2. 验证alert_channel_health检测逻辑
- 模拟elapsed_s > timeout_threshold_s (7200s)
- 确认检测路径正确
3. 清理(无需修改DB)
测试3:通道健康(21:30-21:45)
# 步骤:
1. 检查channel_health表
SELECT check_time, status FROM channel_health ORDER BY check_time DESC LIMIT 5
2. 验证心跳记录正常
- 检查最近5次运行
- 确认间隔≈15分钟
3. 清理(无需修改DB)
任务3:影子cron集成(21:45-22:15)
B3方式执行步骤
| 时间 |
步骤 |
命令 |
| 21:45 |
备份 |
cp C:/Users/Admin/AppData/Local/hermes/cron/jobs.json D:/Hermes/reports/jobs.json.bak_D2_2145 |
| 21:46 |
暂停原cron |
hermes cron pause 1f99cf7364a9 |
| 21:47 |
注册影子cron |
hermes cron create --name "[TEST] 系统环境自检(影子)" --script run_with_alert.py --schedule "30 2 * * *" --deliver local --no-agent "47 21 * * *" |
| 21:50 |
手动触发 |
hermes cron run <影子cron_id> |
| 21:55 |
验证功能 |
SELECT * FROM task_runs WHERE task_id = 'ops_system_health' ORDER BY start_time DESC LIMIT 1 |
| 22:00 |
恢复原cron |
hermes cron resume 1f99cf7364a9 |
| 22:02 |
删除影子 |
hermes cron remove <影子cron_id> |
| 22:05 |
清理数据 |
DELETE FROM task_runs WHERE task_id = 'ops_system_health' AND start_time >= '2026-09-15T21:45:00' |
| 22:10 |
确认状态 |
hermes cron list \| grep 系统环境自检 |
任务4:回归验证(22:15-22:45)
验证清单
□ 原cron状态正常
hermes cron list | grep 系统环境自检
预期: enabled=true, next_run_at正确
□ 无测试残留数据
SELECT COUNT(*) FROM task_runs WHERE start_time >= '2026-09-15T17:30:00'
预期: 0条
□ 无测试残留告警
SELECT COUNT(*) FROM alerts WHERE created_at >= '2026-09-15T17:30:00'
预期: 0条
□ D1观察cron正常运行
hermes cron list | grep 7e8ae2aeca81
预期: active, next_run_at正确
□ 备份文件存在
ls D:/Hermes/reports/*D2*
预期: 多个备份文件
清理清单
数据库清理:
DELETE FROM task_runs WHERE start_time >= '2026-09-15T17:30:00';
DELETE FROM alerts WHERE created_at >= '2026-09-15T17:30:00';
DELETE FROM silence_state WHERE task_id = 'ops_system_health';
文件清理:
保留: D:/Hermes/reports/jobs.json.bak_D2_2145
删除: D:/Hermes/reports/D2_test_*.txt(如有)
Cron清理:
确认无影子cron残留: hermes cron list | grep TEST
回滚方案
# 一键回滚(如遇严重问题)
hermes cron resume 1f99cf7364a9 # 恢复原cron
hermes cron remove <影子cron_id> # 删除影子
cp D:/Hermes/reports/jobs.json.bak_D2_2145 C:/Users/Admin/AppData/Local/hermes/cron/jobs.json
hermes gateway restart # 重新加载配置
# DB回滚(如需)
python -c "
import sqlite3
conn = sqlite3.connect('D:/Hermes/data/state/alert_state.db')
conn.execute(\"DELETE FROM task_runs WHERE start_time >= '2026-09-15T17:30:00'\")
conn.execute(\"DELETE FROM alerts WHERE created_at >= '2026-09-15T17:30:00'\")
conn.commit()
conn.close()
"
验收标准
D2通过条件:
✓ Dead-Man测试: ops_system_health超时被正确检测,TG告警发送成功
✓ 受控测试: 3项测试全部通过,无异常
✓ 影子cron集成: B3方式验证成功,原cron恢复正常
✓ 回归验证: 无测试残留,D1观察cron正常运行
✓ 清理完成: DB和cron状态恢复原样
请确认方案后,回复"进入D2"启动明天执行。