hoteas
66f18a084a
- 新增 shuttingDown 原子布尔标志和 shutdownOnce 保证单次执行 - 添加 DrainTimeout 和 ShutdownTimeout 配置参数，默认 5s 和 30s - 修改 ServeHTTP 拦截新请求返回 503 Service Unavailable - 实现跨平台信号处理：Windows 关窗口和 Linux SIGTERM/SIGINT - 添加 recover 保护避免停机过程中 panic 触发重启 - 优化 HTTP 服务退出判断，排除正常关闭情况 - 新增 log.LoggerNoCaller 用于访问日志等无意义调用方场景 - 更新 README 文档增加优雅停机特性说明和文档链接 - 创建详细的优雅停机文档和测试用例 - 添加 Windows 平台特定信号处理实现 - 新增 .cursor 计划文件记录开发方案
251 lines
7.9 KiB
Markdown
251 lines
7.9 KiB
Markdown
---
|
||
name: 优雅停机方案
|
||
overview: 在 HoTime 框架层实现优雅停机，跨平台（Windows/Linux）支持。收到信号后新请求立即返回 503 让 nginx 切流，等排空后调用 http.Server.Shutdown 等在途请求完成，exit(0) 退出，run_xbc.cmd 检测正常退出不重启。
|
||
todos:
|
||
- id: framework-core
|
||
content: hotimev1.5/application.go：新增 shuttingDown/shutdownOnce/DrainTimeout/ShutdownTimeout 字段，改 ServeHTTP，改 Run() 信号 goroutine，改 recover 保护，提取 initiateGracefulShutdown()
|
||
status: completed
|
||
- id: framework-windows
|
||
content: hotimev1.5/signal_windows.go（新建）：SetConsoleCtrlHandler 处理 CTRL_CLOSE_EVENT（关窗口）
|
||
status: completed
|
||
- id: framework-other
|
||
content: hotimev1.5/signal_other.go（新建）：非 Windows 平台 registerWindowsCloseHandler() 空实现
|
||
status: completed
|
||
- id: nginx-config
|
||
content: Nginx location 块加 proxy_next_upstream error timeout http_503 和 proxy_next_upstream_tries 2
|
||
status: completed
|
||
- id: cmd-exit-check
|
||
content: run_xbc.cmd：检测 EXIT_CODE == 0 时不重启直接退出
|
||
status: completed
|
||
- id: docs
|
||
content: 新建 hotimev1.5/docs/graceful-shutdown.md 文档
|
||
status: completed
|
||
isProject: false
|
||
---
|
||
|
||
# 优雅停机方案
|
||
|
||
## 整体流程
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Op as 运维
|
||
participant OS as 操作系统
|
||
participant App as Go 进程
|
||
participant Nginx as Nginx
|
||
|
||
Op->>OS: Ctrl+C / kill / taskkill / 关窗口
|
||
OS->>App: SIGINT / SIGTERM / CTRL_CLOSE_EVENT
|
||
App->>App: shuttingDown = true（sync.Once 保证只执行一次）
|
||
Note over App,Nginx: 新请求立即返回 503 + Connection:close
|
||
Nginx->>App: 新请求
|
||
App-->>Nginx: 503 Service Unavailable
|
||
Nginx->>Nginx: proxy_next_upstream http_503 → 转另一台
|
||
Note over App: 等 DrainTimeout（默认5s，让nginx完成切流）
|
||
App->>App: server.Shutdown(ShutdownTimeout=30s)
|
||
Note over App: 等所有在途请求跑完
|
||
App->>OS: os.Exit(0)
|
||
Note over Op: run_xbc.cmd 检测 exit 0 → 不重启
|
||
```
|
||
|
||
## 信号触发场景对照表
|
||
|
||
| 操作方式 | 平台 | 信号/事件 | 是否可拦截 | 备注 |
|
||
|---|---|---|---|---|
|
||
| Ctrl+C | Windows/Linux | `os.Interrupt` | ✅ | 标准中断 |
|
||
| `kill <pid>` 或 `kill -15 <pid>` | Linux | `syscall.SIGTERM` | ✅ | 推荐生产关闭方式 |
|
||
| `taskkill /PID <pid>`（无 /F） | Windows | `os.Interrupt` | ✅ | 等价于 Ctrl+C |
|
||
| 关闭控制台窗口 X 按钮 | Windows | `CTRL_CLOSE_EVENT` | ✅（≤5s 限制）| Windows 给 ~5s 后强杀 |
|
||
| `taskkill /F /PID <pid>` | Windows | 强制 SIGKILL | ❌ | 无法拦截，强杀 |
|
||
| `kill -9 <pid>` | Linux | SIGKILL | ❌ | 无法拦截，强杀 |
|
||
|
||
> **关窗口说明**：Windows 在触发 `CTRL_CLOSE_EVENT` 后约 5 秒会强制结束进程。框架对此事件使用更短的 drain（默认 2s）+ shutdown（默认 3s），总计 <5s，可完成正在跑的短接口；长接口（>3s）会被截断。推荐用 `taskkill /PID` 代替关窗口做计划性停机。
|
||
|
||
---
|
||
|
||
## 需要改动/新建的文件
|
||
|
||
### 1. [`hotimev1.5/application.go`](d:/work/hotimev1.5/application.go)
|
||
|
||
**`Application` struct** 新增字段：
|
||
```go
|
||
type Application struct {
|
||
// ...原有字段...
|
||
shuttingDown atomic.Bool
|
||
shutdownOnce sync.Once // 保证多路触发只执行一次
|
||
DrainTimeout time.Duration // 等 nginx 切流，默认 5s
|
||
ShutdownTimeout time.Duration // 等在途请求，默认 30s
|
||
}
|
||
```
|
||
|
||
**`ServeHTTP`** 新增停机检测：
|
||
```go
|
||
func (that *Application) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||
if that.shuttingDown.Load() {
|
||
w.Header().Set("Connection", "close")
|
||
http.Error(w, "Service Unavailable", http.StatusServiceUnavailable)
|
||
return
|
||
}
|
||
that.handler(w, req)
|
||
}
|
||
```
|
||
|
||
**提取 `initiateGracefulShutdown(drain, shutdown time.Duration)` 方法**（被信号 goroutine 和 Windows 关窗 handler 共同调用）：
|
||
```go
|
||
func (that *Application) initiateGracefulShutdown(drain, shutdown time.Duration) {
|
||
that.shutdownOnce.Do(func() {
|
||
that.shuttingDown.Store(true)
|
||
that.Log.Infof("优雅停机：等 %v 让 nginx 切流...", drain)
|
||
time.Sleep(drain)
|
||
ctx, cancel := context.WithTimeout(context.Background(), shutdown)
|
||
defer cancel()
|
||
that.Log.Infof("等进行中请求完成（最多 %v）...", shutdown)
|
||
if err := that.Server.Shutdown(ctx); err != nil {
|
||
that.Log.Errorf("优雅停机超时: %v", err)
|
||
}
|
||
that.Log.Infof("服务已安全关闭")
|
||
os.Exit(0)
|
||
})
|
||
}
|
||
```
|
||
|
||
**`Run()`** 加信号监听 goroutine（在启动 ListenAndServe 之后）：
|
||
```go
|
||
// 注册 Windows 关窗口处理（Linux 上是空实现）
|
||
that.registerWindowsCloseHandler()
|
||
|
||
// 跨平台：Ctrl+C / SIGTERM / kill
|
||
go func() {
|
||
sigCh := make(chan os.Signal, 1)
|
||
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
|
||
<-sigCh
|
||
drain := that.DrainTimeout
|
||
if drain == 0 { drain = 5 * time.Second }
|
||
shutdown := that.ShutdownTimeout
|
||
if shutdown == 0 { shutdown = 30 * time.Second }
|
||
that.initiateGracefulShutdown(drain, shutdown)
|
||
}()
|
||
```
|
||
|
||
**`defer recover`** 加停机保护：
|
||
```go
|
||
defer func() {
|
||
if err := recover(); err != nil {
|
||
if that.shuttingDown.Load() {
|
||
return // 停机中的 panic 不触发重启
|
||
}
|
||
that.Log.Warnf("%v", err)
|
||
that.Run(router)
|
||
}
|
||
}()
|
||
```
|
||
|
||
新增 import：`"context"`, `"os/signal"`, `"sync"`, `"sync/atomic"`, `"syscall"`
|
||
|
||
---
|
||
|
||
### 2. `hotimev1.5/signal_windows.go`（新建）
|
||
|
||
```go
|
||
//go:build windows
|
||
|
||
package hotime
|
||
|
||
import (
|
||
"time"
|
||
"golang.org/x/sys/windows"
|
||
)
|
||
|
||
func (that *Application) registerWindowsCloseHandler() {
|
||
windows.SetConsoleCtrlHandler(func(ctrl uint32) bool {
|
||
// CTRL_CLOSE_EVENT=2 CTRL_LOGOFF_EVENT=5 CTRL_SHUTDOWN_EVENT=6
|
||
if ctrl == 2 || ctrl == 5 || ctrl == 6 {
|
||
that.initiateGracefulShutdown(2*time.Second, 3*time.Second)
|
||
// 阻塞此回调，让 initiateGracefulShutdown 内的 os.Exit(0) 执行
|
||
// Windows 约 5s 后强杀，这里最多等 6s 兜底
|
||
time.Sleep(6 * time.Second)
|
||
return true
|
||
}
|
||
return false
|
||
}, true)
|
||
}
|
||
```
|
||
|
||
> `golang.org/x/sys` 已在 `xbc/go.mod` 中作为间接依赖存在（v0.29.0），无需新增依赖。
|
||
|
||
---
|
||
|
||
### 3. `hotimev1.5/signal_other.go`（新建）
|
||
|
||
```go
|
||
//go:build !windows
|
||
|
||
package hotime
|
||
|
||
func (that *Application) registerWindowsCloseHandler() {}
|
||
```
|
||
|
||
---
|
||
|
||
### 4. Nginx 配置（在已有 location 块中追加）
|
||
|
||
```nginx
|
||
# 让后端返回 503 时自动切换到另一台（优雅停机核心）
|
||
proxy_next_upstream error timeout http_503;
|
||
proxy_next_upstream_tries 2;
|
||
```
|
||
|
||
已有的 `max_fails=1 fail_timeout=3s` 负责连接级失败（强杀后的端口拒绝），两者互补。
|
||
|
||
---
|
||
|
||
### 5. [`run_xbc.cmd`](d:/work/xbc/run_xbc.cmd)
|
||
|
||
在 `goto run_app` 前插入 exit code 判断：
|
||
|
||
```cmd
|
||
:: 优雅退出 (exit 0) 不重启
|
||
if %EXIT_CODE% EQU 0 (
|
||
echo [%date% %time%] Graceful shutdown. Not restarting. >> "%LOG_FILE%"
|
||
echo.
|
||
echo [优雅停机] 服务已安全关闭，不再自动重启。
|
||
pause
|
||
exit /b 0
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
### 6. `hotimev1.5/docs/graceful-shutdown.md`（新建）
|
||
|
||
包含：原理说明、场景对照表、nginx 配置、滚动重启步骤、参数说明（DrainTimeout/ShutdownTimeout）。
|
||
|
||
---
|
||
|
||
## 滚动重启操作步骤（Windows）
|
||
|
||
```
|
||
# 1. 找到端口 9998 的进程 PID
|
||
netstat -ano | findstr 9998
|
||
|
||
# 2. 优雅关闭该实例（不加 /F）
|
||
taskkill /PID <pid>
|
||
|
||
# 3. 等日志出现 "服务已安全关闭" 后再部署新版并启动
|
||
# 4. 再对 9999 执行同样操作
|
||
```
|
||
|
||
## 滚动重启操作步骤（Linux）
|
||
|
||
```bash
|
||
# 找 PID
|
||
pgrep xbc # 或 ps aux | grep xbc
|
||
|
||
# 优雅关闭
|
||
kill <pid> # 发 SIGTERM，触发优雅停机
|
||
# 或
|
||
kill -2 <pid> # 发 SIGINT，等同 Ctrl+C
|
||
|
||
# 等日志出现 "服务已安全关闭" 再部署新版
|
||
```
|