💡 前言

最近在折腾truenas scale上的私人相册神器 Immich，并且为我十万张级别的照片库开启了全量 OCR（文本识别）扫描。为了追求极致的识别率，我启用了 Server 级别的识别模型（PP-OCRv5_server），并把任务交给了 NAS 上的独立显卡（16G 显存）来跑。但很快就遇到了一个极其崩溃的问题：任务跑不了多久，GPU 显存就会被撑爆（OOM），导致 ML（机器学习）容器异常重启。 最终跑了一两天，识别成功的图片寥寥无几，任务队列处于反复崩溃的死循环中。

分析

设置如图

我为了追求高ocr识别，所以提高了immich的ocr默认缩放分辨率736为1024，同时采用PP-OCRv5_server模型，识别更精确。导致显存占用需求比默认配置高不少，从而导致了各种显存问题。分析共同点发现这主要怪 OCR 的工作原理和 ONNX 的“仓鼠症”内存管理：

  • 动态切割（大小不一的碎片）：OCR 分为两步：先“框”出文字，再“读”出文字。每张照片框出来的文字长短、大小都不一样。所以送给识别模型（Recognition）的图片碎片，每一张的尺寸（Shape）都在变化。

  • ONNX 的“仓鼠症”（BFC Arena）：为了追求极致速度，ONNX 底层的内存分配器（BFC Arena）有一个致命逻辑：它会为它见过的每一种尺寸开辟一块专属的显存缓存，并且用完之后死活不还给显卡，而是自己囤着留给下次用。

  • 量变引起质变：当你刚跑前几百张时，尺寸种类不多，显存占用很低。但当你连续跑了一整天，处理了几万张照片，ONNX 见过了成千上万种不同尺寸的文字截图，它在你的 16G 显卡里建了成千上万个大大小小的“蓄水池”，导致 15G 显存全被这些死水池占满了。

最终崩溃：当新来了一张照片，需要申请一个 40MB 的新池子时，虽然显卡名义上有 16G，但已经被切得稀碎，找不出一块连续的 40MB 空间了，于是系统抛出 Failed to allocate memory… size 45889024（分配失败），容器当场去世。

解决经历

修改机器学习相关环境变量

参考：https://docs.immich.app/install/environment-variables/

  1. 关闭内存预分配模式：
    MACHINE_LEARNING_MODEL_ARENA=false

  2. 设置 OCR 模型一次同时处理的最大文本框数量(默认值是 6)：
    注意: 不是官网的MACHINE_LEARNING_MAX_BATCH_SIZE__OCR，这个环境变量设置不生效，而且还会报TypeError: 'NoneType' object cannot be interpreted as an integer异常。
    使用如下变量：
    MACHINE_LEARNING_MAX_BATCH_SIZE__TEXT_RECOGNITION=4
    设置4就行，1和2太小了速度会太慢，6显存爆得更快。
    参考来源：https://github.com/immich-app/immich/issues/23442

  3. 设置 文字识别（OCR） 并发数 为 1
    全量跑的时候，求稳比较好，降低并发，防止爆显存。

  4. 重启immich，执行全量ocr识别任务

此时本以为万事大吉，结果发现依然遇到了如下异常：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
以下就是相关时间段的日志：“2026-03-08 17:02:35.283257+00:00[03/08/26 17:02:35] INFO     Loading detection model 'PP-OCRv5_server' to memory
2026-03-08 17:02:35.286717+00:00[03/08/26 17:02:35] INFO     Setting execution providers to                     
2026-03-08 17:02:35.286788+00:00                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2026-03-08 17:02:35.286830+00:00                             in descending order of preference                  
2026-03-08 17:02:40.079462+00:00[03/08/26 17:02:40] INFO     Loading recognition model 'PP-OCRv5_server' to     
2026-03-08 17:02:40.079612+00:00                             memory                                             
2026-03-08 17:02:40.080680+00:00[03/08/26 17:02:40] INFO     Setting execution providers to                     
2026-03-08 17:02:40.080769+00:00                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2026-03-08 17:02:40.080812+00:00                             in descending order of preference                  
2026-03-08 17:02:40.790773+00:00[INFO] 2026-03-08 17:02:40,790 [RapidOCR] base.py:22: Using engine_name: onnxruntime
2026-03-08 17:44:38.164543+00:002026-03-08 17:44:38.164081849 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Concat node. Name:'Concat.8' Status Message: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:359 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*) Failed to allocate memory for requested buffer of size 28618752
2026-03-08 17:44:38.164776+00:00
2026-03-08 17:44:38.946682+00:00[03/08/26 17:44:38] ERROR    Exception in ASGI application                      
2026-03-08 17:44:38.946836+00:00                                                                                
2026-03-08 17:44:38.946883+00:00                             ╭─────── Traceback (most recent call last) ───────╮
2026-03-08 17:44:38.946949+00:00                             │ /opt/venv/lib/python3.11/site-packages/rapidocr │
2026-03-08 17:44:38.946993+00:00                             │ /inference_engine/onnxruntime/main.py:88 in     │
2026-03-08 17:44:38.947053+00:00                             │ __call__                                        │
2026-03-08 17:44:38.947092+00:00                             │                                                 │
2026-03-08 17:44:38.947129+00:00                             │    85 │   def __call__(self, input_content: np. │
2026-03-08 17:44:38.947247+00:00                             │    86 │   │   input_dict = dict(zip(self.get_in │
2026-03-08 17:44:38.947462+00:00                             │    87 │   │   try:                              │
2026-03-08 17:44:38.947512+00:00                             │ ❱  88 │   │   │   return self.session.run(self. │
2026-03-08 17:44:38.947699+00:00                             │    89 │   │   except Exception as e:            │
2026-03-08 17:44:38.947748+00:00                             │    90 │   │   │   error_info = traceback.format │
2026-03-08 17:44:38.947859+00:00                             │    91 │   │   │   raise ONNXRuntimeError(error_ │
2026-03-08 17:44:38.947906+00:00                             │                                                 │
2026-03-08 17:44:38.948009+00:00                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
2026-03-08 17:44:38.948052+00:00                             │ ime/capi/onnxruntime_inference_collection.py:28 │
2026-03-08 17:44:38.948256+00:00                             │ 7 in run                                        │
2026-03-08 17:44:38.948307+00:00                             │                                                 │
2026-03-08 17:44:38.948411+00:00                             │    284 │   │   if not output_names:             │
2026-03-08 17:44:38.948458+00:00                             │    285 │   │   │   output_names = [output.name  │
2026-03-08 17:44:38.948520+00:00                             │    286 │   │   try:                             │
2026-03-08 17:44:38.948560+00:00                             │ ❱  287 │   │   │   return self._sess.run(output │
2026-03-08 17:44:38.948621+00:00                             │    288 │   │   except C.EPFail as err:          │
2026-03-08 17:44:38.948718+00:00                             │    289 │   │   │   if self._enable_fallback:    │
2026-03-08 17:44:38.948762+00:00                             │    290 │   │   │   │   print(f"EP Error: {err!s │
2026-03-08 17:44:38.948866+00:00                             ╰─────────────────────────────────────────────────╯
2026-03-08 17:44:38.948937+00:00                             RuntimeException: [ONNXRuntimeError] : 6 :         
2026-03-08 17:44:38.948979+00:00                             RUNTIME_EXCEPTION : Non-zero status code returned  
2026-03-08 17:44:38.949016+00:00                             while running Concat node. Name:'Concat.8' Status  
2026-03-08 17:44:38.949084+00:00                             Message:                                           
2026-03-08 17:44:38.949123+00:00                             /onnxruntime_src/onnxruntime/core/framework/bfc_are
2026-03-08 17:44:38.949266+00:00                             na.cc:359 void*                                    
2026-03-08 17:44:38.949307+00:00                             onnxruntime::BFCArena::AllocateRawInternal(size_t, 
2026-03-08 17:44:38.949345+00:00                             bool, onnxruntime::Stream*) Failed to allocate     
2026-03-08 17:44:38.949457+00:00                             memory for requested buffer of size 28618752       
2026-03-08 17:44:38.949501+00:00                                                                                
2026-03-08 17:44:38.949625+00:00                                                                                
2026-03-08 17:44:38.949669+00:00                             The above exception was the direct cause of the    
2026-03-08 17:44:38.949707+00:00                             following exception:                               
2026-03-08 17:44:38.949771+00:00                                                                                
2026-03-08 17:44:38.949810+00:00                             ╭─────── Traceback (most recent call last) ───────╮
2026-03-08 17:44:38.949874+00:00                             │ /usr/src/immich_ml/main.py:191 in predict       │
2026-03-08 17:44:38.949914+00:00                             │                                                 │
2026-03-08 17:44:38.949981+00:00                             │   188 │   │   inputs = text                     │
2026-03-08 17:44:38.950022+00:00                             │   189 │   else:                                 │
2026-03-08 17:44:38.950235+00:00                             │   190 │   │   raise HTTPException(400, "Either  │
2026-03-08 17:44:38.950327+00:00                             │ ❱ 191 │   response = await run_inference(inputs │
2026-03-08 17:44:38.950519+00:00                             │   192 │   return ORJSONResponse(response)       │
2026-03-08 17:44:38.950608+00:00                             │   193                                           │
2026-03-08 17:44:38.950720+00:00                             │   194                                           │
2026-03-08 17:44:38.950794+00:00                             │                                                 │
2026-03-08 17:44:38.950895+00:00                             │ /usr/src/immich_ml/main.py:218 in run_inference │
2026-03-08 17:44:38.950965+00:00                             │                                                 │
2026-03-08 17:44:38.951112+00:00                             │   215 │   without_deps, with_deps = entries     │
2026-03-08 17:44:38.951193+00:00                             │   216 │   await asyncio.gather(*[_run_inference │
2026-03-08 17:44:38.951435+00:00                             │   217 │   if with_deps:                         │
2026-03-08 17:44:38.951515+00:00                             │ ❱ 218 │   │   await asyncio.gather(*[_run_infer │
2026-03-08 17:44:38.951612+00:00                             │   219 │   if isinstance(payload, Image):        │
2026-03-08 17:44:38.951681+00:00                             │   220 │   │   response["imageHeight"], response │
2026-03-08 17:44:38.951874+00:00                             │   221                                           │
2026-03-08 17:44:38.951944+00:00                             │                                                 │
2026-03-08 17:44:38.952042+00:00                             │ /usr/src/immich_ml/main.py:211 in               │
2026-03-08 17:44:38.952108+00:00                             │ _run_inference                                  │
2026-03-08 17:44:38.952300+00:00                             │                                                 │
2026-03-08 17:44:38.952374+00:00                             │   208 │   │   │   │   message = f"Task {entry[' │
2026-03-08 17:44:38.952478+00:00                             │       output of {dep}"                          │
2026-03-08 17:44:38.952544+00:00                             │   209 │   │   │   │   raise HTTPException(400,  │
2026-03-08 17:44:38.952647+00:00                             │   210 │   │   model = await load(model)         │
2026-03-08 17:44:38.952715+00:00                             │ ❱ 211 │   │   output = await run(model.predict, │
2026-03-08 17:44:38.952862+00:00                             │   212 │   │   outputs[model.identity] = output  │
2026-03-08 17:44:38.952965+00:00                             │   213 │   │   response[entry["task"]] = output  │
2026-03-08 17:44:38.953034+00:00                             │   214                                           │
2026-03-08 17:44:38.953170+00:00                             │                                                 │
2026-03-08 17:44:38.953236+00:00                             │ /usr/src/immich_ml/main.py:229 in run           │
2026-03-08 17:44:38.953382+00:00                             │                                                 │
2026-03-08 17:44:38.953450+00:00                             │   226 │   if thread_pool is None:               │
2026-03-08 17:44:38.953625+00:00                             │   227 │   │   return func(*args, **kwargs)      │
2026-03-08 17:44:38.953708+00:00                             │   228 │   partial_func = partial(func, *args, * │
2026-03-08 17:44:38.953878+00:00                             │ ❱ 229 │   return await asyncio.get_running_loop │
2026-03-08 17:44:38.953961+00:00                             │   230                                           │
2026-03-08 17:44:38.954073+00:00                             │   231                                           │
2026-03-08 17:44:38.954150+00:00                             │   232 async def load(model: InferenceModel) ->  │
2026-03-08 17:44:38.954287+00:00                             │                                                 │
2026-03-08 17:44:38.954362+00:00                             │ /usr/local/lib/python3.11/concurrent/futures/th │
2026-03-08 17:44:38.954560+00:00                             │ read.py:58 in run                               │
2026-03-08 17:44:38.954634+00:00                             │                                                 │
2026-03-08 17:44:38.954699+00:00                             │ /usr/src/immich_ml/models/base.py:60 in predict │
2026-03-08 17:44:38.954808+00:00                             │                                                 │
2026-03-08 17:44:38.954879+00:00                             │    57 │   │   self.load()                       │
2026-03-08 17:44:38.955055+00:00                             │    58 │   │   if model_kwargs:                  │
2026-03-08 17:44:38.955126+00:00                             │    59 │   │   │   self.configure(**model_kwargs │
2026-03-08 17:44:38.955280+00:00                             │ ❱  60 │   │   return self._predict(*inputs)     │
2026-03-08 17:44:38.955416+00:00                             │    61 │                                         │
2026-03-08 17:44:38.955480+00:00                             │    62 │   @abstractmethod                       │
2026-03-08 17:44:38.955620+00:00                             │    63 │   def _predict(self, *inputs: Any, **mo │
2026-03-08 17:44:38.955690+00:00                             │                                                 │
2026-03-08 17:44:38.955838+00:00                             │ /usr/src/immich_ml/models/ocr/recognition.py:74 │
2026-03-08 17:44:38.955904+00:00                             │ in _predict                                     │
2026-03-08 17:44:38.956080+00:00                             │                                                 │
2026-03-08 17:44:38.956150+00:00                             │    71 │   │   boxes, box_scores = texts["boxes" │
2026-03-08 17:44:38.956301+00:00                             │    72 │   │   if boxes.shape[0] == 0:           │
2026-03-08 17:44:38.956366+00:00                             │    73 │   │   │   return self._empty            │
2026-03-08 17:44:38.956500+00:00                             │ ❱  74 │   │   rec = self.model(TextRecInput(img │
2026-03-08 17:44:38.956569+00:00                             │    75 │   │   if rec.txts is None:              │
2026-03-08 17:44:38.956678+00:00                             │    76 │   │   │   return self._empty            │
2026-03-08 17:44:38.956819+00:00                             │    77                                           │
2026-03-08 17:44:38.956886+00:00                             │                                                 │
2026-03-08 17:44:38.956950+00:00                             │ /opt/venv/lib/python3.11/site-packages/rapidocr │
2026-03-08 17:44:38.957149+00:00                             │ /ch_ppocr_rec/main.py:120 in __call__           │
2026-03-08 17:44:38.957231+00:00                             │                                                 │
2026-03-08 17:44:38.957351+00:00                             │   117 │   │   │   │   norm_img_batch.append(nor │
2026-03-08 17:44:38.957430+00:00                             │   118 │   │   │   norm_img_batch = np.concatena │
2026-03-08 17:44:38.957606+00:00                             │   119 │   │   │                                 │
2026-03-08 17:44:38.957684+00:00                             │ ❱ 120 │   │   │   preds = self.session(norm_img │
2026-03-08 17:44:38.957877+00:00                             │   121 │   │   │   line_results, word_results =  │
2026-03-08 17:44:38.957953+00:00                             │   122 │   │   │   │   preds,                    │
2026-03-08 17:44:38.958124+00:00                             │   123 │   │   │   │   return_word_box,          │
2026-03-08 17:44:38.958196+00:00                             │                                                 │
2026-03-08 17:44:38.958371+00:00                             │ /opt/venv/lib/python3.11/site-packages/rapidocr │
2026-03-08 17:44:38.958440+00:00                             │ /inference_engine/onnxruntime/main.py:91 in     │
2026-03-08 17:44:38.958624+00:00                             │ __call__                                        │
2026-03-08 17:44:38.958727+00:00                             │                                                 │
2026-03-08 17:44:38.958853+00:00                             │    88 │   │   │   return self.session.run(self. │
2026-03-08 17:44:38.958921+00:00                             │    89 │   │   except Exception as e:            │
2026-03-08 17:44:38.959111+00:00                             │    90 │   │   │   error_info = traceback.format │
2026-03-08 17:44:38.959182+00:00                             │ ❱  91 │   │   │   raise ONNXRuntimeError(error_ │
2026-03-08 17:44:38.959335+00:00                             │    92 │                                         │
2026-03-08 17:44:38.959403+00:00                             │    93 │   def get_input_names(self) -> List[str │
2026-03-08 17:44:38.959558+00:00                             │    94 │   │   return [v.name for v in self.sess │
2026-03-08 17:44:38.959682+00:00                             ╰─────────────────────────────────────────────────╯
2026-03-08 17:44:38.959749+00:00                             ONNXRuntimeError: Traceback (most recent call      
2026-03-08 17:44:38.959872+00:00                             last):                                             
2026-03-08 17:44:38.959940+00:00                               File                                             
2026-03-08 17:44:38.960022+00:00                             "/opt/venv/lib/python3.11/site-packages/rapidocr/in
2026-03-08 17:44:38.960326+00:00                             ference_engine/onnxruntime/main.py", line 88, in   
2026-03-08 17:44:38.960398+00:00                             __call__                                           
2026-03-08 17:44:38.960462+00:00                                 return                                         
2026-03-08 17:44:38.960599+00:00                             self.session.run(self.get_output_names(),          
2026-03-08 17:44:38.960668+00:00                             input_dict)[0]                                     
2026-03-08 17:44:38.960790+00:00                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-08 17:44:38.960856+00:00                             ^^^^^^^^^^^^^                                      
2026-03-08 17:44:38.960918+00:00                               File                                             
2026-03-08 17:44:38.961042+00:00                             "/opt/venv/lib/python3.11/site-packages/onnxruntime
2026-03-08 17:44:38.961110+00:00                             /capi/onnxruntime_inference_collection.py", line   
2026-03-08 17:44:38.961257+00:00                             287, in run                                        
2026-03-08 17:44:38.961323+00:00                                 return self._sess.run(output_names, input_feed,
2026-03-08 17:44:38.961398+00:00                             run_options)                                       
2026-03-08 17:44:38.961537+00:00                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-08 17:44:38.961604+00:00                             ^^^^^^^^^^^^^                                      
2026-03-08 17:44:38.961668+00:00                             onnxruntime.capi.onnxruntime_pybind11_state.Runtime
2026-03-08 17:44:38.961814+00:00                             Exception: [ONNXRuntimeError] : 6 :                
2026-03-08 17:44:38.961881+00:00                             RUNTIME_EXCEPTION : Non-zero status code returned  
2026-03-08 17:44:38.961943+00:00                             while running Concat node. Name:'Concat.8' Status  
2026-03-08 17:44:38.962051+00:00                             Message:                                           
2026-03-08 17:44:38.962117+00:00                             /onnxruntime_src/onnxruntime/core/framework/bfc_are
2026-03-08 17:44:38.962304+00:00                             na.cc:359 void*                                    
2026-03-08 17:44:38.962387+00:00                             onnxruntime::BFCArena::AllocateRawInternal(size_t, 
2026-03-08 17:44:38.962652+00:00                             bool, onnxruntime::Stream*) Failed to allocate     
2026-03-08 17:44:38.962720+00:00                             memory for requested buffer of size 28618752       
2026-03-08 17:44:38.962783+00:00                                                                                
2026-03-08 17:44:38.962895+00:00                                                                                
2026-03-08 17:47:05.953736+00:002026-03-08 17:47:05.953359445 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Concat node. Name:'Concat.8' Status Message: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:359 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*) Failed to allocate memory for requested buffer of size 33945600
2026-03-08 17:47:05.954011+00:00
2026-03-08 17:47:06.616105+00:00[03/08/26 17:47:05] ERROR    Exception in ASGI application ”

这应该是由于 Immich / ONNX 引擎在处理连续数万张高强度全量图片时，底层代码存在细微的内存泄漏导致的。而且遇到有人同步照片以及使用智能搜索功能时，PP-OCRv5_server + XLM-Roberta-Large + buffalo_l，这三个“性能巨兽”根本无法在 16G 的显存里和平共处。这种情况对于参数配置方面的优化已经无能为力，所以当时想到采取高显存时，主动释放显存的方式来解决。

思考解决

初始为了省事采用Automa浏览器插件，设置在定时任务页面定时点击开始/暂停按钮来控制显存的释放与占用，防止显存泄露以及爆显存。但是后续发现，显存的增长很不好估算，有时候几分钟，有时候一个多小时才会爆满异常，所以放弃。最终采用编写脚本，读取GPU显存占用，调用immich官方api控制ocr任务启停的方式成功解决。对了，immich默认释放显存是300秒，所以脚本的停止到继续的间隔必须大于5分钟，让immich自己释放显存。

通过脚本启停OCR全量定时任务，防止爆显存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
#!/bin/bash

# ================= 配置区域 =================
# 1. 你的 Immich 访问地址 (请把 192.168.x.x 换成你 NAS 的真实局域网 IP地址和端口)
API_URL="http://192.168.x.x:xxxx/api"

# 2. 刚才在网页端生成的 API Key
API_KEY="你immich里获取的api_key"

# 3. 危险警戒线 (单位：MB。我的总显存约 16384，设 15000 留出安全缓冲)
THRESHOLD=15000
# 4. 精准打击的任务队列名称：文本识别 (OCR)
JOB_NAME="ocr"
# ============================================

echo "=================================================="
echo " 🛡️ Immich OCR 智能显存雷达 (详细日志版) 已启动！"
echo " 🎯 当前显存警戒线：${THRESHOLD} MB"
echo " 💡 提示：随时按 [Ctrl + C] 可以安全退出本脚本"
echo "=================================================="
echo ""

while true; do
    # 提取当前 GPU 0 的已用显存数字
    VRAM=$(nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits)

    # 判断：如果当前显存 >= 警戒线
    if [ "$VRAM" -ge "$THRESHOLD" ]; then
        echo "" # 换行，打断实时刷新的那一行
        echo "🚨 =================================================="
        echo "🛑 [$(date '+%Y-%m-%d %H:%M:%S')] 触发显存警戒线！"
        echo "🛑 执行【暂停】任务，当时显存占用：${VRAM} MB"
        
        # 调用 API 暂停 OCR 任务
        curl -s -X PUT "$API_URL/jobs/$JOB_NAME" \
             -H "x-api-key: $API_KEY" \
             -H "Content-Type: application/json" \
             -d '{"command": "pause"}' > /dev/null

        echo "⏳ 队列已成功挂起，等待 5 分半钟 (330秒) 释放显存..."
        
        # 前台运行倒计时显示（每 30 秒汇报一次进度）
        for i in {11..1}; do
            sleep 30
            echo "   ...倒计时 $((i * 30)) 秒..."
        done

        # 休息完毕，重新获取一次当前的显存，验证是否已清空！
        CLEARED_VRAM=$(nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits)
        
        echo "▶️ [$(date '+%Y-%m-%d %H:%M:%S')] 休息完毕！"
        echo "▶️ 执行【继续】任务，当时显存已降至：${CLEARED_VRAM} MB"
        
        # 调用 API 恢复 OCR 任务
        curl -s -X PUT "$API_URL/jobs/$JOB_NAME" \
             -H "x-api-key: $API_KEY" \
             -H "Content-Type: application/json" \
             -d '{"command": "resume"}' > /dev/null
             
        echo "✅ =================================================="
        echo "" # 留个空行准备下一轮监控
    else
        # 安全状态下：在同一行实时刷新显存数字，不刷屏
        echo -ne "\r🟢 [$(date '+%H:%M:%S')] 监控中... 当前显存: ${VRAM} MB / 16384 MB  "
    fi

    # 每 3 秒钟扫描一次
    sleep 3
done

脚本日志输出情况：
在这里插入图片描述