[{"data":1,"prerenderedAt":579},["ShallowReactive",2],{"article:agent-observability-delivery":3},{"meta":4,"raw":17,"references":18,"prev":67,"next":75,"body":84,"toc":569},{"slug":5,"title":6,"date":7,"updated":7,"tags":8,"summary":13,"draft":14,"lang":15,"minutes":16},"agent-observability-delivery","Agent 怎么持续优化与上线：Trace、评测、SLO 和灰度交付的证据链","2026-10-11",[9,10,11,12],"Agent","可观测性","评测","CI\u002FCD","对照 OpenAI tracing 与 harness 实践、Anthropic 长任务和评测方法，建立跨模型、工具、队列的观测链；解释 SLO、费用归因、Docker\u002FK8s 发布和可回滚的优化实验。",false,"zh-CN",8,"Agent 回答“完成了”，接口返回 200，服务器也没有异常，但任务仍可能失败：来源过期、工具改错对象、报告缺一项，或成本超过预算。**可观测性解释发生了什么；评测判断是否做对；交付流程决定哪些变化能进入生产。** 三者需要使用同一组任务和版本标识。\n\n本文以截至 2026-10-11 的官方资料为基础。下面的指标口径、发布门槛与流程是可实施的工程建议；示例数字不代表已经测得的服务表现。\n\n## 公开实践：两家公司把验证放在什么位置\n\n[OpenAI Agents SDK tracing](https:\u002F\u002Fopenai.github.io\u002Fopenai-agents-python\u002Ftracing\u002F)提供模型、工具、handoff 和 guardrails 等运行观察。[Harness engineering](https:\u002F\u002Fopenai.com\u002Findex\u002Fharness-engineering\u002F)描述了让 Agent 访问隔离应用环境、浏览器与观测工具，并用机械检查约束架构的经验；这是特定工程实践，不是所有团队必得的效率倍数。\n\nOpenAI 的 [Agent improvement loop 示例](https:\u002F\u002Fdevelopers.openai.com\u002Fcookbook\u002Fexamples\u002Fagents_sdk\u002Fagent_improvement_loop)进一步展示了运行记录、人工反馈、评测与下一轮改动如何连接。可借鉴的是反馈形成回归证据的闭环，而不是直接复制示例的业务与数据。\n\nAnthropic 的 [长任务 harness 报告](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Feffective-harnesses-for-long-running-agents)用初始环境、功能清单、进度记录与增量验证帮助跨会话继续工作。[Demystifying evals](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents)则区分任务、试验轨迹、最终结果和不同 grader。两者共同提醒：保存进度不等于验证完成，轨迹看起来合理也不等于环境结果正确。\n\n## 工程推导：贯穿一次任务的 Trace\n\n![Agent 观测与发布闭环：任务 Trace、分层验收、逐版本实验、灰度与故障回流](\u002Fdiagrams\u002Fagent-observability-delivery.svg)\n\n*图 1 · Trace 解释过程，验收判断结果；同一 release_id 连接离线实验与线上观测。*\n\n建议一次任务的 trace 覆盖入口接收、排队、检索、重排、上下文构建、每轮模型调用、工具执行、验收与结果交付。跨队列显式传播 trace 上下文，重试创建独立 attempt，暂停后恢复可通过关联关系连接；不要假定本进程自动追踪能贯穿所有外部系统。\n\n记录业务 `task_id`、`attempt_id`、`operation_id`、配置 `release_id` 与厂商请求 ID，遇到失败能定位同一件事。它们与 tracing ID 的含义不同：一次任务可以跨多个 trace，一次逻辑操作可以在多个 attempt 中对账。\n\n普通日志保存事件类别、结果、耗时和可用的安全关联 ID。必要的详细样本放受控存储，定义授权、脱敏与留存。SDK 可能自动收集提示和工具参数：OpenAI tracing 文档明确 `trace_include_sensitive_data` 默认开启；上线前按数据约束配置，关闭正文采集后仍要检查自定义 metadata、错误文本和自定义 exporter。\n\n## 指标：把等待、生成与完成拆开\n\n| 指标 | 建议口径 | 能帮助定位什么 |\n| --- | --- | --- |\n| 接收与队列等待 | 接收到持久确认；入队到领取 | 入口、配额与调度瓶颈 |\n| 首 token 时间 | 模型请求或端到端起点分别计时，明确起点 | 网络、prefill 与生成等待 |\n| 首个有用结果时间 | 用户得到可采取行动的结果 | 工具型 Agent 的实际等待体验 |\n| 任务完成时间 | 接收至通过验收并交付 | 全链路用户体验 |\n| 成功与失败分类 | 验收成功、模型错误、工具失败、证据不足、取消等 | 不用 HTTP 200 代替正确性 |\n| 每成功任务成本 | 全部尝试成本除以验收成功任务数 | 失败、重试与审核的真实开销 |\n\np50 和 p95 都要看，并注明任务类型和观察窗口。多步 Agent 可以快速吐出一句“开始处理”，但核心工具仍在等；首 token 改善不自动等于完成更快。具体延迟拆解见[TTFT 篇](\u002Farticles\u002Fttft-first-token-latency)。\n\n费用按模型、缓存、工具、环境、存储、审核与失败重试归因。供应商 usage 是核算依据，预估用于准入和预算；缓存命中与写入、压缩调用、隐藏推理费用按各自契约记录。指标标签只用模型、版本、工具类别等有限维度，用户 ID 和自由文本留在受控日志，避免高基数爆炸。[OpenTelemetry 语义约定](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fconcepts\u002Fsemantic-conventions\u002F)有助统一命名；具体 GenAI 属性仍应固定所用约定版本。\n\n## SLO：用户承诺需要可计算的分子与分母\n\n**示例目标**：在一个明确窗口内，99.5% 的“已接受、属于支持任务类型、无需等待人工审批”的任务，在 30 秒内通过验收并交付。实现前还要定义：何时算接受、取消怎样计入、依赖故障如何分类、低流量时怎样报告不确定性，以及长任务是否使用另一套目标。\n\n如果窗口内有 10,000 个符合定义的任务，0.5% 对应 50 个允许的失败或超时；这是示例 error budget。不能把失败排除出分母来美化 SLO，也不能把“服务可用”与“答案正确”混成同一个百分比。[Google SRE SLO 指南](https:\u002F\u002Fsre.google\u002Fworkbook\u002Fimplementing-slos\u002F)提供了指标与用户体验连接的原始方法。\n\n告警优先针对用户影响与预算消耗：队列增长、长期没有进展、关键工具失败率、权限门槛异常、费用突增和错误预算快速消耗。只盯 CPU 或统一 5xx 会漏掉大量业务失败。告警必须有责任人、诊断步骤、限制流量或回滚动作，并演练人工通道是否可达。\n\n## 评测：验证最终环境，不强求同一条路径\n\n工具 Agent 的测试应冻结初始数据、权限、时钟相关条件与可用工具，运行后验证目标对象和副作用。允许合理的不同调用路径；只有顺序本身决定安全或业务规则时，才要求固定顺序。模型裁判负责开放质量，确定性校验负责可计算事实，人工抽检负责校准与争议。\n\n[τ-bench 论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045)用对话后的数据库状态与目标状态比较，并研究多次运行的一致性。它说明为何“措辞像完成”不足以判断工具任务成功；其旧模型分数不能直接作为当前产品排名。\n\n重放要区分两类：录制工具结果的回放可定位 prompt 或模型差异，但不能证明真实工具仍然可用；隔离环境中的端到端重跑可验证实际交互，却需要控制数据、时间与外部依赖。详见[评测闭环](\u002Farticles\u002Fllm-eval-loop)。\n\n## Docker、K8s 与 CI\u002FCD：发布的是可恢复行为\n\n把应用代码、依赖锁定文件和行为配置构建为可追踪的发布产物。Docker 镜像用 digest 识别，密钥运行时注入，CPU、内存、临时盘与网络权限有界；容器只提供部分隔离，需要结合运行权限和工具策略。\n\nCI 按改动影响检查：代码与类型、schema 和旧客户端契约、权限和幂等、代表性 Agent 评测、费用与延迟趋势。慢或昂贵的完整评测可分层执行，但关键门槛必须在受控流程中真正执行；报告保留逐样本结果与版本，不能只有总分。\n\n[Kubernetes probes](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fprobes\u002F)区分 startup、readiness 与 liveness。readiness 决定是否继续接收流量，liveness 针对进程需要重启的故障；依赖短暂故障若触发所有实例重启，可能进一步扩大事故。任务 worker 停止领取新任务的机制仍要由应用实现。\n\n滚动升级时停止接收或领取新任务，在宽限期内完成可结束步骤，并把剩余任务交回持久化恢复机制；不能假设长任务总能在关机前结束。[Pod 生命周期](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F)描述终止流程，[Deployment](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fdeployment\u002F)描述滚动更新，但它们不自动保存 Agent 内存中的进度。\n\n数据库迁移采用先扩展兼容结构、再切流量、最后清理旧结构的流程。灰度同时观察新旧版本的关键切片，准备回退路由、镜像与行为配置。外部已发生的动作需要对账，数据迁移可能需要单独恢复方案；回滚不能只恢复一个 prompt 字符串。\n\n## 优化实验：每次改善要能说明因果与代价\n\n先用 trace 区分模型慢、排队慢、检索差或工具失效，再选择改动。可以比较简化工具描述、证据去重、小模型路由、受限并行与缓存；不要一次换模型、prompt、索引和并发，再把全部改善归因给其中一个。\n\n实验记录至少包括：假设、基线、改动、任务分布、模型与配置版本、冷\u002F热条件、运行次数、逐任务成功、p95 和总费用。线上并行实验应控制曝光与资源竞争；新版本抢走旧版本配额可能使比较失真。\n\n发布顺序为离线回归 → 隔离端到端验证 → 受控影子请求 → 小范围灰度 → 扩大观察。失败样本经授权脱敏后回流评测集，严重问题写入 incident 复盘和工程约束。优化只有在质量、安全、成本和用户等待的约束下成立，才具有生产意义。\n\n## 参考资料\n\n- [OpenAI Agents SDK — Tracing](https:\u002F\u002Fopenai.github.io\u002Fopenai-agents-python\u002Ftracing\u002F)\n- [OpenAI — Harness engineering](https:\u002F\u002Fopenai.com\u002Findex\u002Fharness-engineering\u002F)\n- [OpenAI Cookbook — Agent improvement loop](https:\u002F\u002Fdevelopers.openai.com\u002Fcookbook\u002Fexamples\u002Fagents_sdk\u002Fagent_improvement_loop)\n- [Anthropic — Effective harnesses for long-running agents](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Feffective-harnesses-for-long-running-agents)\n- [Anthropic — Demystifying evals for AI agents](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents)\n- [OpenTelemetry — Semantic conventions](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fconcepts\u002Fsemantic-conventions\u002F)\n- [Google SRE — Implementing SLOs](https:\u002F\u002Fsre.google\u002Fworkbook\u002Fimplementing-slos\u002F)\n- [τ-bench — Tool-Agent-User Interaction in Real-World Domains](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045)\n- [Kubernetes — Probes](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fprobes\u002F)\n- [Kubernetes — Pod lifecycle](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F)\n- [Kubernetes — Deployments](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fdeployment\u002F)",[19,24,29,34,38,42,46,50,55,59,63],{"title":20,"url":21,"type":22,"note":23,"checked":7},"OpenAI Agents SDK — Tracing","https:\u002F\u002Fopenai.github.io\u002Fopenai-agents-python\u002Ftracing\u002F","官方文档","SDK trace\u002Fspan、敏感内容采集与配置；跨队列与业务步骤需自行接入。",{"title":25,"url":26,"type":27,"note":28,"checked":7},"OpenAI — Harness engineering","https:\u002F\u002Fopenai.com\u002Findex\u002Fharness-engineering\u002F","工程报告","仓库知识、隔离验收环境与机械约束的工程实践；案例效率不能视为普遍承诺。",{"title":30,"url":31,"type":32,"note":33,"checked":7},"OpenAI Cookbook — Agent improvement loop","https:\u002F\u002Fdevelopers.openai.com\u002Fcookbook\u002Fexamples\u002Fagents_sdk\u002Fagent_improvement_loop","官方示例","运行记录、反馈、评测与改动的闭环示例；需按自己的业务与数据约束适配。",{"title":35,"url":36,"type":27,"note":37,"checked":7},"Anthropic — Effective harnesses for long-running agents","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Feffective-harnesses-for-long-running-agents","初始环境、功能清单与增量进度帮助跨会话工作；限于报告中的任务与模型。",{"title":39,"url":40,"type":27,"note":41,"checked":7},"Anthropic — Demystifying evals for AI agents","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents","任务、试验轨迹、结果与 grader 的评测方法；不代替本业务验收标准。",{"title":43,"url":44,"type":22,"note":45,"checked":7},"OpenTelemetry — Semantic conventions","https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fconcepts\u002Fsemantic-conventions\u002F","遥测语义约定的用途；具体属性与稳定性应固定所用版本。",{"title":47,"url":48,"type":22,"note":49,"checked":7},"Google SRE — Implementing SLOs","https:\u002F\u002Fsre.google\u002Fworkbook\u002Fimplementing-slos\u002F","SLI、SLO 与错误预算的制定方法；本文目标数值仅为示例。",{"title":51,"url":52,"type":53,"note":54,"checked":7},"τ-bench — Tool-Agent-User Interaction in Real-World Domains","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045","研究论文","工具对话后的数据库结果验证与多次一致性评测；论文分数有模型和环境边界。",{"title":56,"url":57,"type":22,"note":58,"checked":7},"Kubernetes — Probes","https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fprobes\u002F","startup、readiness、liveness 的不同作用；不自动保证 Agent 任务恢复。",{"title":60,"url":61,"type":22,"note":62,"checked":7},"Kubernetes — Pod lifecycle","https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F","Pod 终止与宽限期行为；应用仍需持久化进度和处理在途任务。",{"title":64,"url":65,"type":22,"note":66,"checked":7},"Kubernetes — Deployments","https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fdeployment\u002F","Deployment 滚动更新与版本恢复机制；业务数据与外部动作另行处理。",{"slug":68,"title":69,"date":7,"updated":7,"tags":70,"summary":74,"draft":14,"lang":15,"minutes":16},"agent-concurrency-state","Agent 服务的并发与状态：异步、任务队列、幂等和故障恢复怎么配合",[9,71,72,73],"服务端","并发","一致性","把厂商会话能力与业务一致性分开，解释 async I\u002FO、有限并发、事务 outbox、任务租约和稳定操作 ID；用崩溃窗口、慢客户端与外部结果未知来检验生产服务。",{"slug":76,"title":77,"date":7,"updated":7,"tags":78,"summary":82,"draft":14,"lang":15,"minutes":83},"production-agent-design","生产级 Agent 怎么设计：从 OpenAI 与 Anthropic 的公开实践看架构边界",[9,79,80,81],"OpenAI","Anthropic","生产架构","对照 Codex agent loop、Agents SDK\u002FAPI 与 Claude Managed Agents，解释模型、执行框架、会话和业务状态的职责；从成功标准出发，设计可恢复执行、权限边界与优化实验。",9,{"type":85,"children":86},"root",[87,103,108,115,136,149,170,176,185,194,199,236,249,255,396,409,422,428,438,451,456,462,467,478,490,496,501,506,517,538,543,549,554,559,564],{"type":88,"tag":89,"props":90,"children":91},"element","p",{},[92,95,101],{"type":93,"value":94},"text","Agent 回答“完成了”，接口返回 200，服务器也没有异常，但任务仍可能失败：来源过期、工具改错对象、报告缺一项，或成本超过预算。",{"type":88,"tag":96,"props":97,"children":98},"strong",{},[99],{"type":93,"value":100},"可观测性解释发生了什么；评测判断是否做对；交付流程决定哪些变化能进入生产。",{"type":93,"value":102}," 三者需要使用同一组任务和版本标识。",{"type":88,"tag":89,"props":104,"children":105},{},[106],{"type":93,"value":107},"本文以截至 2026-10-11 的官方资料为基础。下面的指标口径、发布门槛与流程是可实施的工程建议；示例数字不代表已经测得的服务表现。",{"type":88,"tag":109,"props":110,"children":112},"h2",{"id":111},"公开实践两家公司把验证放在什么位置",[113],{"type":93,"value":114},"公开实践：两家公司把验证放在什么位置",{"type":88,"tag":89,"props":116,"children":117},{},[118,126,128,134],{"type":88,"tag":119,"props":120,"children":123},"a",{"href":21,"rel":121},[122],"nofollow",[124],{"type":93,"value":125},"OpenAI Agents SDK tracing",{"type":93,"value":127},"提供模型、工具、handoff 和 guardrails 等运行观察。",{"type":88,"tag":119,"props":129,"children":131},{"href":26,"rel":130},[122],[132],{"type":93,"value":133},"Harness engineering",{"type":93,"value":135},"描述了让 Agent 访问隔离应用环境、浏览器与观测工具，并用机械检查约束架构的经验；这是特定工程实践，不是所有团队必得的效率倍数。",{"type":88,"tag":89,"props":137,"children":138},{},[139,141,147],{"type":93,"value":140},"OpenAI 的 ",{"type":88,"tag":119,"props":142,"children":144},{"href":31,"rel":143},[122],[145],{"type":93,"value":146},"Agent improvement loop 示例",{"type":93,"value":148},"进一步展示了运行记录、人工反馈、评测与下一轮改动如何连接。可借鉴的是反馈形成回归证据的闭环，而不是直接复制示例的业务与数据。",{"type":88,"tag":89,"props":150,"children":151},{},[152,154,160,162,168],{"type":93,"value":153},"Anthropic 的 ",{"type":88,"tag":119,"props":155,"children":157},{"href":36,"rel":156},[122],[158],{"type":93,"value":159},"长任务 harness 报告",{"type":93,"value":161},"用初始环境、功能清单、进度记录与增量验证帮助跨会话继续工作。",{"type":88,"tag":119,"props":163,"children":165},{"href":40,"rel":164},[122],[166],{"type":93,"value":167},"Demystifying evals",{"type":93,"value":169},"则区分任务、试验轨迹、最终结果和不同 grader。两者共同提醒：保存进度不等于验证完成，轨迹看起来合理也不等于环境结果正确。",{"type":88,"tag":109,"props":171,"children":173},{"id":172},"工程推导贯穿一次任务的-trace",[174],{"type":93,"value":175},"工程推导：贯穿一次任务的 Trace",{"type":88,"tag":89,"props":177,"children":178},{},[179],{"type":88,"tag":180,"props":181,"children":184},"img",{"alt":182,"src":183},"Agent 观测与发布闭环：任务 Trace、分层验收、逐版本实验、灰度与故障回流","\u002Fdiagrams\u002Fagent-observability-delivery.svg",[],{"type":88,"tag":89,"props":186,"children":187},{},[188],{"type":88,"tag":189,"props":190,"children":191},"em",{},[192],{"type":93,"value":193},"图 1 · Trace 解释过程，验收判断结果；同一 release_id 连接离线实验与线上观测。",{"type":88,"tag":89,"props":195,"children":196},{},[197],{"type":93,"value":198},"建议一次任务的 trace 覆盖入口接收、排队、检索、重排、上下文构建、每轮模型调用、工具执行、验收与结果交付。跨队列显式传播 trace 上下文，重试创建独立 attempt，暂停后恢复可通过关联关系连接；不要假定本进程自动追踪能贯穿所有外部系统。",{"type":88,"tag":89,"props":200,"children":201},{},[202,204,211,213,219,220,226,228,234],{"type":93,"value":203},"记录业务 ",{"type":88,"tag":205,"props":206,"children":208},"code",{"className":207},[],[209],{"type":93,"value":210},"task_id",{"type":93,"value":212},"、",{"type":88,"tag":205,"props":214,"children":216},{"className":215},[],[217],{"type":93,"value":218},"attempt_id",{"type":93,"value":212},{"type":88,"tag":205,"props":221,"children":223},{"className":222},[],[224],{"type":93,"value":225},"operation_id",{"type":93,"value":227},"、配置 ",{"type":88,"tag":205,"props":229,"children":231},{"className":230},[],[232],{"type":93,"value":233},"release_id",{"type":93,"value":235}," 与厂商请求 ID，遇到失败能定位同一件事。它们与 tracing ID 的含义不同：一次任务可以跨多个 trace，一次逻辑操作可以在多个 attempt 中对账。",{"type":88,"tag":89,"props":237,"children":238},{},[239,241,247],{"type":93,"value":240},"普通日志保存事件类别、结果、耗时和可用的安全关联 ID。必要的详细样本放受控存储，定义授权、脱敏与留存。SDK 可能自动收集提示和工具参数：OpenAI tracing 文档明确 ",{"type":88,"tag":205,"props":242,"children":244},{"className":243},[],[245],{"type":93,"value":246},"trace_include_sensitive_data",{"type":93,"value":248}," 默认开启；上线前按数据约束配置，关闭正文采集后仍要检查自定义 metadata、错误文本和自定义 exporter。",{"type":88,"tag":109,"props":250,"children":252},{"id":251},"指标把等待生成与完成拆开",[253],{"type":93,"value":254},"指标：把等待、生成与完成拆开",{"type":88,"tag":256,"props":257,"children":258},"table",{},[259,283],{"type":88,"tag":260,"props":261,"children":262},"thead",{},[263],{"type":88,"tag":264,"props":265,"children":266},"tr",{},[267,273,278],{"type":88,"tag":268,"props":269,"children":270},"th",{},[271],{"type":93,"value":272},"指标",{"type":88,"tag":268,"props":274,"children":275},{},[276],{"type":93,"value":277},"建议口径",{"type":88,"tag":268,"props":279,"children":280},{},[281],{"type":93,"value":282},"能帮助定位什么",{"type":88,"tag":284,"props":285,"children":286},"tbody",{},[287,306,324,342,360,378],{"type":88,"tag":264,"props":288,"children":289},{},[290,296,301],{"type":88,"tag":291,"props":292,"children":293},"td",{},[294],{"type":93,"value":295},"接收与队列等待",{"type":88,"tag":291,"props":297,"children":298},{},[299],{"type":93,"value":300},"接收到持久确认；入队到领取",{"type":88,"tag":291,"props":302,"children":303},{},[304],{"type":93,"value":305},"入口、配额与调度瓶颈",{"type":88,"tag":264,"props":307,"children":308},{},[309,314,319],{"type":88,"tag":291,"props":310,"children":311},{},[312],{"type":93,"value":313},"首 token 时间",{"type":88,"tag":291,"props":315,"children":316},{},[317],{"type":93,"value":318},"模型请求或端到端起点分别计时，明确起点",{"type":88,"tag":291,"props":320,"children":321},{},[322],{"type":93,"value":323},"网络、prefill 与生成等待",{"type":88,"tag":264,"props":325,"children":326},{},[327,332,337],{"type":88,"tag":291,"props":328,"children":329},{},[330],{"type":93,"value":331},"首个有用结果时间",{"type":88,"tag":291,"props":333,"children":334},{},[335],{"type":93,"value":336},"用户得到可采取行动的结果",{"type":88,"tag":291,"props":338,"children":339},{},[340],{"type":93,"value":341},"工具型 Agent 的实际等待体验",{"type":88,"tag":264,"props":343,"children":344},{},[345,350,355],{"type":88,"tag":291,"props":346,"children":347},{},[348],{"type":93,"value":349},"任务完成时间",{"type":88,"tag":291,"props":351,"children":352},{},[353],{"type":93,"value":354},"接收至通过验收并交付",{"type":88,"tag":291,"props":356,"children":357},{},[358],{"type":93,"value":359},"全链路用户体验",{"type":88,"tag":264,"props":361,"children":362},{},[363,368,373],{"type":88,"tag":291,"props":364,"children":365},{},[366],{"type":93,"value":367},"成功与失败分类",{"type":88,"tag":291,"props":369,"children":370},{},[371],{"type":93,"value":372},"验收成功、模型错误、工具失败、证据不足、取消等",{"type":88,"tag":291,"props":374,"children":375},{},[376],{"type":93,"value":377},"不用 HTTP 200 代替正确性",{"type":88,"tag":264,"props":379,"children":380},{},[381,386,391],{"type":88,"tag":291,"props":382,"children":383},{},[384],{"type":93,"value":385},"每成功任务成本",{"type":88,"tag":291,"props":387,"children":388},{},[389],{"type":93,"value":390},"全部尝试成本除以验收成功任务数",{"type":88,"tag":291,"props":392,"children":393},{},[394],{"type":93,"value":395},"失败、重试与审核的真实开销",{"type":88,"tag":89,"props":397,"children":398},{},[399,401,407],{"type":93,"value":400},"p50 和 p95 都要看，并注明任务类型和观察窗口。多步 Agent 可以快速吐出一句“开始处理”，但核心工具仍在等；首 token 改善不自动等于完成更快。具体延迟拆解见",{"type":88,"tag":119,"props":402,"children":404},{"href":403},"\u002Farticles\u002Fttft-first-token-latency",[405],{"type":93,"value":406},"TTFT 篇",{"type":93,"value":408},"。",{"type":88,"tag":89,"props":410,"children":411},{},[412,414,420],{"type":93,"value":413},"费用按模型、缓存、工具、环境、存储、审核与失败重试归因。供应商 usage 是核算依据，预估用于准入和预算；缓存命中与写入、压缩调用、隐藏推理费用按各自契约记录。指标标签只用模型、版本、工具类别等有限维度，用户 ID 和自由文本留在受控日志，避免高基数爆炸。",{"type":88,"tag":119,"props":415,"children":417},{"href":44,"rel":416},[122],[418],{"type":93,"value":419},"OpenTelemetry 语义约定",{"type":93,"value":421},"有助统一命名；具体 GenAI 属性仍应固定所用约定版本。",{"type":88,"tag":109,"props":423,"children":425},{"id":424},"slo用户承诺需要可计算的分子与分母",[426],{"type":93,"value":427},"SLO：用户承诺需要可计算的分子与分母",{"type":88,"tag":89,"props":429,"children":430},{},[431,436],{"type":88,"tag":96,"props":432,"children":433},{},[434],{"type":93,"value":435},"示例目标",{"type":93,"value":437},"：在一个明确窗口内，99.5% 的“已接受、属于支持任务类型、无需等待人工审批”的任务，在 30 秒内通过验收并交付。实现前还要定义：何时算接受、取消怎样计入、依赖故障如何分类、低流量时怎样报告不确定性，以及长任务是否使用另一套目标。",{"type":88,"tag":89,"props":439,"children":440},{},[441,443,449],{"type":93,"value":442},"如果窗口内有 10,000 个符合定义的任务，0.5% 对应 50 个允许的失败或超时；这是示例 error budget。不能把失败排除出分母来美化 SLO，也不能把“服务可用”与“答案正确”混成同一个百分比。",{"type":88,"tag":119,"props":444,"children":446},{"href":48,"rel":445},[122],[447],{"type":93,"value":448},"Google SRE SLO 指南",{"type":93,"value":450},"提供了指标与用户体验连接的原始方法。",{"type":88,"tag":89,"props":452,"children":453},{},[454],{"type":93,"value":455},"告警优先针对用户影响与预算消耗：队列增长、长期没有进展、关键工具失败率、权限门槛异常、费用突增和错误预算快速消耗。只盯 CPU 或统一 5xx 会漏掉大量业务失败。告警必须有责任人、诊断步骤、限制流量或回滚动作，并演练人工通道是否可达。",{"type":88,"tag":109,"props":457,"children":459},{"id":458},"评测验证最终环境不强求同一条路径",[460],{"type":93,"value":461},"评测：验证最终环境，不强求同一条路径",{"type":88,"tag":89,"props":463,"children":464},{},[465],{"type":93,"value":466},"工具 Agent 的测试应冻结初始数据、权限、时钟相关条件与可用工具，运行后验证目标对象和副作用。允许合理的不同调用路径；只有顺序本身决定安全或业务规则时，才要求固定顺序。模型裁判负责开放质量，确定性校验负责可计算事实，人工抽检负责校准与争议。",{"type":88,"tag":89,"props":468,"children":469},{},[470,476],{"type":88,"tag":119,"props":471,"children":473},{"href":52,"rel":472},[122],[474],{"type":93,"value":475},"τ-bench 论文",{"type":93,"value":477},"用对话后的数据库状态与目标状态比较，并研究多次运行的一致性。它说明为何“措辞像完成”不足以判断工具任务成功；其旧模型分数不能直接作为当前产品排名。",{"type":88,"tag":89,"props":479,"children":480},{},[481,483,489],{"type":93,"value":482},"重放要区分两类：录制工具结果的回放可定位 prompt 或模型差异，但不能证明真实工具仍然可用；隔离环境中的端到端重跑可验证实际交互，却需要控制数据、时间与外部依赖。详见",{"type":88,"tag":119,"props":484,"children":486},{"href":485},"\u002Farticles\u002Fllm-eval-loop",[487],{"type":93,"value":488},"评测闭环",{"type":93,"value":408},{"type":88,"tag":109,"props":491,"children":493},{"id":492},"dockerk8s-与-cicd发布的是可恢复行为",[494],{"type":93,"value":495},"Docker、K8s 与 CI\u002FCD：发布的是可恢复行为",{"type":88,"tag":89,"props":497,"children":498},{},[499],{"type":93,"value":500},"把应用代码、依赖锁定文件和行为配置构建为可追踪的发布产物。Docker 镜像用 digest 识别，密钥运行时注入，CPU、内存、临时盘与网络权限有界；容器只提供部分隔离，需要结合运行权限和工具策略。",{"type":88,"tag":89,"props":502,"children":503},{},[504],{"type":93,"value":505},"CI 按改动影响检查：代码与类型、schema 和旧客户端契约、权限和幂等、代表性 Agent 评测、费用与延迟趋势。慢或昂贵的完整评测可分层执行，但关键门槛必须在受控流程中真正执行；报告保留逐样本结果与版本，不能只有总分。",{"type":88,"tag":89,"props":507,"children":508},{},[509,515],{"type":88,"tag":119,"props":510,"children":512},{"href":57,"rel":511},[122],[513],{"type":93,"value":514},"Kubernetes probes",{"type":93,"value":516},"区分 startup、readiness 与 liveness。readiness 决定是否继续接收流量，liveness 针对进程需要重启的故障；依赖短暂故障若触发所有实例重启，可能进一步扩大事故。任务 worker 停止领取新任务的机制仍要由应用实现。",{"type":88,"tag":89,"props":518,"children":519},{},[520,522,528,530,536],{"type":93,"value":521},"滚动升级时停止接收或领取新任务，在宽限期内完成可结束步骤，并把剩余任务交回持久化恢复机制；不能假设长任务总能在关机前结束。",{"type":88,"tag":119,"props":523,"children":525},{"href":61,"rel":524},[122],[526],{"type":93,"value":527},"Pod 生命周期",{"type":93,"value":529},"描述终止流程，",{"type":88,"tag":119,"props":531,"children":533},{"href":65,"rel":532},[122],[534],{"type":93,"value":535},"Deployment",{"type":93,"value":537},"描述滚动更新，但它们不自动保存 Agent 内存中的进度。",{"type":88,"tag":89,"props":539,"children":540},{},[541],{"type":93,"value":542},"数据库迁移采用先扩展兼容结构、再切流量、最后清理旧结构的流程。灰度同时观察新旧版本的关键切片，准备回退路由、镜像与行为配置。外部已发生的动作需要对账，数据迁移可能需要单独恢复方案；回滚不能只恢复一个 prompt 字符串。",{"type":88,"tag":109,"props":544,"children":546},{"id":545},"优化实验每次改善要能说明因果与代价",[547],{"type":93,"value":548},"优化实验：每次改善要能说明因果与代价",{"type":88,"tag":89,"props":550,"children":551},{},[552],{"type":93,"value":553},"先用 trace 区分模型慢、排队慢、检索差或工具失效，再选择改动。可以比较简化工具描述、证据去重、小模型路由、受限并行与缓存；不要一次换模型、prompt、索引和并发，再把全部改善归因给其中一个。",{"type":88,"tag":89,"props":555,"children":556},{},[557],{"type":93,"value":558},"实验记录至少包括：假设、基线、改动、任务分布、模型与配置版本、冷\u002F热条件、运行次数、逐任务成功、p95 和总费用。线上并行实验应控制曝光与资源竞争；新版本抢走旧版本配额可能使比较失真。",{"type":88,"tag":89,"props":560,"children":561},{},[562],{"type":93,"value":563},"发布顺序为离线回归 → 隔离端到端验证 → 受控影子请求 → 小范围灰度 → 扩大观察。失败样本经授权脱敏后回流评测集，严重问题写入 incident 复盘和工程约束。优化只有在质量、安全、成本和用户等待的约束下成立，才具有生产意义。",{"type":88,"tag":109,"props":565,"children":567},{"id":566},"参考资料",[568],{"type":93,"value":566},[570,572,573,574,575,576,577,578],{"id":111,"depth":571,"text":114},2,{"id":172,"depth":571,"text":175},{"id":251,"depth":571,"text":254},{"id":424,"depth":571,"text":427},{"id":458,"depth":571,"text":461},{"id":492,"depth":571,"text":495},{"id":545,"depth":571,"text":548},{"id":566,"depth":571,"text":566},1792400815607]