Compare commits

..
Author SHA1 Message Date
abd4d22a67 HEL-543 返工: 补充 observability 运行时可关闭开关 (DATAHUB_OBSERVABILITY)
总工复核 🔴:安全边界要求新增观测功能必须可关闭、关闭后现有功能完全照旧,
但此前实现没有任何运行时开关。修复:

- settings.py: 新增 Settings.observability_enabled 字段,沿用既有
  DATAHUB_SCHEDULER 的环境变量模式,读取 DATAHUB_OBSERVABILITY
  (0/false/off 关闭,默认开启)。
- hub.py: Hub.__init__ 把 settings.observability_enabled 挂到
  self.db 上,让 pipeline/realtime_serve/steward/admin_api 已经
  在传的 db 参数直接带上开关,零额外改造。
- observability.py: 新增 is_enabled(db),缺失该属性时默认按启用处理
  (向后兼容裸 HubDB 用例/测试)。关闭时 observe() 变成纯
  透传(不计时、不分类、不碰数据库),record_call() 直接 no-op。
- admin_api.py: 4 个新只读端点关闭时返回明确的
  {"enabled": false, ...空结构} 而不是静默返回旧数据。
- 新增 10 个测试:开关默认值/环境变量解析、关闭后 observe() 的透传语义
  (含异常原样重新抛出)、关闭后 record_call() 零写入、关闭后重新开启恢复
  记录、4 个 admin 端点在关闭态的响应结构。

全量测试 183/183 通过(新增 10 个,含此前 173 个零回归)。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-14 01:10:58 +08:00
f014eb11bd HEL-543: add data hub observability side-channel (provider status, source catalog, lineage)
- New provider_call_log/provider_health tables (additive-only schema),
  wired via a fail-open observability.observe()/record_call() helper.
- Tushare pipeline keeps its existing src_calls record unchanged and now
  also feeds the unified provider_health/provider_call_log side channel.
- Eastmoney/Tencent realtime_serve.py call sites and the iFinD steward
  call site are wrapped with observability.observe() at the call site
  only; no adapter internals, routing, fallback order, or return values
  are touched.
- New read-only admin API endpoints: /admin/api/providers/status,
  /admin/api/source-catalog, /admin/api/lineage,
  /admin/api/lineage/affected.
- New static, read-only source_catalog.py and lineage.py registries
  documenting existing providers/interfaces/datasets and known
  main-site consumers (cited against backend/features/screener and
  backend/features/heaven call sites).
- provider_call_log is purged by the existing pipeline.cleanup() job
  alongside src_calls/job_runs.
- 47 new unit/integration tests covering classification, fail-open
  behavior under DB/log failures, unchanged payloads/exceptions on
  success and failure paths, and the new HTTP endpoints. Full suite:
  173 tests, all green.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-14 00:59:03 +08:00
3 changed files with 323 additions and 1660 deletions
+245 -1047
View File
File diff suppressed because it is too large Load Diff
+30 -89
View File
@@ -1,18 +1,16 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>小白复盘 · 数据中枢</title>
<link rel="stylesheet" href="/admin/styles.css" />
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>xiaobai-datahub 管理后台</title>
<link rel="stylesheet" href="/admin/styles.css" />
</head>
<body>
<div id="app">
<div id="login-view" class="auth-wrap bg-ambient" hidden>
<section class="panel auth-panel">
<div id="app">
<section id="login-view" class="panel auth-panel">
<h1>数据中枢</h1>
<p class="muted">内网管理后台 · 运行总览 / 数据源配置 / 数据血缘</p>
<p class="muted">内网管理后台,用于查看源状态、调度和盘后发布批次</p>
<form id="login-form">
<label>账号 <input name="username" value="hub_admin" autocomplete="username" /></label>
<label>密码 <input name="password" type="password" autocomplete="current-password" /></label>
@@ -20,93 +18,36 @@
<p id="login-error" class="error" hidden></p>
</form>
</section>
</div>
<div id="change-view" class="auth-wrap bg-ambient" hidden>
<section class="panel auth-panel">
<section id="change-view" class="panel auth-panel" hidden>
<h1>修改初始密码</h1>
<p class="muted">首次登录需先设置新密码才能进入。</p>
<form id="change-form">
<label>当前密码 <input name="current" type="password" autocomplete="current-password" /></label>
<label>新密码(至少 8 位) <input name="new_password" type="password" autocomplete="new-password" /></label>
<label>当前密码 <input name="current" type="password" /></label>
<label>新密码(至少 8 位) <input name="new_password" type="password" /></label>
<button type="submit">保存并继续</button>
<p id="change-error" class="error" hidden></p>
</form>
</section>
<section id="shell" hidden>
<header class="top">
<strong>xiaobai-datahub</strong>
<span id="phase" class="pill"></span>
<span id="who" class="muted"></span>
<button type="button" id="theme-btn" class="ghost">夜间</button>
<button type="button" id="logout-btn" class="ghost">退出</button>
</header>
<nav>
<button data-page="overview" class="active">总览</button>
<button data-page="sources">数据源</button>
<button data-page="jobs">调度任务</button>
<button data-page="release">盘后发布</button>
<button data-page="datasets">数据集</button>
<button data-page="audit">审计</button>
</nav>
<main id="page"></main>
</section>
</div>
<div class="app bg-ambient" id="appRoot" hidden>
<div class="bg-layer bg-gridlines z0"></div>
<div class="bg-layer scanlines z50"></div>
<header class="hdr">
<div class="hdr-in">
<div class="radar" style="width:26px;height:26px" id="radarLogo">
<div class="sweep radar-sweep"></div>
<div class="ring1" style="inset:5.72px"></div>
<div class="ring2" style="inset:9.88px"></div>
<div class="center"></div>
</div>
<div class="brand"><span class="b1">小白复盘 <span style="color:#22d3ee">·</span> 数据中枢</span><span class="b2">DATA-HUB · HEL-529</span></div>
<span class="vsep"></span>
<nav class="nav" id="navEl">
<button class="navbtn" data-nav="overview"><span class="tri"></span>运行总览<span class="en">OVERVIEW</span></button>
<button class="navbtn" data-nav="sources"><span class="tri"></span>数据源配置<span class="en">SOURCES</span></button>
<button class="navbtn" data-nav="lineage"><span class="tri"></span>数据血缘<span class="en">LINEAGE</span></button>
</nav>
<span class="flex1"></span>
<span class="mdtag" id="phaseTag"></span>
<span class="livespan" id="liveSpan"><span class="livedot pulse-dot"></span>LIVE</span>
<span class="clock num" id="clock"><span id="ckD"></span><span class="csep">|</span><span class="ct"><span id="ckH"></span><span class="csep">:</span><span id="ckM"></span><span class="csep">:</span><span id="ckS"></span></span></span>
<span class="vsep"></span>
<button class="tbtn" id="opsBtn" style="font-size:10px">调度 / 发布 / 审计</button>
<button class="tbtn" id="calmBtn" style="font-size:10px">减少动态</button>
<span id="who" class="muted" style="font-size:10px"></span>
<button class="tbtn" id="logout-btn" style="font-size:10px">退出</button>
</div>
</header>
<main class="main" id="mainEl"></main>
<footer class="tape">
<span class="tape-label" id="tapeLed">EVENT TAPE</span>
<div class="tape-view"><div class="marquee-track" id="marqueeTrack"><span style="display:inline-flex;align-items:center" id="tapeA"></span><span style="display:inline-flex;align-items:center" id="tapeB"></span></div></div>
</footer>
</div>
<div class="drawer-mask" id="drawerMask">
<aside class="drawer">
<div class="drawer-hd">
<span class="ttl">调度 / 发布 / 审计</span>
<span class="sub" id="drawerSub"></span>
<button class="drawer-close" id="drawerClose">×</button>
</div>
<div class="drawer-tabs" id="drawerTabs">
<button class="drawer-tab" data-dtab="jobs">调度任务</button>
<button class="drawer-tab" data-dtab="release">盘后发布</button>
<button class="drawer-tab" data-dtab="audit">审计</button>
</div>
<div class="drawer-body" id="drawerBody"></div>
</aside>
</div>
<div class="modal-mask" id="modalMask">
<div class="modal-box">
<h3 id="modalTitle">危险操作确认</h3>
<p id="modalDesc"></p>
<label>管理密码 <input id="modalPassword" type="password" autocomplete="current-password" /></label>
<label id="modalConfirmWrap">请输入确认词 <span id="modalConfirmWord" class="num" style="color:#fbbf24"></span> <input id="modalConfirm" type="text" autocomplete="off" /></label>
<p class="modal-err" id="modalErr"></p>
<div class="modal-actions">
<button class="tbtn" id="modalCancel">取消</button>
<button class="tbtn danger" id="modalOk">确认执行</button>
</div>
</div>
</div>
<div class="toast" id="toastEl"></div>
</div>
<script src="/admin/app.js"></script>
<script src="/admin/app.js"></script>
</body>
</html>
File diff suppressed because one or more lines are too long