HEL-543: add data hub observability side-channel (provider status, source catalog, lineage)

- New provider_call_log/provider_health tables (additive-only schema),
  wired via a fail-open observability.observe()/record_call() helper.
- Tushare pipeline keeps its existing src_calls record unchanged and now
  also feeds the unified provider_health/provider_call_log side channel.
- Eastmoney/Tencent realtime_serve.py call sites and the iFinD steward
  call site are wrapped with observability.observe() at the call site
  only; no adapter internals, routing, fallback order, or return values
  are touched.
- New read-only admin API endpoints: /admin/api/providers/status,
  /admin/api/source-catalog, /admin/api/lineage,
  /admin/api/lineage/affected.
- New static, read-only source_catalog.py and lineage.py registries
  documenting existing providers/interfaces/datasets and known
  main-site consumers (cited against backend/features/screener and
  backend/features/heaven call sites).
- provider_call_log is purged by the existing pipeline.cleanup() job
  alongside src_calls/job_runs.
- 47 new unit/integration tests covering classification, fail-open
  behavior under DB/log failures, unchanged payloads/exceptions on
  success and failure paths, and the new HTTP endpoints. Full suite:
  173 tests, all green.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>

xiaobai-datahub/datahub/pipeline.py

@@ -20,6 +20,7 @@ from datahub.governance.ratelimit import TokenBucket
 from datahub.governance.retry import RetryError, retry_call
 from datahub.logutil import get_logger
 from datahub.normalize import finite_number, normalize_daily
+from datahub import observability
 from datahub.revision import (
     compare_fields,
     diff_published_vs_upstream,
@@ -1634,6 +1635,7 @@ class Pipeline:
                 deleted += cur.rowcount
             connection.execute("DELETE FROM job_runs WHERE started_at < ?", (cutoff_jobs,))
             connection.execute("DELETE FROM src_calls WHERE created_at < ?", (cutoff_jobs,))
+            connection.execute("DELETE FROM provider_call_log WHERE created_at < ?", (cutoff_jobs,))
         return {"staging_deleted": deleted}
 
     def audit(self, actor: str, action: str, target: str = "", detail: str = "") -> None:
@@ -1768,6 +1770,18 @@ class Pipeline:
             "INSERT INTO src_calls(provider, endpoint, ok, latency_ms, error, created_at) VALUES (?,?,?,?,?,?)",
             ("tushare", endpoint, 1 if ok else 0, latency_ms, error, isoformat(self.clock())),
         )
+        # HEL-543 side channel: unified cross-provider call log/health. Kept
+        # strictly additive and fail-open; the src_calls insert above (the
+        # existing, already-compatible Tushare record) is unaffected either
+        # way.
+        if ok:
+            status, reason = "ok", ""
+        else:
+            status, reason = observability.classify_error(error)
+        observability.record_call(
+            self.db, "tushare", endpoint,
+            status=status, latency_ms=latency_ms, error=error, fallback_reason=reason,
+        )
 
     def _persist_health(self, state: str, error: str = "") -> None:
         snap = self.breaker.snapshot()