migration: close candidate maintenance audit

This commit is contained in:
leefer
2026-07-31 21:24:37 +08:00
parent bc25e06416
commit aabfb621aa
10 changed files with 728 additions and 91 deletions
+59 -29
View File
@@ -1,40 +1,70 @@
# Architecture # Candidate architecture
The normative governance contract is documented in `app/` is the behavior-preserving modular candidate. The original `webapp/` runtime remains
`docs/governance/architecture-standard.md`. This file describes the currently deployed the product and visual baseline until manual acceptance. `next/` is a rejected, frozen
shape; the standard defines the target boundaries and the rules applied during migration. implementation and is not a source for this directory.
The application intentionally keeps a small deployment footprint: one Python process, one The application deliberately remains a modular monolith: one Python process, one SQLite WAL
SQLite database, and a build-free browser client. The internal boundaries are nevertheless database, and a build-free HTML/CSS/JavaScript client. The migration changed source ownership
explicit so new features do not bypass account isolation or data-quality rules. and imports, not the technology stack or observable product behavior.
## Backend boundaries ## Runtime path
- `server.py`: application services and HTTP request/response wiring. ```text
- `api_access.py`: the single authorization policy for authenticated, member, and admin APIs. browser
- `app_config.py`: runtime paths, local environment loading, and shared input validation. -> frontend/shared/api.js
- `database.py`: SQLite schema, migrations, and persistence operations. -> backend HTTP transport and feature HTTP mixins
- `tushare_client.py` and `realtime_aggregator.py`: external market-data adapters. -> feature services
- `sentiment_engine.py`, `screener.py`, and `heaven_engine.py`: deterministic domain logic. -> repositories / DataGateway / LLMGateway
- `mentor_agent.py`, `heaven_agent.py`, and `llm_strategy.py`: bounded LLM adapters. -> SQLite / market providers / model providers
## Data ownership background scheduler
-> backend/jobs
-> the same feature services and repositories
```
Public market snapshots, stock factors, built-in strategies, limit-up reasons, seat aliases, ## Source ownership
and sector-element mappings are shared. Only administrators can modify shared knowledge.
Watchlists, review notes, custom strategies, screener runs, mentor conversations, birth data, - `server.py` is the stable command/import facade. Runtime composition lives in
alerts, trading journals, and assistant conversations are owned by a user ID and must be `backend/application.py` and `backend/bootstrap/`.
queried with that ID. LLM features additionally require active membership. - `backend/http/` owns common authentication, request IDs, responses, static delivery, and
error normalization. Feature-specific transport handlers live beside their feature.
- `backend/features/<feature>/` owns the mechanically moved service, repository, HTTP, agent,
or deterministic calculation code for that product area.
- `backend/data/` owns provider construction, source policy, provenance, units, freshness,
coverage, and display-versus-calculation eligibility.
- `backend/database/` owns connection management, ordered migrations, and narrow repository
adapters. Root `database.py` remains the legacy schema/composition anchor and combines the
feature repository mixins; do not add feature queries to it.
- `backend/jobs/` owns job definitions, locks, retries, idempotency, and persisted run state.
- `backend/llm/` owns model selection, membership/quota checks, fallback, streaming rules,
and call audit. Feature agents only prepare context and provider payloads.
- `frontend/shared/` is the only browser API/state/Shell/component boundary.
- `frontend/pages/` owns page-local behavior. The original runtime was split mechanically;
source markers and preservation tests prove that the pieces reassemble to the audited
original, apart from explicitly registered trial retirements.
- `frontend/styles/`, `frontend/shared/tokens.css`, and the Wentian page stylesheet preserve
the approved cascade and light/dark/mobile behavior.
- `config/` is the versioned registry for pages, features, APIs, datasets, quality rules,
jobs, and the generated candidate architecture inventory.
## Data integrity Root modules such as `screener.py`, `tushare_client.py`, and `mentor_agent.py` are compatibility
aliases to canonical modules. They contain no second implementation and remain only because
the original public import surface is part of the preservation contract.
Production reads never synthesize market prices. A failed live request may use the latest real ## Non-negotiable maintenance rules
snapshot at or before the requested date. When no real snapshot exists, the API reports that
the data is unavailable. Demo builders remain test fixtures only.
## Change contract 1. Preserve account ownership in every user-private query and test it with two accounts.
2. Browser requests go through `frontend/shared/api.js`; provider calls go through the data
boundary; model calls go through `backend/llm/`.
3. Calculation datasets fail closed when required source, date, unit, freshness, or coverage
evidence is missing. Display fallbacks do not silently enter calculations.
4. Do not implement logic in both a root compatibility module and a canonical module.
5. Do not remove compatibility or uncertain code without reference scanning, old/new
differential evidence, browser checks, and manual acceptance.
6. Run `python tools/verify_baseline.py` for every change and add `--e2e` when runtime or
frontend behavior can be affected.
New endpoints must be added to `api_access.required_role` when they need member or admin The authoritative migration constraints and handoff procedure are in
access. New user-owned tables must include `user_id`, an ownership index, and cross-account `../docs/migration/原版保真迁移总纲.md` and
tests. API payload compatibility is protected by the Python and Playwright suites. `../docs/migration/人工维护与本地切换指南.md`.
+9 -5
View File
@@ -2,23 +2,27 @@
一个面向 A 股盘后复盘的本地 Web 工作台。后端使用 Python 访问 Tushare Pro,前端不依赖构建工具。 一个面向 A 股盘后复盘的本地 Web 工作台。后端使用 Python 访问 Tushare Pro,前端不依赖构建工具。
本目录是从原版源码逐项移动、机械拆分并完成差分验证的模块化候选,不是依据规格书重新开发的
第二套产品。人工验收和正式切换前,`webapp/`根目录仍是唯一正式基线;冻结的`next/`不得用于
部署或后续开发。目录职责见[ARCHITECTURE.md](ARCHITECTURE.md)。
当前包含集合竞价、涨停池、炸板池、跌停板、昨日涨停、涨停表现、市场天梯、板块轮动、题材库、人气热榜、龙虎榜和个人复盘工作区。交易日快照与同步记录保存在本地 SQLite 数据库 `data/review.db` 当前包含集合竞价、涨停池、炸板池、跌停板、昨日涨停、涨停表现、市场天梯、板块轮动、题材库、人气热榜、龙虎榜和个人复盘工作区。交易日快照与同步记录保存在本地 SQLite 数据库 `data/review.db`
集合竞价中心采用盘前生命周期:9:15 前显示预告,9:15–9:25 明确等待最终竞价,9:25–9:30 自动读取并重试最终竞价筛选,9:30 后停止更新并冻结为复盘归档。当前 Tushare 只提供 9:25 最终竞价快照,不将其表述为动态虚拟撮合行情。 集合竞价中心采用盘前生命周期:9:15 前显示预告,9:15–9:25 明确等待最终竞价,9:25–9:30 自动读取并重试最终竞价筛选,9:30 后停止更新并冻结为复盘归档。当前 Tushare 只提供 9:25 最终竞价快照,不将其表述为动态虚拟撮合行情。
第三阶段加入了机构席位、席位别名、个股复权日 K、资金流、自选股、涨停原因修订、个股笔记、每日复盘和历史数据回补。 第三阶段加入了机构席位、席位别名、个股复权日 K、资金流、自选股、涨停原因修订、个股笔记、每日复盘和历史数据回补。
股票代码在桌面端悬停后会显示分时与日 K 快速预览,默认优先展示日 K;移动端点击代码后从底部打开预览面板。股票详情以及板块、题材、指数详情均可在日 K 与最新分时之间切换。日 K 复用个股详情缓存;分时使用隔离的东方财富分钟图表源和短时内存缓存,只负责展示,不写入主行情、不参与情绪、选股或问天计算。图表源不可用时界面会明确显示“分时不可用”,不会使用日 K 数据模拟分时走势。 股票代码在桌面端悬停后会显示分时与日 K 快速预览,默认优先展示日 K;移动端点击代码后从底部打开预览面板。股票详情以及板块、题材、指数详情均可在日 K 与最新分时之间切换。日 K 复用个股详情缓存;分时优先使用 iFinD,东方财富仅作隔离的展示兜底,并使用短时内存缓存。图表数据不写入主行情、不参与情绪、选股或问天计算不可用时明确显示“分时不可用”,不会用日 K 模拟分时走势。
智能选股模块包含 45 日全市场因子库、六阶段市场识别、七套内置策略、受控公式 DSL、自然语言策略编译、候选排名和滚动回测。竞价涨幅、竞价成交额、竞价换手率与竞价量比随因子数据一并同步,可用于自定义公式和历史回测。首次使用需在页面点击“同步因子数据”。未配置 LLM 时使用本地策略模板;配置兼容 API 后自动切换为主模型编译,主模型失败时自动使用辅助模型,两者均支持独立连通性测试 智能选股包含六阶段盘后候选、29 套精选策略、自定义公式 DSL、自然语言公式编译、候选排名和滚动回测。阶段与精选策略在当日行情更新后由后台确定性计算;自定义选股由用户手动执行,LLM 只负责编译自然语言条件,不参与候选筛选。竞价、估值、财务、资金、人气和席位等字段按已登记的数据可用性进入因子库,缺失时明确显示覆盖问题
每次选股结果会自动进入五交易日持续跟踪,展示 T+1 开盘/收盘、T+3、T+5、最大涨幅与最大回撤。提醒中心支持手工日期提醒,并在策略首日反馈和五日跟踪完成时生成账号私有的站内提醒。 候选只有经用户手动加入后才进入五交易日持续跟踪,展示 T+1 开盘/收盘、T+3、T+5、最大涨幅与最大回撤。提醒中心支持手工日期提醒,并在策略首日反馈和五日跟踪完成时生成账号私有的站内提醒。
问师模块会读取当前复盘、近十日市场情绪、涨跌停、昨日反馈、板块轮动、市场阶段、龙虎榜和指定个股数据,再按选中的游资思维 Skill 进行单师对话。对话记录按账号、老师和交易日期保存在服务端;主模型不可用时自动切换辅助模型。 问师模块会读取当前复盘、近十日市场情绪、涨跌停、昨日反馈、板块轮动、市场阶段、龙虎榜和指定个股数据,再按选中的游资思维 Skill 进行单师对话。对话记录按账号、老师和交易日期保存在服务端;主模型不可用时自动切换辅助模型。
新增公开问师角色时,在 `游资skills` 下增加一个包含 `SKILL.md` 的独立目录,并在 `游资skills/mentor_catalog.json` 中登记素材等级与结构质检。管理员私有角色放在 `data/private-mentor-skills`,该目录不进入 Git 或 Docker 镜像,且只会出现在管理员的问师列表中。系统会从 Skill 的 frontmatter、一级标题、核心模型和引用语中自动生成角色信息,无需修改注册代码。 新增公开问师角色时,在 `游资skills` 下增加一个包含 `SKILL.md` 的独立目录,并在 `游资skills/mentor_catalog.json` 中登记素材等级与结构质检。管理员私有角色放在 `data/private-mentor-skills`,该目录不进入 Git 或 Docker 镜像,且只会出现在管理员的问师列表中。系统会从 Skill 的 frontmatter、一级标题、核心模型和引用语中自动生成角色信息,无需修改注册代码。
问天模块包含三个相互独立的部分:观势以市场数据生成三才六爻,用于观察“势”,行情缺失或自动取象明显偏差时可显式手动校准六爻,人工结果与自动来源严格区分;观气依据干支、精确节气、五运六气及客主加临关系观察“运”,行业五行仅作传统取象归类;观心通过30秒静心、六次三枚铜钱起卦、察念和解卦完成一次不输入问题的问心仪式。卦象、干支、节气与气机关系均由本地确定性程序计算,LLM只负责解释,不参与起卦或改动结果。 问天模块包含三个相互独立的部分:观势以市场数据生成三才六爻,用于观察“势”,行情缺失或自动取象明显偏差时可显式手动校准六爻,人工结果与自动来源严格区分;观气依据干支、精确节气、五运六气及客主加临关系观察“运”,行业五行仅作传统取象归类;观心先准备1秒,再完成5轮“吸3秒、顿2秒、呼4秒”,随后以六次三枚铜钱起卦、察念和解卦完成一次不输入问题的问心仪式。卦象、干支、节气与气机关系均由本地确定性程序计算,LLM只负责解释,不参与起卦或改动结果。
问天模块使用项目本地的 `lunar-python` 计算历法,并使用 `data/iching_zh.json` 中的固定六十四卦、卦辞和爻辞。第三方授权见 `THIRD_PARTY_NOTICES.md` 问天模块使用项目本地的 `lunar-python` 计算历法,并使用 `data/iching_zh.json` 中的固定六十四卦、卦辞和爻辞。第三方授权见 `THIRD_PARTY_NOTICES.md`
@@ -27,7 +31,7 @@
## 启动 ## 启动
```powershell ```powershell
cd webapp cd webapp\app
python -m pip install -r requirements.txt python -m pip install -r requirements.txt
python server.py python server.py
``` ```
+11 -6
View File
@@ -1,12 +1,15 @@
# Governance Registries # Governance Registries
These registries describe the approved product surface during architecture migration. These registries describe the approved product surface of the modular preservation candidate.
- `pages.config.json`: primary page identity, navigation group, access expectation, scrolling, - `pages.config.json`: primary page identity, navigation group, access expectation, scrolling,
and mobile composition policy. and mobile composition policy.
- `features.config.json`: feature ownership, backend access class, data scope, and availability. - `features.config.json`: feature ownership, backend access class, data scope, and availability.
- `api.config.json`: transitional inventory of current routes, generated from `server.py` and - `api.config.json`: current routes generated from the preserved `server.py` API surface, with
assigned to a feature owner. one feature owner and backend access class per route. Its dispatcher implementation lives in
`backend/application.py`.
- `architecture-inventory.json`: generated inventory of candidate pages, routes, tables,
providers, model entry points, CSS layers, and remaining code hotspots.
- `data-fields.config.json`: canonical data products, provider eligibility, intended use, and - `data-fields.config.json`: canonical data products, provider eligibility, intended use, and
known blocked datasets. known blocked datasets.
- `data-quality.config.json`: freshness, coverage, units, adjustment, point-in-time, and - `data-quality.config.json`: freshness, coverage, units, adjustment, point-in-time, and
@@ -14,13 +17,15 @@ These registries describe the approved product surface during architecture migra
- `jobs.config.json`: background schedules, dependencies, lock keys, retry policy, timeouts, - `jobs.config.json`: background schedules, dependencies, lock keys, retry policy, timeouts,
and output versions. and output versions.
During Stage 04 these files are contract inputs, not runtime replacements. Backend access in The registries are governance contracts, not substitutes for runtime authorization. Backend
`api_access.py` remains authoritative until the HTTP governance phase switches it atomically. access in `backend/http/routes.py` is authoritative; frontend visibility is only a presentation
Frontend visibility remains a presentation concern and never grants backend access. concern and never grants access.
Regenerate the transitional API inventory after a route change: Regenerate the transitional API inventory after a route change:
```shell ```shell
python tools/build_api_registry.py python tools/build_api_registry.py
python tools/build_api_registry.py --check python tools/build_api_registry.py --check
python tools/build_architecture_inventory.py
python tools/build_architecture_inventory.py --check
``` ```
+336
View File
@@ -0,0 +1,336 @@
{
"schema_version": 1,
"captured_from": "app modular preservation candidate",
"runtime": {
"http_server": "http.server.ThreadingHTTPServer",
"application_processes": 1,
"database": "SQLite WAL",
"frontend": "build-free HTML/CSS/JavaScript",
"container_port": 8765
},
"counts": {
"primary_pages": 16,
"api_exact_paths": 53,
"api_prefixes": 0,
"api_patterns": 11,
"database_tables": 36
},
"pages": [
{
"id": "sentimentCycleView",
"title": "情绪周期"
},
{
"id": "limitPool",
"title": "涨停池"
},
{
"id": "brokenView",
"title": "炸板池"
},
{
"id": "downView",
"title": "跌停板"
},
{
"id": "yesterdayView",
"title": "昨日涨停"
},
{
"id": "performanceView",
"title": "涨停表现"
},
{
"id": "ladderView",
"title": "市场天梯"
},
{
"id": "rotationView",
"title": "板块轮动"
},
{
"id": "auctionView",
"title": "集合竞价"
},
{
"id": "themeLibraryView",
"title": "题材库"
},
{
"id": "popularityView",
"title": "人气热榜"
},
{
"id": "dragonView",
"title": "龙虎榜"
},
{
"id": "screenerView",
"title": "智能选股"
},
{
"id": "mentorView",
"title": "问师"
},
{
"id": "heavenView",
"title": "问天"
},
{
"id": "reviewWorkspaceView",
"title": "我的复盘"
}
],
"api": {
"exact": [
"/api/account/birth-profile",
"/api/account/password",
"/api/account/status",
"/api/admin/membership",
"/api/admin/refresh",
"/api/admin/settings",
"/api/admin/settings/test",
"/api/alerts",
"/api/alerts/read-all",
"/api/assistant/chat",
"/api/assistant/messages",
"/api/auction",
"/api/auth/login",
"/api/auth/logout",
"/api/auth/me",
"/api/auth/register",
"/api/backfill",
"/api/chart/intraday",
"/api/dashboard",
"/api/dragon-tiger",
"/api/dragon-tiger/profiles",
"/api/health",
"/api/heaven/hexagram",
"/api/heaven/interpret",
"/api/heaven/personal",
"/api/heaven/readings",
"/api/heaven/sector-phases",
"/api/heaven/setup",
"/api/mentors/chat",
"/api/mentors/messages",
"/api/mentors/preferences",
"/api/mentors/setup",
"/api/notes",
"/api/popularity",
"/api/realtime-aggregate/health",
"/api/reasons",
"/api/rotation/history",
"/api/rotation/members",
"/api/screener/compile",
"/api/screener/run",
"/api/screener/setup",
"/api/screener/strategies",
"/api/screener/sync",
"/api/screener/tracking",
"/api/screener/tracking/refresh",
"/api/search",
"/api/search/detail",
"/api/seat-aliases",
"/api/sentiment/history",
"/api/themes",
"/api/themes/detail",
"/api/trades",
"/api/watchlist"
],
"prefixes": [],
"patterns": [
"/api/alerts/(\\d+)",
"/api/alerts/(\\d+)/read",
"/api/heaven/readings/(\\d+)",
"/api/heaven/sector-phases/(.+)",
"/api/notes/(\\d+)",
"/api/screener/strategies/(\\d+)",
"/api/screener/tracking/(\\d+)",
"/api/stock/(\\d{6})",
"/api/stock/(\\d{6})/preview",
"/api/trades/(\\d+)",
"/api/watchlist/(\\d{6})"
]
},
"database_tables": [
"users",
"user_sessions",
"user_credentials",
"user_birth_profiles",
"system_settings",
"llm_usage",
"dashboard_snapshots",
"sync_runs",
"data_snapshots",
"watchlist",
"review_notes",
"reason_overrides",
"seat_aliases",
"sector_phase_overrides",
"stock_master",
"daily_bars",
"benchmark_bars",
"daily_indicators",
"fundamental_indicators",
"moneyflow_daily",
"auction_factors",
"earnings_events",
"popularity_factors",
"lhb_institution_daily",
"screener_strategies",
"screener_runs",
"mentor_messages",
"mentor_preferences",
"wencai_saved_queries",
"strategy_tracks",
"alerts",
"trade_entries",
"assistant_messages",
"heaven_readings",
"job_runs",
"schema_migrations"
],
"background_job_methods": [
"_background_refresh_tick"
],
"external_data_adapters": [
{
"provider": "tushare",
"path": "backend/data/providers/tushare_client.py",
"runtime_role": "primary deterministic market data"
},
{
"provider": "ifind",
"path": "backend/data/providers/ifind_client.py",
"runtime_role": "realtime, charts, snapshots, enrichment"
},
{
"provider": "eastmoney",
"path": "backend/features/market/charts.py",
"runtime_role": "display chart fallback"
},
{
"provider": "eastmoney",
"path": "backend/data/realtime.py",
"runtime_role": "isolated realtime observation"
},
{
"provider": "tencent",
"path": "backend/data/realtime.py",
"runtime_role": "index observation fallback"
}
],
"llm_entrypoints": [
{
"function": "stream_with_mentor",
"path": "backend/features/mentor/agent.py"
},
{
"function": "interpret_heaven",
"path": "backend/features/heaven/agent.py"
},
{
"function": "stream_review_assistant",
"path": "backend/features/review/agent.py"
},
{
"function": "compile_strategy_with_llm",
"path": "backend/features/screener/compiler.py"
},
{
"function": "test_llm_connection",
"path": "backend/features/screener/compiler.py"
}
],
"css_layers": [
"/shared/tokens.css?v=20260729-1",
"/styles/styles.css",
"/styles/renovation.css?v=20260725-5",
"/styles/redesign-v2.css?v=20260728-1",
"/styles/design-system.css?v=20260728-4",
"/styles/theme.css?v=20260728-2",
"/pages/heaven/page.css?v=20260728-7"
],
"code_hotspots": [
{
"path": "frontend/styles/styles.css",
"bytes": 361780,
"lines": 15465
},
{
"path": "frontend/styles/redesign-v2.css",
"bytes": 263539,
"lines": 8570
},
{
"path": "frontend/index.html",
"bytes": 135019,
"lines": 1892
},
{
"path": "backend/features/screener/engine.py",
"bytes": 108552,
"lines": 2213
},
{
"path": "backend/data/providers/tushare_client.py",
"bytes": 94329,
"lines": 2175
},
{
"path": "frontend/app.js",
"bytes": 91151,
"lines": 1939
},
{
"path": "frontend/pages/heaven/page.js",
"bytes": 88322,
"lines": 1830
},
{
"path": "frontend/styles/renovation.css",
"bytes": 83949,
"lines": 1553
},
{
"path": "frontend/pages/heaven/page.css",
"bytes": 73222,
"lines": 1084
},
{
"path": "backend/features/heaven/service.py",
"bytes": 64421,
"lines": 1303
},
{
"path": "backend/features/market/insights.py",
"bytes": 58150,
"lines": 1314
},
{
"path": "frontend/pages/market/runtime.js",
"bytes": 57053,
"lines": 1333
},
{
"path": "backend/features/heaven/engine.py",
"bytes": 51670,
"lines": 1181
},
{
"path": "backend/application.py",
"bytes": 50934,
"lines": 1165
},
{
"path": "frontend/styles/theme.css",
"bytes": 36427,
"lines": 1253
},
{
"path": "database.py",
"bytes": 34013,
"lines": 746
}
]
}
+68
View File
@@ -0,0 +1,68 @@
from __future__ import annotations
import json
import subprocess
import sys
import unittest
from pathlib import Path
from tools.build_api_registry import build as build_api_registry
from tools.build_architecture_inventory import build as build_architecture_inventory
ROOT = Path(__file__).resolve().parents[1]
class MaintenanceToolTests(unittest.TestCase):
def test_generated_candidate_registries_are_current(self) -> None:
expected_api = json.loads(
(ROOT / "config" / "api.config.json").read_text(encoding="utf-8")
)
expected_architecture = json.loads(
(ROOT / "config" / "architecture-inventory.json").read_text(
encoding="utf-8"
)
)
self.assertEqual(expected_api, build_api_registry())
self.assertEqual(expected_architecture, build_architecture_inventory())
def test_baseline_verifier_uses_the_candidate_frontend(self) -> None:
source = (ROOT / "tools" / "verify_baseline.py").read_text(encoding="utf-8")
self.assertIn('FRONTEND_ROOT = ROOT / "frontend"', source)
self.assertIn('FRONTEND_ROOT.rglob("*.js")', source)
self.assertIn("start_e2e_server()", source)
self.assertIn("stop_e2e_server(server)", source)
self.assertNotIn('"static/app.js"', source)
def test_every_tool_has_a_non_mutating_help_path(self) -> None:
for path in sorted((ROOT / "tools").glob("*.py")):
if path.name.startswith("_"):
continue
with self.subTest(tool=path.name):
result = subprocess.run(
[sys.executable, str(path), "--help"],
cwd=ROOT,
capture_output=True,
text=True,
timeout=15,
check=False,
)
self.assertEqual(result.returncode, 0, result.stderr)
def test_migration_only_tools_are_explicitly_classified(self) -> None:
readme = (ROOT / "tools" / "README.md").read_text(encoding="utf-8")
for name in (
"build_preservation_manifest.py",
"move_class_methods.py",
"split_frontend_runtime.py",
):
self.assertIn(name, readme)
manifest = (ROOT / "tools" / "build_preservation_manifest.py").read_text(
encoding="utf-8"
)
self.assertNotIn('TARGET = ROOT / "app"', manifest)
self.assertIn('required=True', manifest)
if __name__ == "__main__":
unittest.main()
+36
View File
@@ -0,0 +1,36 @@
# Candidate tools
Run these commands from `webapp/app/`. The tools are separated by responsibility so a
maintenance command cannot be mistaken for a historical migration rewrite.
## Normal maintenance
- `python tools/verify_baseline.py`: candidate unit tests, registry checks, every frontend
JavaScript syntax check, Git whitespace check, and read-only SQLite integrity check.
- `python tools/verify_baseline.py --e2e`: the same checks plus Playwright. The verifier owns
the local static-server lifecycle so the command exits cleanly on Windows.
- `python tools/build_api_registry.py [--check]`: generate or verify
`config/api.config.json` from `backend/application.py`.
- `python tools/build_architecture_inventory.py [--check]`: generate or verify
`config/architecture-inventory.json` from the candidate source tree.
## Acceptance and differential checks
- `run_preservation_runtime.py`: start an isolated original or candidate runtime with an
explicitly selected data directory and port.
- `compare_preservation_apis.py`: compare authenticated responses from two isolated runtimes.
- `compare_preservation_databases.py`: compare schema and selected table contents from two
SQLite copies.
These tools require explicit paths and do not select the production database automatically.
## Migration-only tools
- `build_preservation_manifest.py`: builds an exact-copy manifest for a specified source and
target. `--output` is mandatory so committed historical evidence is not overwritten.
- `move_class_methods.py`: mechanically moves named class methods between explicit files.
- `split_frontend_runtime.py`: reproduces the one-time Slice 10 split. It refuses to write
unless `--apply` is supplied and is not a normal maintenance command.
The migration-only tools are retained for audit and reproducibility. They are not part of
application startup, normal testing, or future feature development.
+45 -23
View File
@@ -9,7 +9,7 @@ from typing import Any
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
OUTPUT = ROOT / "docs" / "governance" / "architecture-inventory.json" OUTPUT = ROOT / "config" / "architecture-inventory.json"
def relative(path: Path) -> str: def relative(path: Path) -> str:
@@ -51,8 +51,15 @@ def api_inventory(server: str) -> dict[str, list[str]]:
return {"exact": exact, "prefixes": prefixes, "patterns": patterns} return {"exact": exact, "prefixes": prefixes, "patterns": patterns}
def database_inventory(database: str) -> list[str]: def database_inventory(sources: list[str]) -> list[str]:
return re.findall(r"CREATE TABLE IF NOT EXISTS\s+([a-zA-Z0-9_]+)", database) tables: list[str] = []
for database in sources:
tables.extend(
re.findall(
r"CREATE TABLE IF NOT EXISTS\s+([a-zA-Z0-9_]+)", database
)
)
return list(dict.fromkeys(tables))
def python_functions(path: str, prefixes: tuple[str, ...]) -> list[str]: def python_functions(path: str, prefixes: tuple[str, ...]) -> list[str]:
@@ -70,11 +77,13 @@ def css_layers(html: str) -> list[str]:
def code_hotspots() -> list[dict[str, Any]]: def code_hotspots() -> list[dict[str, Any]]:
candidates = [ candidates = [
"server.py", "backend/application.py",
"database.py", "database.py",
"screener.py", "backend/features/screener/engine.py",
"market_insights.py", "backend/features/market/insights.py",
"tushare_client.py", "backend/data/providers/tushare_client.py",
"backend/features/heaven/service.py",
"backend/features/heaven/engine.py",
"frontend/index.html", "frontend/index.html",
"frontend/app.js", "frontend/app.js",
"frontend/styles/styles.css", "frontend/styles/styles.css",
@@ -88,6 +97,8 @@ def code_hotspots() -> list[dict[str, Any]]:
rows = [] rows = []
for name in candidates: for name in candidates:
path = ROOT / name path = ROOT / name
if not path.is_file():
continue
rows.append( rows.append(
{ {
"path": name, "path": name,
@@ -100,14 +111,21 @@ def code_hotspots() -> list[dict[str, Any]]:
def build() -> dict[str, Any]: def build() -> dict[str, Any]:
html = source("frontend/index.html") html = source("frontend/index.html")
server = source("server.py") server = source("backend/application.py")
database = source("database.py") database_sources = [source("database.py")]
database_sources.extend(
path.read_text(encoding="utf-8")
for path in sorted((ROOT / "backend" / "database" / "migrations").glob("m*.py"))
)
database_sources.append(
source("backend/database/migrations/runner.py")
)
pages = page_inventory(html) pages = page_inventory(html)
api = api_inventory(server) api = api_inventory(server)
tables = database_inventory(database) tables = database_inventory(database_sources)
return { return {
"schema_version": 1, "schema_version": 1,
"captured_from": "governed source tree", "captured_from": "app modular preservation candidate",
"runtime": { "runtime": {
"http_server": "http.server.ThreadingHTTPServer", "http_server": "http.server.ThreadingHTTPServer",
"application_processes": 1, "application_processes": 1,
@@ -126,21 +144,22 @@ def build() -> dict[str, Any]:
"api": api, "api": api,
"database_tables": tables, "database_tables": tables,
"background_job_methods": python_functions( "background_job_methods": python_functions(
"server.py", ("_background", "_run_background", "_schedule_", "run_automatic") "backend/application.py",
("_background", "_run_background", "_schedule_", "run_automatic"),
), ),
"external_data_adapters": [ "external_data_adapters": [
{"provider": "tushare", "path": "tushare_client.py", "runtime_role": "primary deterministic market data"}, {"provider": "tushare", "path": "backend/data/providers/tushare_client.py", "runtime_role": "primary deterministic market data"},
{"provider": "ifind", "path": "ifind_client.py", "runtime_role": "realtime, charts, snapshots, enrichment"}, {"provider": "ifind", "path": "backend/data/providers/ifind_client.py", "runtime_role": "realtime, charts, snapshots, enrichment"},
{"provider": "eastmoney", "path": "chart_data_provider.py", "runtime_role": "display chart fallback"}, {"provider": "eastmoney", "path": "backend/features/market/charts.py", "runtime_role": "display chart fallback"},
{"provider": "eastmoney", "path": "realtime_aggregator.py", "runtime_role": "isolated realtime observation"}, {"provider": "eastmoney", "path": "backend/data/realtime.py", "runtime_role": "isolated realtime observation"},
{"provider": "tencent", "path": "realtime_aggregator.py", "runtime_role": "index observation fallback"}, {"provider": "tencent", "path": "backend/data/realtime.py", "runtime_role": "index observation fallback"},
], ],
"llm_entrypoints": [ "llm_entrypoints": [
{"function": "stream_with_mentor", "path": "mentor_agent.py"}, {"function": "stream_with_mentor", "path": "backend/features/mentor/agent.py"},
{"function": "interpret_heaven", "path": "heaven_agent.py"}, {"function": "interpret_heaven", "path": "backend/features/heaven/agent.py"},
{"function": "stream_review_assistant", "path": "assistant_agent.py"}, {"function": "stream_review_assistant", "path": "backend/features/review/agent.py"},
{"function": "compile_strategy_with_llm", "path": "llm_strategy.py"}, {"function": "compile_strategy_with_llm", "path": "backend/features/screener/compiler.py"},
{"function": "test_llm_connection", "path": "llm_strategy.py"}, {"function": "test_llm_connection", "path": "backend/features/screener/compiler.py"},
], ],
"css_layers": css_layers(html), "css_layers": css_layers(html),
"code_hotspots": code_hotspots(), "code_hotspots": code_hotspots(),
@@ -154,7 +173,10 @@ def main() -> int:
rendered = json.dumps(build(), ensure_ascii=False, indent=2) + "\n" rendered = json.dumps(build(), ensure_ascii=False, indent=2) + "\n"
if args.check: if args.check:
if not OUTPUT.exists() or OUTPUT.read_text(encoding="utf-8") != rendered: if not OUTPUT.exists() or OUTPUT.read_text(encoding="utf-8") != rendered:
raise SystemExit("architecture inventory is stale; run tools/build_architecture_inventory.py") raise SystemExit(
"architecture inventory is stale; run "
"tools/build_architecture_inventory.py"
)
print("Architecture inventory is current.") print("Architecture inventory is current.")
return 0 return 0
OUTPUT.parent.mkdir(parents=True, exist_ok=True) OUTPUT.parent.mkdir(parents=True, exist_ok=True)
+64 -21
View File
@@ -1,14 +1,13 @@
from __future__ import annotations from __future__ import annotations
import argparse
import hashlib import hashlib
import json import json
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[1] REPOSITORY_ROOT = Path(__file__).resolve().parents[2]
TARGET = ROOT / "app"
OUTPUT = ROOT / "docs" / "migration" / "原版资产清单.json"
DIRECTORIES = ( DIRECTORIES = (
"backend", "backend",
@@ -69,33 +68,57 @@ def digest(path: Path) -> str:
return checksum.hexdigest() return checksum.hexdigest()
def source_files() -> list[Path]: def display_path(path: Path, base: Path) -> str:
files = [ROOT / name for name in ROOT_FILES] try:
return path.relative_to(base).as_posix() or "."
except ValueError:
return path.as_posix()
def source_files(source_root: Path) -> list[Path]:
files = [source_root / name for name in ROOT_FILES]
for directory in DIRECTORIES: for directory in DIRECTORIES:
root = source_root / directory
if not root.is_dir():
continue
files.extend( files.extend(
path path
for path in (ROOT / directory).rglob("*") for path in root.rglob("*")
if path.is_file() and "__pycache__" not in path.parts if path.is_file() and "__pycache__" not in path.parts
) )
return sorted(set(files), key=lambda path: path.relative_to(ROOT).as_posix()) return sorted(
set(files), key=lambda path: path.relative_to(source_root).as_posix()
)
def build_manifest() -> dict[str, object]: def build_manifest(
source_root: Path,
target_root: Path,
source_commit: str,
) -> dict[str, object]:
assets = [] assets = []
mismatches = [] mismatches = []
for source in source_files(): for source in source_files(source_root):
relative = source.relative_to(ROOT) relative = source.relative_to(source_root)
target = TARGET / relative target = target_root / relative
source_hash = digest(source) source_exists = source.is_file()
source_hash = digest(source) if source_exists else ""
target_hash = digest(target) if target.is_file() else "" target_hash = digest(target) if target.is_file() else ""
status = "identical" if source_hash == target_hash else "mismatch" if not source_exists:
status = "missing_source"
elif not target.is_file():
status = "missing_target"
elif source_hash == target_hash:
status = "identical"
else:
status = "mismatch"
if status != "identical": if status != "identical":
mismatches.append(relative.as_posix()) mismatches.append(relative.as_posix())
assets.append( assets.append(
{ {
"source": relative.as_posix(), "source": relative.as_posix(),
"target": f"app/{relative.as_posix()}", "target": display_path(target, REPOSITORY_ROOT),
"bytes": source.stat().st_size, "bytes": source.stat().st_size if source_exists else 0,
"sha256": source_hash, "sha256": source_hash,
"disposition": "original_copy_pending_move", "disposition": "original_copy_pending_move",
"status": status, "status": status,
@@ -104,9 +127,9 @@ def build_manifest() -> dict[str, object]:
return { return {
"schema_version": 1, "schema_version": 1,
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"), "generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"source_commit": "41329943c4878fc09ed82ec376eb93ab151e4092", "source_commit": source_commit,
"source_root": ".", "source_root": display_path(source_root, REPOSITORY_ROOT),
"target_root": "app", "target_root": display_path(target_root, REPOSITORY_ROOT),
"excluded": [ "excluded": [
"next/", "next/",
"data/review.db and SQLite sidecars", "data/review.db and SQLite sidecars",
@@ -123,14 +146,34 @@ def build_manifest() -> dict[str, object]:
def main() -> int: def main() -> int:
manifest = build_manifest() parser = argparse.ArgumentParser(
OUTPUT.write_text( description=(
"Build an exact-copy manifest for a preservation migration stage. "
"This is a migration-only tool; it does not describe the final moved layout."
)
)
parser.add_argument("--source-root", type=Path, default=REPOSITORY_ROOT)
parser.add_argument("--target-root", type=Path, default=REPOSITORY_ROOT / "app")
parser.add_argument("--source-commit", default="")
parser.add_argument(
"--output",
type=Path,
required=True,
help="write to a new audit path; do not overwrite committed slice evidence",
)
args = parser.parse_args()
source_root = args.source_root.resolve()
target_root = args.target_root.resolve()
output = args.output.resolve()
manifest = build_manifest(source_root, target_root, args.source_commit)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8", encoding="utf-8",
) )
print( print(
f"assets={manifest['asset_count']} mismatches={manifest['mismatch_count']} " f"assets={manifest['asset_count']} mismatches={manifest['mismatch_count']} "
f"output={OUTPUT}" f"output={output}"
) )
return 1 if manifest["mismatch_count"] else 0 return 1 if manifest["mismatch_count"] else 0
+15
View File
@@ -1,5 +1,6 @@
from __future__ import annotations from __future__ import annotations
import argparse
import hashlib import hashlib
import json import json
import re import re
@@ -91,6 +92,20 @@ def extract_written_chunks(paths: list[Path]) -> dict[tuple[int, int], str]:
def main() -> None: def main() -> None:
parser = argparse.ArgumentParser(
description="Reproduce the one-time Slice 10 frontend source split"
)
parser.add_argument(
"--apply",
action="store_true",
help="perform the historical split; this rewrites frontend runtime files",
)
args = parser.parse_args()
if not args.apply:
parser.error(
"this migration-only tool rewrites files; pass --apply only when "
"reproducing Slice 10 from its documented pre-split checkpoint"
)
source_path = ORIGINAL_STATIC / "app.js" source_path = ORIGINAL_STATIC / "app.js"
target_path = FRONTEND_ROOT / "app.js" target_path = FRONTEND_ROOT / "app.js"
source_bytes = source_path.read_bytes() source_bytes = source_path.read_bytes()
+85 -7
View File
@@ -5,10 +5,15 @@ import shutil
import sqlite3 import sqlite3
import subprocess import subprocess
import sys import sys
import time
import urllib.error
import urllib.request
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
FRONTEND_ROOT = ROOT / "frontend"
E2E_URL = "http://127.0.0.1:8876/index.html"
def run(label: str, command: list[str]) -> None: def run(label: str, command: list[str]) -> None:
@@ -28,8 +33,63 @@ def verify_database() -> None:
print(f"\n[database] integrity_check=ok size={database.stat().st_size}") print(f"\n[database] integrity_check=ok size={database.stat().st_size}")
def url_reachable(url: str) -> bool:
try:
with urllib.request.urlopen(url, timeout=1) as response:
return response.status == 200
except (OSError, urllib.error.URLError):
return False
def start_e2e_server() -> subprocess.Popen[bytes] | None:
if url_reachable(E2E_URL):
print(f"\n[e2e-server] reusing {E2E_URL}")
return None
creation_flags = subprocess.CREATE_NO_WINDOW if sys.platform == "win32" else 0
process = subprocess.Popen(
[
sys.executable,
"-m",
"http.server",
"8876",
"--bind",
"127.0.0.1",
"--directory",
"frontend",
],
cwd=ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
creationflags=creation_flags,
)
deadline = time.monotonic() + 15
while time.monotonic() < deadline:
if process.poll() is not None:
raise RuntimeError("Playwright static server exited before becoming ready")
if url_reachable(E2E_URL):
print(f"\n[e2e-server] started {E2E_URL}")
return process
time.sleep(0.1)
stop_e2e_server(process)
raise RuntimeError("Playwright static server did not become ready within 15 seconds")
def stop_e2e_server(process: subprocess.Popen[bytes] | None) -> None:
if process is None:
return
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=5)
print("\n[e2e-server] stopped")
def main() -> int: def main() -> int:
parser = argparse.ArgumentParser(description="Verify the governance regression baseline") parser = argparse.ArgumentParser(
description="Verify the modular preservation candidate"
)
parser.add_argument( parser.add_argument(
"--e2e", "--e2e",
action="store_true", action="store_true",
@@ -38,19 +98,37 @@ def main() -> int:
args = parser.parse_args() args = parser.parse_args()
run("python", [sys.executable, "-m", "unittest", "discover", "-s", "tests"]) run("python", [sys.executable, "-m", "unittest", "discover", "-s", "tests"])
run(
"api-registry",
[sys.executable, "tools/build_api_registry.py", "--check"],
)
run(
"architecture-inventory",
[sys.executable, "tools/build_architecture_inventory.py", "--check"],
)
node = shutil.which("node") node = shutil.which("node")
if not node: if not node:
raise RuntimeError("node is required for JavaScript syntax checks") raise RuntimeError("node is required for JavaScript syntax checks")
for script in ("static/app.js", "static/heaven-loading-v2.js"): scripts = sorted(FRONTEND_ROOT.rglob("*.js"))
run("javascript", [node, "--check", script]) if not scripts:
raise RuntimeError(f"no JavaScript files found under {FRONTEND_ROOT}")
for script in scripts:
run(
"javascript",
[node, "--check", script.relative_to(ROOT).as_posix()],
)
run("patch", ["git", "diff", "--check"]) run("patch", ["git", "diff", "--check"])
verify_database() verify_database()
if args.e2e: if args.e2e:
npm = shutil.which("npm.cmd" if sys.platform == "win32" else "npm") npx = shutil.which("npx.cmd" if sys.platform == "win32" else "npx")
if not npm: if not npx:
raise RuntimeError("npm is required for the Playwright suite") raise RuntimeError("npx is required for the Playwright suite")
run("playwright", [npm, "run", "test:e2e"]) server = start_e2e_server()
try:
run("playwright", [npx, "playwright", "test", "--reporter=dot"])
finally:
stop_e2e_server(server)
print("\nBaseline verification passed.") print("\nBaseline verification passed.")
return 0 return 0