diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ff5170d..789a695 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,40 +1,70 @@ -# Architecture +# Candidate architecture -The normative governance contract is documented in -`docs/governance/architecture-standard.md`. This file describes the currently deployed -shape; the standard defines the target boundaries and the rules applied during migration. +`app/` is the behavior-preserving modular candidate. The original `webapp/` runtime remains +the product and visual baseline until manual acceptance. `next/` is a rejected, frozen +implementation and is not a source for this directory. -The application intentionally keeps a small deployment footprint: one Python process, one -SQLite database, and a build-free browser client. The internal boundaries are nevertheless -explicit so new features do not bypass account isolation or data-quality rules. +The application deliberately remains a modular monolith: one Python process, one SQLite WAL +database, and a build-free HTML/CSS/JavaScript client. The migration changed source ownership +and imports, not the technology stack or observable product behavior. -## Backend boundaries +## Runtime path -- `server.py`: application services and HTTP request/response wiring. -- `api_access.py`: the single authorization policy for authenticated, member, and admin APIs. -- `app_config.py`: runtime paths, local environment loading, and shared input validation. -- `database.py`: SQLite schema, migrations, and persistence operations. -- `tushare_client.py` and `realtime_aggregator.py`: external market-data adapters. -- `sentiment_engine.py`, `screener.py`, and `heaven_engine.py`: deterministic domain logic. -- `mentor_agent.py`, `heaven_agent.py`, and `llm_strategy.py`: bounded LLM adapters. +```text +browser + -> frontend/shared/api.js + -> backend HTTP transport and feature HTTP mixins + -> feature services + -> repositories / DataGateway / LLMGateway + -> SQLite / market providers / model providers -## Data ownership +background scheduler + -> backend/jobs + -> the same feature services and repositories +``` -Public market snapshots, stock factors, built-in strategies, limit-up reasons, seat aliases, -and sector-element mappings are shared. Only administrators can modify shared knowledge. +## Source ownership -Watchlists, review notes, custom strategies, screener runs, mentor conversations, birth data, -alerts, trading journals, and assistant conversations are owned by a user ID and must be -queried with that ID. LLM features additionally require active membership. +- `server.py` is the stable command/import facade. Runtime composition lives in + `backend/application.py` and `backend/bootstrap/`. +- `backend/http/` owns common authentication, request IDs, responses, static delivery, and + error normalization. Feature-specific transport handlers live beside their feature. +- `backend/features//` owns the mechanically moved service, repository, HTTP, agent, + or deterministic calculation code for that product area. +- `backend/data/` owns provider construction, source policy, provenance, units, freshness, + coverage, and display-versus-calculation eligibility. +- `backend/database/` owns connection management, ordered migrations, and narrow repository + adapters. Root `database.py` remains the legacy schema/composition anchor and combines the + feature repository mixins; do not add feature queries to it. +- `backend/jobs/` owns job definitions, locks, retries, idempotency, and persisted run state. +- `backend/llm/` owns model selection, membership/quota checks, fallback, streaming rules, + and call audit. Feature agents only prepare context and provider payloads. +- `frontend/shared/` is the only browser API/state/Shell/component boundary. +- `frontend/pages/` owns page-local behavior. The original runtime was split mechanically; + source markers and preservation tests prove that the pieces reassemble to the audited + original, apart from explicitly registered trial retirements. +- `frontend/styles/`, `frontend/shared/tokens.css`, and the Wentian page stylesheet preserve + the approved cascade and light/dark/mobile behavior. +- `config/` is the versioned registry for pages, features, APIs, datasets, quality rules, + jobs, and the generated candidate architecture inventory. -## Data integrity +Root modules such as `screener.py`, `tushare_client.py`, and `mentor_agent.py` are compatibility +aliases to canonical modules. They contain no second implementation and remain only because +the original public import surface is part of the preservation contract. -Production reads never synthesize market prices. A failed live request may use the latest real -snapshot at or before the requested date. When no real snapshot exists, the API reports that -the data is unavailable. Demo builders remain test fixtures only. +## Non-negotiable maintenance rules -## Change contract +1. Preserve account ownership in every user-private query and test it with two accounts. +2. Browser requests go through `frontend/shared/api.js`; provider calls go through the data + boundary; model calls go through `backend/llm/`. +3. Calculation datasets fail closed when required source, date, unit, freshness, or coverage + evidence is missing. Display fallbacks do not silently enter calculations. +4. Do not implement logic in both a root compatibility module and a canonical module. +5. Do not remove compatibility or uncertain code without reference scanning, old/new + differential evidence, browser checks, and manual acceptance. +6. Run `python tools/verify_baseline.py` for every change and add `--e2e` when runtime or + frontend behavior can be affected. -New endpoints must be added to `api_access.required_role` when they need member or admin -access. New user-owned tables must include `user_id`, an ownership index, and cross-account -tests. API payload compatibility is protected by the Python and Playwright suites. +The authoritative migration constraints and handoff procedure are in +`../docs/migration/原版保真迁移总纲.md` and +`../docs/migration/人工维护与本地切换指南.md`. diff --git a/README.md b/README.md index eda2e01..82bcdf9 100644 --- a/README.md +++ b/README.md @@ -2,23 +2,27 @@ 一个面向 A 股盘后复盘的本地 Web 工作台。后端使用 Python 访问 Tushare Pro,前端不依赖构建工具。 +本目录是从原版源码逐项移动、机械拆分并完成差分验证的模块化候选,不是依据规格书重新开发的 +第二套产品。人工验收和正式切换前,`webapp/`根目录仍是唯一正式基线;冻结的`next/`不得用于 +部署或后续开发。目录职责见[ARCHITECTURE.md](ARCHITECTURE.md)。 + 当前包含集合竞价、涨停池、炸板池、跌停板、昨日涨停、涨停表现、市场天梯、板块轮动、题材库、人气热榜、龙虎榜和个人复盘工作区。交易日快照与同步记录保存在本地 SQLite 数据库 `data/review.db`。 集合竞价中心采用盘前生命周期:9:15 前显示预告,9:15–9:25 明确等待最终竞价,9:25–9:30 自动读取并重试最终竞价筛选,9:30 后停止更新并冻结为复盘归档。当前 Tushare 只提供 9:25 最终竞价快照,不将其表述为动态虚拟撮合行情。 第三阶段加入了机构席位、席位别名、个股复权日 K、资金流、自选股、涨停原因修订、个股笔记、每日复盘和历史数据回补。 -股票代码在桌面端悬停后会显示分时与日 K 快速预览,默认优先展示日 K;移动端点击代码后从底部打开预览面板。股票详情以及板块、题材、指数详情均可在日 K 与最新分时之间切换。日 K 复用个股详情缓存;分时使用隔离的东方财富分钟图表源和短时内存缓存,只负责展示,不写入主行情、不参与情绪、选股或问天计算。图表源不可用时界面会明确显示“分时不可用”,不会使用日 K 数据模拟分时走势。 +股票代码在桌面端悬停后会显示分时与日 K 快速预览,默认优先展示日 K;移动端点击代码后从底部打开预览面板。股票详情以及板块、题材、指数详情均可在日 K 与最新分时之间切换。日 K 复用个股详情缓存;分时优先使用 iFinD,东方财富仅作隔离的展示兜底,并使用短时内存缓存。图表数据不写入主行情、不参与情绪、选股或问天计算;不可用时明确显示“分时不可用”,不会用日 K 模拟分时走势。 -智能选股模块包含 45 日全市场因子库、六阶段市场识别、七套内置策略、受控公式 DSL、自然语言策略编译、候选排名和滚动回测。竞价涨幅、竞价成交额、竞价换手率与竞价量比随因子数据一并同步,可用于自定义公式和历史回测。首次使用需在页面点击“同步因子数据”。未配置 LLM 时使用本地策略模板;配置兼容 API 后自动切换为主模型编译,主模型失败时自动使用辅助模型,两者均支持独立连通性测试。 +智能选股包含六阶段盘后候选、29 套精选策略、自定义公式 DSL、自然语言公式编译、候选排名和滚动回测。阶段与精选策略在当日行情更新后由后台确定性计算;自定义选股由用户手动执行,LLM 只负责编译自然语言条件,不参与候选筛选。竞价、估值、财务、资金、人气和席位等字段按已登记的数据可用性进入因子库,缺失时明确显示覆盖问题。 -每次选股结果会自动进入五交易日持续跟踪,展示 T+1 开盘/收盘、T+3、T+5、最大涨幅与最大回撤。提醒中心支持手工日期提醒,并在策略首日反馈和五日跟踪完成时生成账号私有的站内提醒。 +候选只有经用户手动加入后才进入五交易日持续跟踪,展示 T+1 开盘/收盘、T+3、T+5、最大涨幅与最大回撤。提醒中心支持手工日期提醒,并在策略首日反馈和五日跟踪完成时生成账号私有的站内提醒。 问师模块会读取当前复盘、近十日市场情绪、涨跌停、昨日反馈、板块轮动、市场阶段、龙虎榜和指定个股数据,再按选中的游资思维 Skill 进行单师对话。对话记录按账号、老师和交易日期保存在服务端;主模型不可用时自动切换辅助模型。 新增公开问师角色时,在 `游资skills` 下增加一个包含 `SKILL.md` 的独立目录,并在 `游资skills/mentor_catalog.json` 中登记素材等级与结构质检。管理员私有角色放在 `data/private-mentor-skills`,该目录不进入 Git 或 Docker 镜像,且只会出现在管理员的问师列表中。系统会从 Skill 的 frontmatter、一级标题、核心模型和引用语中自动生成角色信息,无需修改注册代码。 -问天模块包含三个相互独立的部分:观势以市场数据生成三才六爻,用于观察“势”,行情缺失或自动取象明显偏差时可显式手动校准六爻,人工结果与自动来源严格区分;观气依据干支、精确节气、五运六气及客主加临关系观察“运”,行业五行仅作传统取象归类;观心通过30秒静心、六次三枚铜钱起卦、察念和解卦完成一次不输入问题的问心仪式。卦象、干支、节气与气机关系均由本地确定性程序计算,LLM只负责解释,不参与起卦或改动结果。 +问天模块包含三个相互独立的部分:观势以市场数据生成三才六爻,用于观察“势”,行情缺失或自动取象明显偏差时可显式手动校准六爻,人工结果与自动来源严格区分;观气依据干支、精确节气、五运六气及客主加临关系观察“运”,行业五行仅作传统取象归类;观心先准备1秒,再完成5轮“吸3秒、顿2秒、呼4秒”,随后以六次三枚铜钱起卦、察念和解卦完成一次不输入问题的问心仪式。卦象、干支、节气与气机关系均由本地确定性程序计算,LLM只负责解释,不参与起卦或改动结果。 问天模块使用项目本地的 `lunar-python` 计算历法,并使用 `data/iching_zh.json` 中的固定六十四卦、卦辞和爻辞。第三方授权见 `THIRD_PARTY_NOTICES.md`。 @@ -27,7 +31,7 @@ ## 启动 ```powershell -cd webapp +cd webapp\app python -m pip install -r requirements.txt python server.py ``` diff --git a/config/README.md b/config/README.md index cc512e0..f27497a 100644 --- a/config/README.md +++ b/config/README.md @@ -1,12 +1,15 @@ # Governance Registries -These registries describe the approved product surface during architecture migration. +These registries describe the approved product surface of the modular preservation candidate. - `pages.config.json`: primary page identity, navigation group, access expectation, scrolling, and mobile composition policy. - `features.config.json`: feature ownership, backend access class, data scope, and availability. -- `api.config.json`: transitional inventory of current routes, generated from `server.py` and - assigned to a feature owner. +- `api.config.json`: current routes generated from the preserved `server.py` API surface, with + one feature owner and backend access class per route. Its dispatcher implementation lives in + `backend/application.py`. +- `architecture-inventory.json`: generated inventory of candidate pages, routes, tables, + providers, model entry points, CSS layers, and remaining code hotspots. - `data-fields.config.json`: canonical data products, provider eligibility, intended use, and known blocked datasets. - `data-quality.config.json`: freshness, coverage, units, adjustment, point-in-time, and @@ -14,13 +17,15 @@ These registries describe the approved product surface during architecture migra - `jobs.config.json`: background schedules, dependencies, lock keys, retry policy, timeouts, and output versions. -During Stage 04 these files are contract inputs, not runtime replacements. Backend access in -`api_access.py` remains authoritative until the HTTP governance phase switches it atomically. -Frontend visibility remains a presentation concern and never grants backend access. +The registries are governance contracts, not substitutes for runtime authorization. Backend +access in `backend/http/routes.py` is authoritative; frontend visibility is only a presentation +concern and never grants access. Regenerate the transitional API inventory after a route change: ```shell python tools/build_api_registry.py python tools/build_api_registry.py --check +python tools/build_architecture_inventory.py +python tools/build_architecture_inventory.py --check ``` diff --git a/config/architecture-inventory.json b/config/architecture-inventory.json new file mode 100644 index 0000000..6577d3d --- /dev/null +++ b/config/architecture-inventory.json @@ -0,0 +1,336 @@ +{ + "schema_version": 1, + "captured_from": "app modular preservation candidate", + "runtime": { + "http_server": "http.server.ThreadingHTTPServer", + "application_processes": 1, + "database": "SQLite WAL", + "frontend": "build-free HTML/CSS/JavaScript", + "container_port": 8765 + }, + "counts": { + "primary_pages": 16, + "api_exact_paths": 53, + "api_prefixes": 0, + "api_patterns": 11, + "database_tables": 36 + }, + "pages": [ + { + "id": "sentimentCycleView", + "title": "情绪周期" + }, + { + "id": "limitPool", + "title": "涨停池" + }, + { + "id": "brokenView", + "title": "炸板池" + }, + { + "id": "downView", + "title": "跌停板" + }, + { + "id": "yesterdayView", + "title": "昨日涨停" + }, + { + "id": "performanceView", + "title": "涨停表现" + }, + { + "id": "ladderView", + "title": "市场天梯" + }, + { + "id": "rotationView", + "title": "板块轮动" + }, + { + "id": "auctionView", + "title": "集合竞价" + }, + { + "id": "themeLibraryView", + "title": "题材库" + }, + { + "id": "popularityView", + "title": "人气热榜" + }, + { + "id": "dragonView", + "title": "龙虎榜" + }, + { + "id": "screenerView", + "title": "智能选股" + }, + { + "id": "mentorView", + "title": "问师" + }, + { + "id": "heavenView", + "title": "问天" + }, + { + "id": "reviewWorkspaceView", + "title": "我的复盘" + } + ], + "api": { + "exact": [ + "/api/account/birth-profile", + "/api/account/password", + "/api/account/status", + "/api/admin/membership", + "/api/admin/refresh", + "/api/admin/settings", + "/api/admin/settings/test", + "/api/alerts", + "/api/alerts/read-all", + "/api/assistant/chat", + "/api/assistant/messages", + "/api/auction", + "/api/auth/login", + "/api/auth/logout", + "/api/auth/me", + "/api/auth/register", + "/api/backfill", + "/api/chart/intraday", + "/api/dashboard", + "/api/dragon-tiger", + "/api/dragon-tiger/profiles", + "/api/health", + "/api/heaven/hexagram", + "/api/heaven/interpret", + "/api/heaven/personal", + "/api/heaven/readings", + "/api/heaven/sector-phases", + "/api/heaven/setup", + "/api/mentors/chat", + "/api/mentors/messages", + "/api/mentors/preferences", + "/api/mentors/setup", + "/api/notes", + "/api/popularity", + "/api/realtime-aggregate/health", + "/api/reasons", + "/api/rotation/history", + "/api/rotation/members", + "/api/screener/compile", + "/api/screener/run", + "/api/screener/setup", + "/api/screener/strategies", + "/api/screener/sync", + "/api/screener/tracking", + "/api/screener/tracking/refresh", + "/api/search", + "/api/search/detail", + "/api/seat-aliases", + "/api/sentiment/history", + "/api/themes", + "/api/themes/detail", + "/api/trades", + "/api/watchlist" + ], + "prefixes": [], + "patterns": [ + "/api/alerts/(\\d+)", + "/api/alerts/(\\d+)/read", + "/api/heaven/readings/(\\d+)", + "/api/heaven/sector-phases/(.+)", + "/api/notes/(\\d+)", + "/api/screener/strategies/(\\d+)", + "/api/screener/tracking/(\\d+)", + "/api/stock/(\\d{6})", + "/api/stock/(\\d{6})/preview", + "/api/trades/(\\d+)", + "/api/watchlist/(\\d{6})" + ] + }, + "database_tables": [ + "users", + "user_sessions", + "user_credentials", + "user_birth_profiles", + "system_settings", + "llm_usage", + "dashboard_snapshots", + "sync_runs", + "data_snapshots", + "watchlist", + "review_notes", + "reason_overrides", + "seat_aliases", + "sector_phase_overrides", + "stock_master", + "daily_bars", + "benchmark_bars", + "daily_indicators", + "fundamental_indicators", + "moneyflow_daily", + "auction_factors", + "earnings_events", + "popularity_factors", + "lhb_institution_daily", + "screener_strategies", + "screener_runs", + "mentor_messages", + "mentor_preferences", + "wencai_saved_queries", + "strategy_tracks", + "alerts", + "trade_entries", + "assistant_messages", + "heaven_readings", + "job_runs", + "schema_migrations" + ], + "background_job_methods": [ + "_background_refresh_tick" + ], + "external_data_adapters": [ + { + "provider": "tushare", + "path": "backend/data/providers/tushare_client.py", + "runtime_role": "primary deterministic market data" + }, + { + "provider": "ifind", + "path": "backend/data/providers/ifind_client.py", + "runtime_role": "realtime, charts, snapshots, enrichment" + }, + { + "provider": "eastmoney", + "path": "backend/features/market/charts.py", + "runtime_role": "display chart fallback" + }, + { + "provider": "eastmoney", + "path": "backend/data/realtime.py", + "runtime_role": "isolated realtime observation" + }, + { + "provider": "tencent", + "path": "backend/data/realtime.py", + "runtime_role": "index observation fallback" + } + ], + "llm_entrypoints": [ + { + "function": "stream_with_mentor", + "path": "backend/features/mentor/agent.py" + }, + { + "function": "interpret_heaven", + "path": "backend/features/heaven/agent.py" + }, + { + "function": "stream_review_assistant", + "path": "backend/features/review/agent.py" + }, + { + "function": "compile_strategy_with_llm", + "path": "backend/features/screener/compiler.py" + }, + { + "function": "test_llm_connection", + "path": "backend/features/screener/compiler.py" + } + ], + "css_layers": [ + "/shared/tokens.css?v=20260729-1", + "/styles/styles.css", + "/styles/renovation.css?v=20260725-5", + "/styles/redesign-v2.css?v=20260728-1", + "/styles/design-system.css?v=20260728-4", + "/styles/theme.css?v=20260728-2", + "/pages/heaven/page.css?v=20260728-7" + ], + "code_hotspots": [ + { + "path": "frontend/styles/styles.css", + "bytes": 361780, + "lines": 15465 + }, + { + "path": "frontend/styles/redesign-v2.css", + "bytes": 263539, + "lines": 8570 + }, + { + "path": "frontend/index.html", + "bytes": 135019, + "lines": 1892 + }, + { + "path": "backend/features/screener/engine.py", + "bytes": 108552, + "lines": 2213 + }, + { + "path": "backend/data/providers/tushare_client.py", + "bytes": 94329, + "lines": 2175 + }, + { + "path": "frontend/app.js", + "bytes": 91151, + "lines": 1939 + }, + { + "path": "frontend/pages/heaven/page.js", + "bytes": 88322, + "lines": 1830 + }, + { + "path": "frontend/styles/renovation.css", + "bytes": 83949, + "lines": 1553 + }, + { + "path": "frontend/pages/heaven/page.css", + "bytes": 73222, + "lines": 1084 + }, + { + "path": "backend/features/heaven/service.py", + "bytes": 64421, + "lines": 1303 + }, + { + "path": "backend/features/market/insights.py", + "bytes": 58150, + "lines": 1314 + }, + { + "path": "frontend/pages/market/runtime.js", + "bytes": 57053, + "lines": 1333 + }, + { + "path": "backend/features/heaven/engine.py", + "bytes": 51670, + "lines": 1181 + }, + { + "path": "backend/application.py", + "bytes": 50934, + "lines": 1165 + }, + { + "path": "frontend/styles/theme.css", + "bytes": 36427, + "lines": 1253 + }, + { + "path": "database.py", + "bytes": 34013, + "lines": 746 + } + ] +} diff --git a/tests/test_maintenance_tools.py b/tests/test_maintenance_tools.py new file mode 100644 index 0000000..169ced0 --- /dev/null +++ b/tests/test_maintenance_tools.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import json +import subprocess +import sys +import unittest +from pathlib import Path + +from tools.build_api_registry import build as build_api_registry +from tools.build_architecture_inventory import build as build_architecture_inventory + + +ROOT = Path(__file__).resolve().parents[1] + + +class MaintenanceToolTests(unittest.TestCase): + def test_generated_candidate_registries_are_current(self) -> None: + expected_api = json.loads( + (ROOT / "config" / "api.config.json").read_text(encoding="utf-8") + ) + expected_architecture = json.loads( + (ROOT / "config" / "architecture-inventory.json").read_text( + encoding="utf-8" + ) + ) + self.assertEqual(expected_api, build_api_registry()) + self.assertEqual(expected_architecture, build_architecture_inventory()) + + def test_baseline_verifier_uses_the_candidate_frontend(self) -> None: + source = (ROOT / "tools" / "verify_baseline.py").read_text(encoding="utf-8") + self.assertIn('FRONTEND_ROOT = ROOT / "frontend"', source) + self.assertIn('FRONTEND_ROOT.rglob("*.js")', source) + self.assertIn("start_e2e_server()", source) + self.assertIn("stop_e2e_server(server)", source) + self.assertNotIn('"static/app.js"', source) + + def test_every_tool_has_a_non_mutating_help_path(self) -> None: + for path in sorted((ROOT / "tools").glob("*.py")): + if path.name.startswith("_"): + continue + with self.subTest(tool=path.name): + result = subprocess.run( + [sys.executable, str(path), "--help"], + cwd=ROOT, + capture_output=True, + text=True, + timeout=15, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_migration_only_tools_are_explicitly_classified(self) -> None: + readme = (ROOT / "tools" / "README.md").read_text(encoding="utf-8") + for name in ( + "build_preservation_manifest.py", + "move_class_methods.py", + "split_frontend_runtime.py", + ): + self.assertIn(name, readme) + manifest = (ROOT / "tools" / "build_preservation_manifest.py").read_text( + encoding="utf-8" + ) + self.assertNotIn('TARGET = ROOT / "app"', manifest) + self.assertIn('required=True', manifest) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..6f4462e --- /dev/null +++ b/tools/README.md @@ -0,0 +1,36 @@ +# Candidate tools + +Run these commands from `webapp/app/`. The tools are separated by responsibility so a +maintenance command cannot be mistaken for a historical migration rewrite. + +## Normal maintenance + +- `python tools/verify_baseline.py`: candidate unit tests, registry checks, every frontend + JavaScript syntax check, Git whitespace check, and read-only SQLite integrity check. +- `python tools/verify_baseline.py --e2e`: the same checks plus Playwright. The verifier owns + the local static-server lifecycle so the command exits cleanly on Windows. +- `python tools/build_api_registry.py [--check]`: generate or verify + `config/api.config.json` from `backend/application.py`. +- `python tools/build_architecture_inventory.py [--check]`: generate or verify + `config/architecture-inventory.json` from the candidate source tree. + +## Acceptance and differential checks + +- `run_preservation_runtime.py`: start an isolated original or candidate runtime with an + explicitly selected data directory and port. +- `compare_preservation_apis.py`: compare authenticated responses from two isolated runtimes. +- `compare_preservation_databases.py`: compare schema and selected table contents from two + SQLite copies. + +These tools require explicit paths and do not select the production database automatically. + +## Migration-only tools + +- `build_preservation_manifest.py`: builds an exact-copy manifest for a specified source and + target. `--output` is mandatory so committed historical evidence is not overwritten. +- `move_class_methods.py`: mechanically moves named class methods between explicit files. +- `split_frontend_runtime.py`: reproduces the one-time Slice 10 split. It refuses to write + unless `--apply` is supplied and is not a normal maintenance command. + +The migration-only tools are retained for audit and reproducibility. They are not part of +application startup, normal testing, or future feature development. diff --git a/tools/build_architecture_inventory.py b/tools/build_architecture_inventory.py index 3ead445..bccec7f 100644 --- a/tools/build_architecture_inventory.py +++ b/tools/build_architecture_inventory.py @@ -9,7 +9,7 @@ from typing import Any ROOT = Path(__file__).resolve().parents[1] -OUTPUT = ROOT / "docs" / "governance" / "architecture-inventory.json" +OUTPUT = ROOT / "config" / "architecture-inventory.json" def relative(path: Path) -> str: @@ -51,8 +51,15 @@ def api_inventory(server: str) -> dict[str, list[str]]: return {"exact": exact, "prefixes": prefixes, "patterns": patterns} -def database_inventory(database: str) -> list[str]: - return re.findall(r"CREATE TABLE IF NOT EXISTS\s+([a-zA-Z0-9_]+)", database) +def database_inventory(sources: list[str]) -> list[str]: + tables: list[str] = [] + for database in sources: + tables.extend( + re.findall( + r"CREATE TABLE IF NOT EXISTS\s+([a-zA-Z0-9_]+)", database + ) + ) + return list(dict.fromkeys(tables)) def python_functions(path: str, prefixes: tuple[str, ...]) -> list[str]: @@ -70,11 +77,13 @@ def css_layers(html: str) -> list[str]: def code_hotspots() -> list[dict[str, Any]]: candidates = [ - "server.py", + "backend/application.py", "database.py", - "screener.py", - "market_insights.py", - "tushare_client.py", + "backend/features/screener/engine.py", + "backend/features/market/insights.py", + "backend/data/providers/tushare_client.py", + "backend/features/heaven/service.py", + "backend/features/heaven/engine.py", "frontend/index.html", "frontend/app.js", "frontend/styles/styles.css", @@ -88,6 +97,8 @@ def code_hotspots() -> list[dict[str, Any]]: rows = [] for name in candidates: path = ROOT / name + if not path.is_file(): + continue rows.append( { "path": name, @@ -100,14 +111,21 @@ def code_hotspots() -> list[dict[str, Any]]: def build() -> dict[str, Any]: html = source("frontend/index.html") - server = source("server.py") - database = source("database.py") + server = source("backend/application.py") + database_sources = [source("database.py")] + database_sources.extend( + path.read_text(encoding="utf-8") + for path in sorted((ROOT / "backend" / "database" / "migrations").glob("m*.py")) + ) + database_sources.append( + source("backend/database/migrations/runner.py") + ) pages = page_inventory(html) api = api_inventory(server) - tables = database_inventory(database) + tables = database_inventory(database_sources) return { "schema_version": 1, - "captured_from": "governed source tree", + "captured_from": "app modular preservation candidate", "runtime": { "http_server": "http.server.ThreadingHTTPServer", "application_processes": 1, @@ -126,21 +144,22 @@ def build() -> dict[str, Any]: "api": api, "database_tables": tables, "background_job_methods": python_functions( - "server.py", ("_background", "_run_background", "_schedule_", "run_automatic") + "backend/application.py", + ("_background", "_run_background", "_schedule_", "run_automatic"), ), "external_data_adapters": [ - {"provider": "tushare", "path": "tushare_client.py", "runtime_role": "primary deterministic market data"}, - {"provider": "ifind", "path": "ifind_client.py", "runtime_role": "realtime, charts, snapshots, enrichment"}, - {"provider": "eastmoney", "path": "chart_data_provider.py", "runtime_role": "display chart fallback"}, - {"provider": "eastmoney", "path": "realtime_aggregator.py", "runtime_role": "isolated realtime observation"}, - {"provider": "tencent", "path": "realtime_aggregator.py", "runtime_role": "index observation fallback"}, + {"provider": "tushare", "path": "backend/data/providers/tushare_client.py", "runtime_role": "primary deterministic market data"}, + {"provider": "ifind", "path": "backend/data/providers/ifind_client.py", "runtime_role": "realtime, charts, snapshots, enrichment"}, + {"provider": "eastmoney", "path": "backend/features/market/charts.py", "runtime_role": "display chart fallback"}, + {"provider": "eastmoney", "path": "backend/data/realtime.py", "runtime_role": "isolated realtime observation"}, + {"provider": "tencent", "path": "backend/data/realtime.py", "runtime_role": "index observation fallback"}, ], "llm_entrypoints": [ - {"function": "stream_with_mentor", "path": "mentor_agent.py"}, - {"function": "interpret_heaven", "path": "heaven_agent.py"}, - {"function": "stream_review_assistant", "path": "assistant_agent.py"}, - {"function": "compile_strategy_with_llm", "path": "llm_strategy.py"}, - {"function": "test_llm_connection", "path": "llm_strategy.py"}, + {"function": "stream_with_mentor", "path": "backend/features/mentor/agent.py"}, + {"function": "interpret_heaven", "path": "backend/features/heaven/agent.py"}, + {"function": "stream_review_assistant", "path": "backend/features/review/agent.py"}, + {"function": "compile_strategy_with_llm", "path": "backend/features/screener/compiler.py"}, + {"function": "test_llm_connection", "path": "backend/features/screener/compiler.py"}, ], "css_layers": css_layers(html), "code_hotspots": code_hotspots(), @@ -154,7 +173,10 @@ def main() -> int: rendered = json.dumps(build(), ensure_ascii=False, indent=2) + "\n" if args.check: if not OUTPUT.exists() or OUTPUT.read_text(encoding="utf-8") != rendered: - raise SystemExit("architecture inventory is stale; run tools/build_architecture_inventory.py") + raise SystemExit( + "architecture inventory is stale; run " + "tools/build_architecture_inventory.py" + ) print("Architecture inventory is current.") return 0 OUTPUT.parent.mkdir(parents=True, exist_ok=True) diff --git a/tools/build_preservation_manifest.py b/tools/build_preservation_manifest.py index aa12c9a..dae5bea 100644 --- a/tools/build_preservation_manifest.py +++ b/tools/build_preservation_manifest.py @@ -1,14 +1,13 @@ from __future__ import annotations +import argparse import hashlib import json from datetime import datetime from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] -TARGET = ROOT / "app" -OUTPUT = ROOT / "docs" / "migration" / "原版资产清单.json" +REPOSITORY_ROOT = Path(__file__).resolve().parents[2] DIRECTORIES = ( "backend", @@ -69,33 +68,57 @@ def digest(path: Path) -> str: return checksum.hexdigest() -def source_files() -> list[Path]: - files = [ROOT / name for name in ROOT_FILES] +def display_path(path: Path, base: Path) -> str: + try: + return path.relative_to(base).as_posix() or "." + except ValueError: + return path.as_posix() + + +def source_files(source_root: Path) -> list[Path]: + files = [source_root / name for name in ROOT_FILES] for directory in DIRECTORIES: + root = source_root / directory + if not root.is_dir(): + continue files.extend( path - for path in (ROOT / directory).rglob("*") + for path in root.rglob("*") if path.is_file() and "__pycache__" not in path.parts ) - return sorted(set(files), key=lambda path: path.relative_to(ROOT).as_posix()) + return sorted( + set(files), key=lambda path: path.relative_to(source_root).as_posix() + ) -def build_manifest() -> dict[str, object]: +def build_manifest( + source_root: Path, + target_root: Path, + source_commit: str, +) -> dict[str, object]: assets = [] mismatches = [] - for source in source_files(): - relative = source.relative_to(ROOT) - target = TARGET / relative - source_hash = digest(source) + for source in source_files(source_root): + relative = source.relative_to(source_root) + target = target_root / relative + source_exists = source.is_file() + source_hash = digest(source) if source_exists else "" target_hash = digest(target) if target.is_file() else "" - status = "identical" if source_hash == target_hash else "mismatch" + if not source_exists: + status = "missing_source" + elif not target.is_file(): + status = "missing_target" + elif source_hash == target_hash: + status = "identical" + else: + status = "mismatch" if status != "identical": mismatches.append(relative.as_posix()) assets.append( { "source": relative.as_posix(), - "target": f"app/{relative.as_posix()}", - "bytes": source.stat().st_size, + "target": display_path(target, REPOSITORY_ROOT), + "bytes": source.stat().st_size if source_exists else 0, "sha256": source_hash, "disposition": "original_copy_pending_move", "status": status, @@ -104,9 +127,9 @@ def build_manifest() -> dict[str, object]: return { "schema_version": 1, "generated_at": datetime.now().astimezone().isoformat(timespec="seconds"), - "source_commit": "41329943c4878fc09ed82ec376eb93ab151e4092", - "source_root": ".", - "target_root": "app", + "source_commit": source_commit, + "source_root": display_path(source_root, REPOSITORY_ROOT), + "target_root": display_path(target_root, REPOSITORY_ROOT), "excluded": [ "next/", "data/review.db and SQLite sidecars", @@ -123,14 +146,34 @@ def build_manifest() -> dict[str, object]: def main() -> int: - manifest = build_manifest() - OUTPUT.write_text( + parser = argparse.ArgumentParser( + description=( + "Build an exact-copy manifest for a preservation migration stage. " + "This is a migration-only tool; it does not describe the final moved layout." + ) + ) + parser.add_argument("--source-root", type=Path, default=REPOSITORY_ROOT) + parser.add_argument("--target-root", type=Path, default=REPOSITORY_ROOT / "app") + parser.add_argument("--source-commit", default="") + parser.add_argument( + "--output", + type=Path, + required=True, + help="write to a new audit path; do not overwrite committed slice evidence", + ) + args = parser.parse_args() + source_root = args.source_root.resolve() + target_root = args.target_root.resolve() + output = args.output.resolve() + manifest = build_manifest(source_root, target_root, args.source_commit) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text( json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8", ) print( f"assets={manifest['asset_count']} mismatches={manifest['mismatch_count']} " - f"output={OUTPUT}" + f"output={output}" ) return 1 if manifest["mismatch_count"] else 0 diff --git a/tools/split_frontend_runtime.py b/tools/split_frontend_runtime.py index e61e199..41a32d6 100644 --- a/tools/split_frontend_runtime.py +++ b/tools/split_frontend_runtime.py @@ -1,5 +1,6 @@ from __future__ import annotations +import argparse import hashlib import json import re @@ -91,6 +92,20 @@ def extract_written_chunks(paths: list[Path]) -> dict[tuple[int, int], str]: def main() -> None: + parser = argparse.ArgumentParser( + description="Reproduce the one-time Slice 10 frontend source split" + ) + parser.add_argument( + "--apply", + action="store_true", + help="perform the historical split; this rewrites frontend runtime files", + ) + args = parser.parse_args() + if not args.apply: + parser.error( + "this migration-only tool rewrites files; pass --apply only when " + "reproducing Slice 10 from its documented pre-split checkpoint" + ) source_path = ORIGINAL_STATIC / "app.js" target_path = FRONTEND_ROOT / "app.js" source_bytes = source_path.read_bytes() diff --git a/tools/verify_baseline.py b/tools/verify_baseline.py index f788c82..9bf656c 100644 --- a/tools/verify_baseline.py +++ b/tools/verify_baseline.py @@ -5,10 +5,15 @@ import shutil import sqlite3 import subprocess import sys +import time +import urllib.error +import urllib.request from pathlib import Path ROOT = Path(__file__).resolve().parents[1] +FRONTEND_ROOT = ROOT / "frontend" +E2E_URL = "http://127.0.0.1:8876/index.html" def run(label: str, command: list[str]) -> None: @@ -28,8 +33,63 @@ def verify_database() -> None: print(f"\n[database] integrity_check=ok size={database.stat().st_size}") +def url_reachable(url: str) -> bool: + try: + with urllib.request.urlopen(url, timeout=1) as response: + return response.status == 200 + except (OSError, urllib.error.URLError): + return False + + +def start_e2e_server() -> subprocess.Popen[bytes] | None: + if url_reachable(E2E_URL): + print(f"\n[e2e-server] reusing {E2E_URL}") + return None + creation_flags = subprocess.CREATE_NO_WINDOW if sys.platform == "win32" else 0 + process = subprocess.Popen( + [ + sys.executable, + "-m", + "http.server", + "8876", + "--bind", + "127.0.0.1", + "--directory", + "frontend", + ], + cwd=ROOT, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + creationflags=creation_flags, + ) + deadline = time.monotonic() + 15 + while time.monotonic() < deadline: + if process.poll() is not None: + raise RuntimeError("Playwright static server exited before becoming ready") + if url_reachable(E2E_URL): + print(f"\n[e2e-server] started {E2E_URL}") + return process + time.sleep(0.1) + stop_e2e_server(process) + raise RuntimeError("Playwright static server did not become ready within 15 seconds") + + +def stop_e2e_server(process: subprocess.Popen[bytes] | None) -> None: + if process is None: + return + process.terminate() + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=5) + print("\n[e2e-server] stopped") + + def main() -> int: - parser = argparse.ArgumentParser(description="Verify the governance regression baseline") + parser = argparse.ArgumentParser( + description="Verify the modular preservation candidate" + ) parser.add_argument( "--e2e", action="store_true", @@ -38,19 +98,37 @@ def main() -> int: args = parser.parse_args() run("python", [sys.executable, "-m", "unittest", "discover", "-s", "tests"]) + run( + "api-registry", + [sys.executable, "tools/build_api_registry.py", "--check"], + ) + run( + "architecture-inventory", + [sys.executable, "tools/build_architecture_inventory.py", "--check"], + ) node = shutil.which("node") if not node: raise RuntimeError("node is required for JavaScript syntax checks") - for script in ("static/app.js", "static/heaven-loading-v2.js"): - run("javascript", [node, "--check", script]) + scripts = sorted(FRONTEND_ROOT.rglob("*.js")) + if not scripts: + raise RuntimeError(f"no JavaScript files found under {FRONTEND_ROOT}") + for script in scripts: + run( + "javascript", + [node, "--check", script.relative_to(ROOT).as_posix()], + ) run("patch", ["git", "diff", "--check"]) verify_database() if args.e2e: - npm = shutil.which("npm.cmd" if sys.platform == "win32" else "npm") - if not npm: - raise RuntimeError("npm is required for the Playwright suite") - run("playwright", [npm, "run", "test:e2e"]) + npx = shutil.which("npx.cmd" if sys.platform == "win32" else "npx") + if not npx: + raise RuntimeError("npx is required for the Playwright suite") + server = start_e2e_server() + try: + run("playwright", [npx, "playwright", "test", "--reporter=dot"]) + finally: + stop_e2e_server(server) print("\nBaseline verification passed.") return 0