Compare commits

..
Author SHA1 Message Date
总管andmultica-agent b950ea4344 fix(HEL-494): keep filtered reference lookups inside datahub
Co-authored-by: multica-agent <github@multica.ai>
2026-09-09 00:50:33 +08:00
总管andmultica-agent 41f8509a98 fix(HEL-494): retry incomplete sector publications
Co-authored-by: multica-agent <github@multica.ai>
2026-09-09 00:46:18 +08:00
总管andmultica-agent c9e2d30780 fix(HEL-494): enforce datahub as sole website market boundary
Co-authored-by: multica-agent <github@multica.ai>
2026-09-09 00:22:27 +08:00
8a7d1f3698 fix(HEL-494): 网站市场客户端改为纯中枢 Facade,并迁移 iFinD 凭据到中枢
生产 gateway 不再读取 Tushare token 或实例化 TushareProvider/TushareClient;问财凭据经带鉴权的中枢接口加密入库,避免发版后 iFinD 未配置。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 23:37:53 +08:00
100752f43c fix(HEL-494): 切断网站生产装配外源直连,iFinD 与实时观察改走中枢
生产 gateway 不再实例化 iFinD、东财图和免费实时聚合器;问财与竞价快照作为中枢内部数据源。全站阻断外源测试覆盖日K、报价、图表、问财和竞价快照。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 22:41:51 +08:00
0b8419abca fix(HEL-494): 数据中枢独占调度,主网站不再回退旧接口
主网站只向中枢要业务数据;来源选择、切源、补数全部在中枢内部完成,失败不再走东财/腾讯/Tushare 保底。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 21:43:31 +08:00
ef13d6feb5 fix(HEL-494): 盘中双免费源故障切换并禁止问天假0覆盖
主源东财失败后自动改走腾讯行情,成功结果写入缓存;两源都失败时返回最近真实快照并标明延迟,不再显示假0。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 17:38:38 +08:00
b5d65ecb41 fix(HEL-494): 日K默认45根并修复问天行业0/205覆盖
悬浮窗和详情页只画最近45个交易日,中枢仍保留250日历史。盘后缺sw_daily时保留成分日线内核,外显走免费申万;成分行情改为全市场快照+分页,不再截成前60只。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 17:00:01 +08:00
3e828b346c fix(HEL-494): 修复个股缺失指标、问天遮罩、四爻外显并回补250日K
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 16:26:13 +08:00
c8a9376adb fix(HEL-490): 真实装配接通中枢并收编估值晚间复核
把 query/行情钩子绑到内层 TushareClient,图表接受不完整日K窗口;收编现网 HEL-423 未提交的估值复核,避免换版丢掉。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 15:03:34 +08:00
1c2f2ac057 feat(HEL-490): 剩余行情改由数据中枢主线路提供
正式页面以 8766 为主线路,旧接口只作故障备用;compose 钉死全部 DATAHUB_READ_*,避免现网残留 0 造成假完成。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 12:03:30 +08:00
5d3465987d fix(HEL-488): 盘中日K补上今天实时变化的一根
悬浮窗和详情页在 Tushare rt_k / iFinD 不可用时,改用免费实时行情或当日分时生成今日K,收盘后正式日K就绪再无缝替换。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 11:18:21 +08:00
dd89a09643 fix(HEL-487): 盘中当天看板在 rt_k 无权限时降级到免费实时源
rt_k 失败、无权限、超时或空结果时改用东财全市场快照,再失败则用腾讯批量行情;两者都失败仍不退回昨天。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 10:52:10 +08:00
a043bc9eb1 fix(HEL-485): 盘中选择当天不再整页退回昨天
交易时段缺少盘后正式数据时继续展示当天盘中行情,只有开盘前、周末和历史日期才沿用最近收盘结果。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 10:16:52 +08:00
acde4de40d fix(HEL-484): 中枢分时接口空 date 按当天查询
缺少或为空的 date 不再 400,按当天处理;显式历史日期保持原行为。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 10:07:44 +08:00
3d2c1252f1 fix(HEL-482): 开盘前分时回退最近交易日,并接通中枢失败回旧通道
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-08 09:44:00 +08:00
605f97e5df feat(HEL-463): 接入剩余行情数据到 datahub
扩展盘后正式集(涨跌停/人气/龙虎榜/板块日线)与盘中观察 API(报价/指数/分时),网站 bridge 按开关接入并回退旧链路;问天改为按数据依赖跟随开关,不再整栈强制旧路径。

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: multica-agent <github@multica.ai>
2026-09-05 17:30:58 +08:00
97 changed files with 9012 additions and 725 deletions
+9 -7
View File
@@ -1,18 +1,20 @@
# Generated automatically when omitted. Back it up together with the database. # Generated automatically when omitted. Back it up together with the database.
APP_ENCRYPTION_KEY= APP_ENCRYPTION_KEY=
# Initial shared market-data credential. After first launch it is encrypted into # Market-source credentials are consumed and encrypted only by xiaobai-datahub.
# the system settings; all accounts use the same backend market snapshot. # compose.yaml masks them from the xiaobai-review website process.
TUSHARE_TOKEN=your_tushare_token_here TUSHARE_TOKEN=your_tushare_token_here
# Optional xiaobai-datahub client. All DATAHUB_READ_* / DATAHUB_SHADOW_* flags # Official xiaobai-datahub client. Read flags default on in config/datahub.config.json.
# default off in config/datahub.config.json, so the website keeps using Tushare. # compose.yaml pins every DATAHUB_READ_* to 1. The website has no provider
# fallback; source selection and failover happen inside xiaobai-datahub.
# DATAHUB_SHADOW_* can still override a single dataset.
DATAHUB_BASE_URL=http://127.0.0.1:8766 DATAHUB_BASE_URL=http://127.0.0.1:8766
DATAHUB_TOKEN= DATAHUB_TOKEN=
# Optional iFinD HTTP credential. The backend exchanges it for a short-lived # iFinD credentials live on xiaobai-datahub, not the website process.
# access token and never exposes either token to browsers. # IFIND_REFRESH_TOKEN=your_ifind_refresh_token_here
IFIND_REFRESH_TOKEN=your_ifind_refresh_token_here # IFIND_ACCESS_TOKEN=
# Initial platform member models (OpenAI-compatible). After first launch these # Initial platform member models (OpenAI-compatible). After first launch these
# are encrypted into system settings and used only by admins and active members. # are encrypted into system settings and used only by admins and active members.
+8 -3
View File
@@ -38,10 +38,15 @@ background scheduler
fields, and feature-specific exceptions belong to `backend/features/<feature>/routes.py`. fields, and feature-specific exceptions belong to `backend/features/<feature>/routes.py`.
- `backend/features/<feature>/` owns the mechanically moved service, repository, HTTP, agent, - `backend/features/<feature>/` owns the mechanically moved service, repository, HTTP, agent,
or deterministic calculation code for that product area. or deterministic calculation code for that product area.
- `backend/data/` owns provider construction, source policy, provenance, units, freshness, - `backend/data/` owns the website-side DataHub client, stable dataset contracts, provenance,
coverage, display-versus-calculation eligibility, and shared numeric normalization policies. units, freshness, coverage, display-versus-calculation eligibility, and shared numeric
normalization policies. The website process does not construct or configure external market
providers; provider credentials, source selection, retries, fallbacks, caching, and backfill
belong exclusively to the `xiaobai-datahub` service.
- `backend/data/providers/tushare_client.py` is the stable public `TushareClient` facade and - `backend/data/providers/tushare_client.py` is the stable public `TushareClient` facade and
owns only its dataclass fields and shared cache state. Tushare HTTP transport belongs to is retained as the dataset-contract compatibility surface and isolated test facade. Production
website services never instantiate it directly: its query methods are served by the DataHub
proxy. Its split modules document the stable contract: Tushare HTTP transport belongs to
`tushare_transport.py`; market overview and realtime breadth belong to `tushare_transport.py`; market overview and realtime breadth belong to
`tushare_dashboard.py`; indices belong to `tushare_indices.py`; Shenwan membership and `tushare_dashboard.py`; indices belong to `tushare_indices.py`; Shenwan membership and
industry snapshots belong to `tushare_industries.py`; generic sector snapshots belong to industry snapshots belong to `tushare_industries.py`; generic sector snapshots belong to
+7 -4
View File
@@ -22,9 +22,10 @@ xiaobai-review 容器 :8765
`-- heaven_knowledge.json 优先读取;缺失时回退到上方 seed `-- heaven_knowledge.json 优先读取;缺失时回退到上方 seed
``` ```
账号、加密后的公共数据 Token、平台模型 API Key、生辰资料、行情快照和复盘数据均在 账号、平台模型 API Key、生辰资料、行情快照和复盘数据均在 `data/review.db`。外部行情源
`data/review.db`。解密密钥来自 `.env` 中的 `APP_ENCRYPTION_KEY`。数据库与 Token 只允许保存在 `xiaobai-datahub` 的环境或凭据库,网站进程不读取、不保存,也不向
密钥必须成对备份,任意一个丢失都无法恢复账号内的加密资料。 提供方直接发请求。解密密钥来自 `.env` 中的 `APP_ENCRYPTION_KEY`。数据库与密钥必须成对
备份,任意一个丢失都无法恢复账号内的加密资料。
问天静态知识文件: 问天静态知识文件:
@@ -41,7 +42,9 @@ xiaobai-review 容器 :8765
挂载进入容器,但被 Git 与 Docker 构建上下文排除,不会进入 Gitea 或镜像。私有 Skill 挂载进入容器,但被 Git 与 Docker 构建上下文排除,不会进入 Gitea 或镜像。私有 Skill
只对管理员账号返回和开放调用,也会随本指南的 `data` 备份一起保存。 只对管理员账号返回和开放调用,也会随本指南的 `data` 备份一起保存。
首个注册账号自动成为管理员。管理员在“系统管理”中配置全站共享行情、后台刷新、平台会员模型及手动会员;普通用户的“账号设置”用于个人资料、会员状态、修改密码和切换账号。后台行情更新不会主动刷新任何浏览器页面。 首个注册账号自动成为管理员。管理员在网站“系统管理”中查看数据中枢状态并配置后台刷新、
平台会员模型及手动会员;行情源凭据和调度策略在数据中枢后台统一管理。普通用户的“账号设置”
用于个人资料、会员状态、修改密码和切换账号。后台行情更新不会主动刷新任何浏览器页面。
## 2. 服务器要求 ## 2. 服务器要求
-17
View File
@@ -60,22 +60,6 @@ from backend.llm.service import LLMServiceMixin
from database import ReviewDatabase from database import ReviewDatabase
LEGACY_SECRET_KEYS = {
"TUSHARE_TOKEN",
"IFIND_REFRESH_TOKEN",
"IFIND_ACCESS_TOKEN",
"LLM_API_KEY",
"LLM_BASE_URL",
"LLM_MODEL",
"LLM_PRIMARY_API_KEY",
"LLM_PRIMARY_BASE_URL",
"LLM_PRIMARY_MODEL",
"LLM_FALLBACK_API_KEY",
"LLM_FALLBACK_BASE_URL",
"LLM_FALLBACK_MODEL",
}
class DashboardService( class DashboardService(
SystemServiceMixin, SystemServiceMixin,
AccountApplicationMixin, AccountApplicationMixin,
@@ -121,7 +105,6 @@ class DashboardService(
self._system_credentials, self._system_credentials,
MENTOR_SKILLS_DIR, MENTOR_SKILLS_DIR,
PRIVATE_MENTOR_SKILLS_DIR, PRIVATE_MENTOR_SKILLS_DIR,
lambda: self.token,
) )
self.data_gateway = self.container.data_gateway self.data_gateway = self.container.data_gateway
self.ifind = self.container.ifind self.ifind = self.container.ifind
+5 -7
View File
@@ -2,7 +2,6 @@ from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from collections.abc import Callable
from backend.data import DataGateway, build_data_gateway from backend.data import DataGateway, build_data_gateway
from backend.database.repositories import RepositoryBundle, build_repository_bundle from backend.database.repositories import RepositoryBundle, build_repository_bundle
@@ -13,8 +12,8 @@ from backend.features.screener.engine import ScreenerEngine
from backend.features.screener.tracking import StrategyTrackingService from backend.features.screener.tracking import StrategyTrackingService
from backend.jobs import InProcessJobRunner, JobRegistry, SQLiteJobRunRepository from backend.jobs import InProcessJobRunner, JobRegistry, SQLiteJobRunRepository
from database import ReviewDatabase from database import ReviewDatabase
from backend.data.providers.ifind_client import IfindHttpClient from backend.data.datahub.ifind_proxy import HubIfindProxy
from backend.data.realtime import WebRealtimeAggregator from backend.data.datahub.realtime_proxy import HubRealtimeProxy
from backend.features.market.charts import MarketChartClient from backend.features.market.charts import MarketChartClient
@@ -23,13 +22,13 @@ class ApplicationContainer:
database: ReviewDatabase database: ReviewDatabase
repositories: RepositoryBundle repositories: RepositoryBundle
data_gateway: DataGateway data_gateway: DataGateway
ifind: IfindHttpClient ifind: HubIfindProxy
screener: ScreenerEngine screener: ScreenerEngine
strategy_tracking: StrategyTrackingService strategy_tracking: StrategyTrackingService
alert_service: AlertService alert_service: AlertService
trade_journal: TradeJournalService trade_journal: TradeJournalService
mentor_skills: MentorSkillRegistry mentor_skills: MentorSkillRegistry
realtime_aggregator: WebRealtimeAggregator realtime_aggregator: HubRealtimeProxy
chart_data: MarketChartClient chart_data: MarketChartClient
jobs: InProcessJobRunner jobs: InProcessJobRunner
@@ -39,9 +38,8 @@ def build_application_container(
credentials: dict[str, object], credentials: dict[str, object],
mentor_skills_dir: Path, mentor_skills_dir: Path,
private_mentor_skills_dir: Path, private_mentor_skills_dir: Path,
tushare_token_supplier: Callable[[], str] | None = None,
) -> ApplicationContainer: ) -> ApplicationContainer:
data_gateway = build_data_gateway(credentials, tushare_token_supplier) data_gateway = build_data_gateway(credentials)
repositories = build_repository_bundle(database) repositories = build_repository_bundle(database)
jobs = InProcessJobRunner(JobRegistry.load(), SQLiteJobRunRepository(database)) jobs = InProcessJobRunner(JobRegistry.load(), SQLiteJobRunRepository(database))
return ApplicationContainer( return ApplicationContainer(
-3
View File
@@ -10,11 +10,8 @@ from backend.features.accounts.security import SecretVault
def environment_credentials(environment: Mapping[str, str]) -> dict[str, str]: def environment_credentials(environment: Mapping[str, str]) -> dict[str, str]:
return { return {
"tushare_token": str(environment.get("TUSHARE_TOKEN") or "").strip(),
"datahub_token": str(environment.get("DATAHUB_TOKEN") or "").strip(), "datahub_token": str(environment.get("DATAHUB_TOKEN") or "").strip(),
"datahub_base_url": str(environment.get("DATAHUB_BASE_URL") or "").strip(), "datahub_base_url": str(environment.get("DATAHUB_BASE_URL") or "").strip(),
"ifind_refresh_token": str(environment.get("IFIND_REFRESH_TOKEN") or "").strip(),
"ifind_access_token": str(environment.get("IFIND_ACCESS_TOKEN") or "").strip(),
"platform_llm_primary_api_key": str( "platform_llm_primary_api_key": str(
environment.get("LLM_PRIMARY_API_KEY") or environment.get("LLM_API_KEY") or "" environment.get("LLM_PRIMARY_API_KEY") or environment.get("LLM_API_KEY") or ""
).strip(), ).strip(),
+408 -60
View File
@@ -2,7 +2,8 @@ from __future__ import annotations
import logging import logging
import sys import sys
from typing import Any, Callable from threading import Lock
from typing import Any, Callable, ClassVar
from backend.data.datahub.client import DatahubClient, DatahubResponse from backend.data.datahub.client import DatahubClient, DatahubResponse
from backend.data.datahub.compare import compare_rows from backend.data.datahub.compare import compare_rows
@@ -16,16 +17,47 @@ from backend.data.datahub.native import (
yyyymmdd, yyyymmdd,
) )
from backend.data.datahub.redact import redact_text, redact_value from backend.data.datahub.redact import redact_text, redact_value
from backend.data.datahub.route_state import LEDGER
from backend.data.datahub.settings import DatahubSettings from backend.data.datahub.settings import DatahubSettings
from backend.data.providers.tushare_client import TushareClient from backend.data.providers.tushare_daily import DailyMarketMixin
from backend.data.providers.tushare_dashboard import DashboardMixin
from backend.data.providers.tushare_dragon_tiger import DragonTigerMixin
from backend.data.providers.tushare_indices import IndexMixin
from backend.data.providers.tushare_industries import ShenwanIndustryMixin
from backend.data.providers.tushare_sectors import SectorMixin
from backend.data.providers.tushare_stocks import StockMixin
from backend.data.providers.tushare_transport import TushareError
LOGGER = logging.getLogger("xiaobai.datahub") LOGGER = logging.getLogger("xiaobai.datahub")
ShadowSink = Callable[[dict[str, Any]], None] ShadowSink = Callable[[dict[str, Any]], None]
EMPTY_FAIL_DATASETS = {"stocks", "daily", "index_daily", "valuation", "moneyflow", "auction"}
def _usable_intraday_points(rows: list[Any]) -> list[dict[str, Any]]:
points: list[dict[str, Any]] = []
for row in rows:
if not isinstance(row, dict):
continue
try:
close = float(row.get("close") or 0)
except (TypeError, ValueError):
close = 0.0
if close <= 0:
continue
point = dict(row)
if "average" not in point and point.get("avg_price") is not None:
point["average"] = point.get("avg_price")
points.append(point)
return points
EMPTY_FAIL_DATASETS = {
"stocks", "daily", "index_daily", "valuation", "moneyflow", "auction",
"limit_events", "sector_daily",
}
def looks_like_heaven(module_name: str, filename: str = "") -> bool: def looks_like_heaven(module_name: str, filename: str = "") -> bool:
"""问天调用栈识别。问天未永久冻结,只是本阶段仍走旧 Tushare 链路。""" """问天调用栈识别(诊断用)。问天按数据集依赖接入,不再整栈强制旧链路。"""
path = filename.replace("\\", "/") path = filename.replace("\\", "/")
return module_name.startswith("backend.features.heaven") or "/features/heaven/" in path return module_name.startswith("backend.features.heaven") or "/features/heaven/" in path
@@ -88,55 +120,228 @@ class DatahubBridge:
self._log_failure("status", exc) self._log_failure("status", exc)
return None return None
def try_intraday(self, code: str) -> dict[str, Any] | None:
flags = self.settings.flags("intraday")
if not flags.read:
return None
try:
response = self.client.intraday_points(code=code)
data = response.data
if not isinstance(data, dict):
raise DatahubError("EMPTY", "datahub intraday payload invalid")
points = _usable_intraday_points(data.get("points") or [])
if not points:
raise DatahubError("EMPTY", "datahub intraday empty")
if (response.meta or {}).get("stale"):
raise DatahubError("STALE", "datahub intraday stale")
self._record_route("intraday", "datahub", str((response.meta or {}).get("source") or "datahub"))
return {
"entity_type": str(data.get("entity_type") or "stock"),
"identifier": str(data.get("identifier") or code),
"name": str(data.get("name") or ""),
"code": str(data.get("code") or code),
"trade_date": str(data.get("trade_date") or points[-1].get("date") or ""),
"previous_close": float(data.get("previous_close") or 0),
"points": points,
"source": "datahub",
}
except Exception as exc:
self._log_failure("intraday", exc)
return None
def try_market_quotes(self, trade_date: str = "") -> list[dict[str, Any]] | None:
return self._try_quote_rows("quotes", {}, expected_date=trade_date, minimum=200)
def try_quotes(self, codes: list[str]) -> list[dict[str, Any]] | None:
cleaned = [str(item or "").strip() for item in codes if str(item or "").strip()]
if not cleaned:
return None
return self._try_quote_rows("quotes", {"codes": ",".join(cleaned)}, minimum=1)
def try_index_quotes(self) -> list[dict[str, Any]] | None:
flags = self.settings.flags("index_quotes")
if not flags.read:
return None
try:
response = self.client.index_quotes()
rows = [dict(item) for item in (response.data or []) if isinstance(item, dict)]
if len(rows) < 3:
raise DatahubError("EMPTY", "datahub index quotes incomplete")
if (response.meta or {}).get("stale"):
raise DatahubError("STALE", "datahub index quotes stale")
self._record_route(
"index_quotes",
"datahub",
str((response.meta or {}).get("source") or "datahub"),
)
return rows
except Exception as exc:
self._log_failure("index_quotes", exc)
return None
def try_sector_quote(self, code: str, trade_date: str = "") -> dict[str, Any] | None:
flags = self.settings.flags("quotes")
if not flags.read:
return None
try:
response = self.client.sector_quote(code, trade_date)
data = response.data
if not isinstance(data, dict) or not data:
raise DatahubError("EMPTY", "datahub sector quote empty")
row = dict(data)
if (response.meta or {}).get("stale"):
row["delayed"] = True
row["delay_seconds"] = int((response.meta or {}).get("staleness_seconds") or 0)
row["delay_notice"] = str((response.meta or {}).get("delay_notice") or "")
self._record_route("quotes", "datahub", str((response.meta or {}).get("source") or "datahub"))
return row
except Exception as exc:
self._log_failure("quotes", exc)
return None
def try_limit_pool(self, trade_date: str = "") -> list[dict[str, Any]] | None:
flags = self.settings.flags("limit_events")
if not flags.read:
return None
try:
response = self.client.limit_pool(trade_date)
rows = [dict(item) for item in (response.data or []) if isinstance(item, dict)]
if not rows:
raise DatahubError("EMPTY", "datahub limit pool empty")
self._record_route(
"limit_events",
"datahub",
str((response.meta or {}).get("source") or "datahub"),
)
return rows
except Exception as exc:
self._log_failure("limit_events", exc)
return None
def try_daily_chart(
self,
code: str,
end_date: str,
limit: int = 90,
dataset: str = "daily",
) -> list[dict[str, Any]] | None:
flags = self.settings.flags(dataset)
if not flags.read:
return None
compact_end = yyyymmdd(end_date)
if not compact_end:
return None
try:
start = _shift_yyyymmdd(compact_end, -max(190, int(limit) * 3))
if dataset == "index_daily":
response = self._paginate(
self.client.index_bars,
{"code": code, "from": start, "to": compact_end},
)
elif dataset == "sector_daily":
response = self._paginate(
self.client.sectors,
{"code": code, "from": start, "to": compact_end},
)
else:
response = self._paginate(
self.client.daily_bars,
{"code": code, "from": start, "to": compact_end, "adjust": "none"},
)
# Charts can use a partial history window; do not discard usable bars
# just because the requested lookback is not fully covered.
self._validate_usable(
dataset,
list(response.data or []),
response,
require_complete=False,
)
rows = _chart_bars(list(response.data or []))
if not rows:
raise DatahubError("EMPTY", f"{dataset} chart empty")
self._record_route(dataset, "datahub", str((response.meta or {}).get("source") or "datahub"))
return rows[-max(1, int(limit)):]
except Exception as exc:
self._log_failure(dataset, exc)
return None
def record_legacy(self, dataset: str, source: str = "", error: str = "") -> None:
self._record_route(dataset, "legacy", source, error)
def route_snapshot(self) -> list[dict[str, Any]]:
return LEDGER.snapshot()
def _try_quote_rows(
self,
dataset: str,
params: dict[str, Any],
expected_date: str = "",
minimum: int = 1,
) -> list[dict[str, Any]] | None:
flags = self.settings.flags(dataset)
if not flags.read:
return None
try:
response = self.client.quotes_latest(**params)
rows = [_native_quote(item) for item in (response.data or []) if isinstance(item, dict)]
rows = [item for item in rows if item]
want = yyyymmdd(expected_date)
if want:
dated = [item for item in rows if not item.get("quote_date") or item.get("quote_date") == want]
if dated:
rows = dated
if len(rows) < minimum:
raise DatahubError("EMPTY", f"datahub {dataset} empty")
stale = bool((response.meta or {}).get("stale"))
delay = int((response.meta or {}).get("staleness_seconds") or 0)
notice = str((response.meta or {}).get("delay_notice") or "")
source = str((response.meta or {}).get("source") or "datahub")
if stale:
for item in rows:
item["delayed"] = True
item["delay_seconds"] = delay
item["delay_notice"] = notice
item["source"] = source
self._record_route(dataset, "datahub", source)
return rows
except Exception as exc:
self._log_failure(dataset, exc)
return None
def query( def query(
self, self,
api_name: str, api_name: str,
params: dict[str, Any] | None, params: dict[str, Any] | None = None,
fields: str, fields: str = "",
legacy_query: Callable[..., list[dict[str, Any]]],
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
if api_name == "rt_sw_k":
raise TushareError("rt_sw_k is disabled; use published sw_daily or free Shenwan realtime")
dataset = API_TO_DATASET.get(api_name) dataset = API_TO_DATASET.get(api_name)
# 问天允许后续纳入 datahub;首批只读接入仍保持旧链路,避免误切。 if dataset:
if not dataset or self.heaven_guard(): flags = self.settings.flags(dataset)
return legacy_query(api_name, params, fields) if flags.read:
flags = self.settings.flags(dataset) try:
if not flags.read and not flags.shadow: response = self._fetch_dataset(dataset, params or {}, api_name=api_name)
return legacy_query(api_name, params, fields) hub_canonical = self._extract_rows(dataset, response, params or {})
hub_rows = to_native_rows(dataset, hub_canonical)
hub_rows: list[dict[str, Any]] | None = None self._validate_usable(dataset, hub_rows, response)
hub_meta: dict[str, Any] = {} self._record_route(dataset, "datahub", str(response.meta.get("source") or "datahub"))
hub_error: str | None = None
hub_canonical: list[dict[str, Any]] = []
try:
response = self._fetch_dataset(dataset, params or {})
hub_canonical = self._extract_rows(dataset, response, params or {})
hub_rows = to_native_rows(dataset, hub_canonical)
hub_meta = dict(response.meta)
self._validate_usable(dataset, hub_rows, response)
except Exception as exc:
hub_error = self._error_text(exc)
self._log_failure(dataset, exc)
if flags.shadow:
try:
legacy_rows = legacy_query(api_name, params, fields)
except Exception as exc:
if flags.read and hub_rows is not None and hub_error is None:
self._emit_shadow(
compare_rows(dataset, [], hub_canonical, hub_meta, self._error_text(exc), fields)
)
return project_fields(hub_rows, fields) return project_fields(hub_rows, fields)
raise except Exception as exc:
self._emit_shadow(compare_rows(dataset, legacy_rows, hub_canonical, hub_meta, hub_error, fields)) self._log_failure(dataset, exc)
if flags.read and hub_rows is not None and hub_error is None: try:
return project_fields(hub_rows, fields) response = self.client.query_api(api_name, params or {}, fields)
return legacy_rows rows = [dict(item) for item in (response.data or []) if isinstance(item, dict)]
if dataset:
self._record_route(dataset, "datahub", str((response.meta or {}).get("source") or "datahub"))
else:
self._record_route(api_name, "datahub", str((response.meta or {}).get("source") or "datahub"))
return rows if not fields else project_fields(rows, fields)
except Exception as exc:
self._log_failure(dataset or api_name, exc)
raise TushareError(self._error_text(exc)) from exc
if flags.read and hub_rows is not None and hub_error is None: def _fetch_dataset(self, dataset: str, params: dict[str, Any], api_name: str = "") -> DatahubResponse:
return project_fields(hub_rows, fields)
return legacy_query(api_name, params, fields)
def _fetch_dataset(self, dataset: str, params: dict[str, Any]) -> DatahubResponse:
date = yyyymmdd(params.get("trade_date") or params.get("date")) date = yyyymmdd(params.get("trade_date") or params.get("date"))
start = yyyymmdd(params.get("start_date") or params.get("from") or date) start = yyyymmdd(params.get("start_date") or params.get("from") or date)
end = yyyymmdd(params.get("end_date") or params.get("to") or date) end = yyyymmdd(params.get("end_date") or params.get("to") or date)
@@ -153,6 +358,10 @@ class DatahubBridge:
"valuation": self.client.valuation, "valuation": self.client.valuation,
"moneyflow": self.client.moneyflow, "moneyflow": self.client.moneyflow,
"auction": self.client.auction, "auction": self.client.auction,
"limit_events": self.client.limit_events,
"popularity": self.client.popularity,
"dragon_tiger": self.client.dragon_tiger,
"sector_daily": self.client.sectors,
} }
fetcher = fetchers[dataset] fetcher = fetchers[dataset]
query: dict[str, Any] = {} query: dict[str, Any] = {}
@@ -167,6 +376,23 @@ class DatahubBridge:
query["to"] = end query["to"] = end
if dataset == "daily": if dataset == "daily":
query["adjust"] = "none" query["adjust"] = "none"
if dataset == "limit_events":
limit_type = str(params.get("limit_type") or "").strip().upper()
if limit_type:
query["limit_type"] = limit_type
if dataset == "popularity":
if api_name == "ths_hot":
query["source"] = "ths"
elif api_name == "dc_hot":
query["source"] = "dc"
if dataset == "sector_daily":
family = {
"ths_daily": "ths",
"dc_index": "dc",
"sw_daily": "sw",
}.get(api_name, "")
if family:
query["family"] = family
return self._paginate(fetcher, query) return self._paginate(fetcher, query)
def _paginate(self, fetcher: Callable[..., DatahubResponse], params: dict[str, Any]) -> DatahubResponse: def _paginate(self, fetcher: Callable[..., DatahubResponse], params: dict[str, Any]) -> DatahubResponse:
@@ -203,7 +429,13 @@ class DatahubBridge:
return filter_stock_rows(rows, params) return filter_stock_rows(rows, params)
return rows return rows
def _validate_usable(self, dataset: str, rows: list[dict[str, Any]], response: DatahubResponse) -> None: def _validate_usable(
self,
dataset: str,
rows: list[dict[str, Any]],
response: DatahubResponse,
require_complete: bool = True,
) -> None:
meta = response.meta or {} meta = response.meta or {}
stale_seconds = int(meta.get("staleness_seconds") or 0) stale_seconds = int(meta.get("staleness_seconds") or 0)
if meta.get("stale") or stale_seconds > self.settings.stale_seconds_max: if meta.get("stale") or stale_seconds > self.settings.stale_seconds_max:
@@ -211,7 +443,7 @@ class DatahubBridge:
if dataset in EMPTY_FAIL_DATASETS and not rows: if dataset in EMPTY_FAIL_DATASETS and not rows:
raise DatahubError("EMPTY", f"{dataset} returned no rows") raise DatahubError("EMPTY", f"{dataset} returned no rows")
coverage = meta.get("coverage") if isinstance(meta.get("coverage"), dict) else {} coverage = meta.get("coverage") if isinstance(meta.get("coverage"), dict) else {}
if meta.get("incomplete") is True or coverage.get("complete") is False: if require_complete and (meta.get("incomplete") is True or coverage.get("complete") is False):
missing = coverage.get("missing_count") missing = coverage.get("missing_count")
raise DatahubError("INCOMPLETE", f"{dataset} range is incomplete missing={missing}") raise DatahubError("INCOMPLETE", f"{dataset} range is incomplete missing={missing}")
@@ -226,11 +458,12 @@ class DatahubBridge:
self.shadow_sink(report) self.shadow_sink(report)
def _log_failure(self, dataset: str, exc: Exception) -> None: def _log_failure(self, dataset: str, exc: Exception) -> None:
LOGGER.warning( error = redact_text(self._error_text(exc), self.settings.secrets())
"datahub fallback dataset=%s error=%s", LOGGER.warning("datahub unavailable dataset=%s error=%s", dataset, error)
dataset, self._record_route(dataset, "datahub", "unavailable", error)
redact_text(self._error_text(exc), self.settings.secrets()),
) def _record_route(self, dataset: str, route: str, source: str = "", error: str = "") -> None:
LEDGER.record(dataset, route, source, redact_text(error, self.settings.secrets()))
def _error_text(self, exc: Exception) -> str: def _error_text(self, exc: Exception) -> str:
if isinstance(exc, DatahubError): if isinstance(exc, DatahubError):
@@ -240,10 +473,110 @@ class DatahubBridge:
return redact_text(text, self.settings.secrets()) return redact_text(text, self.settings.secrets())
class DatahubAwareTushareClient: def _native_quote(row: dict[str, Any]) -> dict[str, Any] | None:
def __init__(self, legacy: TushareClient, bridge: DatahubBridge) -> None: ts_code = str(row.get("ts_code") or "").strip()
self._legacy = legacy close = _finite(row.get("close") if row.get("close") not in (None, "") else row.get("price"))
self._bridge = bridge previous = _finite(
row.get("pre_close") if row.get("pre_close") not in (None, "") else row.get("previous_close")
)
if not ts_code or close <= 0 or previous <= 0:
return None
volume = _finite(row.get("vol") if row.get("vol") not in (None, "") else row.get("volume"))
payload = {
"ts_code": ts_code,
"name": str(row.get("name") or ts_code).strip(),
"pre_close": previous,
"open": _finite(row.get("open")),
"high": _finite(row.get("high")),
"low": _finite(row.get("low")),
"close": close,
"vol": volume,
"amount": _finite(row.get("amount")),
"num": 0,
"quote_date": yyyymmdd(row.get("quote_date") or row.get("trade_date")),
"source": str(row.get("source") or "datahub"),
}
if row.get("delayed"):
payload["delayed"] = True
payload["delay_seconds"] = int(row.get("delay_seconds") or 0)
payload["delay_notice"] = str(row.get("delay_notice") or "")
return payload
def _chart_bars(rows: list[Any]) -> list[dict[str, Any]]:
normalized: list[dict[str, Any]] = []
for row in rows:
if not isinstance(row, dict):
continue
compact = yyyymmdd(row.get("trade_date"))
close = _finite(row.get("close"))
if len(compact) != 8 or close <= 0:
continue
volume = _finite(row.get("volume") if row.get("volume") not in (None, "") else row.get("vol"))
amount = _finite(row.get("amount"))
if volume and volume < close * 10 and amount > 1000:
volume = volume * 100
trade_date = f"{compact[:4]}-{compact[4:6]}-{compact[6:8]}"
previous = normalized[-1]["close"] if normalized else 0.0
normalized.append(
{
"trade_date": trade_date,
"open": _finite(row.get("open")),
"high": _finite(row.get("high")),
"low": _finite(row.get("low")),
"close": close,
"change": round((close / previous - 1) * 100, 4) if previous else _finite(row.get("pct_chg")),
"volume": volume,
"amount_billion": amount / 100_000_000,
}
)
return normalized
def _shift_yyyymmdd(value: str, days: int) -> str:
from datetime import datetime, timedelta
stamp = datetime.strptime(value, "%Y%m%d")
return (stamp + timedelta(days=days)).strftime("%Y%m%d")
def _finite(value: Any) -> float:
try:
return float(value or 0)
except (TypeError, ValueError):
return 0.0
class DatahubAwareTushareClient(
DashboardMixin,
IndexMixin,
ShenwanIndustryMixin,
SectorMixin,
DragonTigerMixin,
StockMixin,
DailyMarketMixin,
):
"""Website market facade. Mixins call query(); query talks only to the hub."""
_realtime_reference_cache: ClassVar[dict[str, dict[str, Any]]] = {}
_realtime_reference_lock: ClassVar[Lock] = Lock()
_capital_cache: ClassVar[dict[str, dict[str, Any]]] = {}
_latest_realtime_market: ClassVar[dict[str, dict[str, Any]]] = {}
_stock_activity_cache: ClassVar[dict[str, dict[str, Any]]] = {}
_stock_listing_cache: ClassVar[dict[str, Any]] = {}
_stock_listing_lock: ClassVar[Lock] = Lock()
_suspension_cache: ClassVar[dict[str, dict[str, str] | None]] = {}
_suspension_lock: ClassVar[Lock] = Lock()
_sw_member_cache: ClassVar[dict[str, Any]] = {}
_sw_member_lock: ClassVar[Lock] = Lock()
def __init__(self, first: Any, second: Any | None = None) -> None:
# Production: DatahubAwareTushareClient(bridge)
# Older tests: DatahubAwareTushareClient(unused_legacy, bridge)
self._bridge = second if second is not None else first
self.token = "datahub"
self.timeout = 30
self.realtime_aggregator = None
def query( def query(
self, self,
@@ -251,7 +584,22 @@ class DatahubAwareTushareClient:
params: dict[str, Any] | None = None, params: dict[str, Any] | None = None,
fields: str = "", fields: str = "",
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
return self._bridge.query(api_name, params, fields, self._legacy.query) return self._bridge.query(api_name, params, fields)
def __getattr__(self, name: str) -> Any: def try_market_quotes(self, trade_date: str = "") -> list[dict[str, Any]] | None:
return getattr(self._legacy, name) return self._bridge.try_market_quotes(trade_date)
def try_quotes(self, codes: list[str]) -> list[dict[str, Any]] | None:
return self._bridge.try_quotes(codes)
def try_index_quotes(self) -> list[dict[str, Any]] | None:
return self._bridge.try_index_quotes()
def try_sector_quote(self, code: str, trade_date: str = "") -> dict[str, Any] | None:
return self._bridge.try_sector_quote(code, trade_date)
def try_limit_pool(self, trade_date: str = "") -> list[dict[str, Any]] | None:
return self._bridge.try_limit_pool(trade_date)
def record_datahub_legacy(self, dataset: str, source: str = "", error: str = "") -> None:
self._bridge.record_legacy(dataset, source, error)
+72 -7
View File
@@ -60,6 +60,27 @@ class DatahubClient:
def auction(self, **params: Any) -> DatahubResponse: def auction(self, **params: Any) -> DatahubResponse:
return self.get("/v1/auction", params) return self.get("/v1/auction", params)
def limit_events(self, **params: Any) -> DatahubResponse:
return self.get("/v1/limit-events", params)
def popularity(self, **params: Any) -> DatahubResponse:
return self.get("/v1/popularity", params)
def dragon_tiger(self, **params: Any) -> DatahubResponse:
return self.get("/v1/dragon-tiger", params)
def sectors(self, **params: Any) -> DatahubResponse:
return self.get("/v1/sectors", params)
def quotes_latest(self, **params: Any) -> DatahubResponse:
return self.get("/v1/quotes/latest", params)
def index_quotes(self, **params: Any) -> DatahubResponse:
return self.get("/v1/indexes/quotes", params)
def intraday_points(self, **params: Any) -> DatahubResponse:
return self.get("/v1/intraday/points", params)
def dataset_status(self, date: str) -> DatahubResponse: def dataset_status(self, date: str) -> DatahubResponse:
return self.get("/v1/datasets/status", {"date": date}) return self.get("/v1/datasets/status", {"date": date})
@@ -69,6 +90,24 @@ class DatahubClient:
params["dataset"] = dataset params["dataset"] = dataset
return self.get("/v1/batches", params) return self.get("/v1/batches", params)
def query_api(self, api_name: str, params: dict[str, Any] | None = None, fields: str = "") -> DatahubResponse:
return self.post(
"/v1/query",
{"api_name": api_name, "params": params or {}, "fields": fields},
)
def sector_quote(self, code: str, date: str = "") -> DatahubResponse:
payload: dict[str, Any] = {"code": code}
if date:
payload["date"] = date
return self.get("/v1/sectors/quote", payload)
def limit_pool(self, trade_date: str = "") -> DatahubResponse:
params: dict[str, Any] = {}
if trade_date:
params["date"] = trade_date
return self.get("/v1/limit-pool", params)
def get(self, path: str, params: dict[str, Any] | None = None) -> DatahubResponse: def get(self, path: str, params: dict[str, Any] | None = None) -> DatahubResponse:
if not self.settings.token: if not self.settings.token:
raise DatahubError("NOT_CONFIGURED", "DATAHUB_TOKEN is not configured") raise DatahubError("NOT_CONFIGURED", "DATAHUB_TOKEN is not configured")
@@ -97,15 +136,41 @@ class DatahubClient:
) )
raise last_error or DatahubError("INTERNAL", "datahub request failed") raise last_error or DatahubError("INTERNAL", "datahub request failed")
def _request(self, url: str) -> DatahubResponse: def post(self, path: str, body: dict[str, Any] | None = None) -> DatahubResponse:
if not self.settings.token:
raise DatahubError("NOT_CONFIGURED", "DATAHUB_TOKEN is not configured")
url = self.settings.base_url + path
attempts = 1 + max(0, self.settings.retries)
last_error: DatahubError | None = None
payload = json.dumps(body or {}, ensure_ascii=False).encode("utf-8")
for attempt in range(attempts):
try:
return self._request(url, method="POST", data=payload)
except DatahubError as exc:
last_error = exc
if exc.code not in {"TIMEOUT", "UNAVAILABLE"} or attempt + 1 >= attempts:
raise
LOGGER.warning(
"datahub retry %s/%s %s",
attempt + 1,
attempts,
redact_text(str(exc), self.settings.secrets()),
)
raise last_error or DatahubError("INTERNAL", "datahub request failed")
def _request(self, url: str, method: str = "GET", data: bytes | None = None) -> DatahubResponse:
headers = {
"Accept": "application/json",
"X-Datahub-Token": self.settings.token,
"User-Agent": "XiaobaiReviewDatahub/1.0",
}
if data is not None:
headers["Content-Type"] = "application/json"
request = urllib.request.Request( request = urllib.request.Request(
url, url,
headers={ data=data,
"Accept": "application/json", headers=headers,
"X-Datahub-Token": self.settings.token, method=method,
"User-Agent": "XiaobaiReviewDatahub/1.0",
},
method="GET",
) )
try: try:
with self._urlopen(request, timeout=self.settings.timeout_seconds) as response: with self._urlopen(request, timeout=self.settings.timeout_seconds) as response:
+140
View File
@@ -0,0 +1,140 @@
from __future__ import annotations
import time
from typing import Any
from backend.data.datahub.bridge import DatahubBridge
from backend.data.datahub.errors import DatahubError
from backend.data.providers.ifind_client import IfindError
class HubIfindProxy:
"""Website-facing iFinD facade. Talks only to xiaobai-datahub."""
def __init__(self, datahub: DatahubBridge) -> None:
self._datahub = datahub
self._status: dict[str, Any] | None = None
self._status_at = 0.0
@property
def configured(self) -> bool:
return bool(self.status().get("configured"))
def status(self) -> dict[str, Any]:
now = time.monotonic()
if self._status is not None and now - self._status_at < 30:
return dict(self._status)
fallback = {"configured": False, "access_ready": False, "access_expires_at": ""}
if not self._datahub.settings.token:
self._status = fallback
self._status_at = now
return dict(fallback)
try:
rows = self._rows("ifind_status", {})
except IfindError:
self._status = fallback
self._status_at = now
return dict(fallback)
row = rows[0] if rows else {}
status = {
"configured": bool(row.get("configured")),
"access_ready": bool(row.get("access_ready")),
"access_expires_at": str(row.get("access_expires_at") or ""),
}
self._status = status
self._status_at = now
return dict(status)
def wencai(self, query: str, search_type: str = "stock", cache_ttl: int = 300) -> list[dict[str, Any]]:
return self._rows(
"ifind_wencai",
{"query": query, "search_type": search_type, "cache_ttl": cache_ttl},
)
def snapshots(
self,
codes: str | list[str],
indicators: list[str],
start_time: str,
end_time: str,
cache_ttl: int = 8,
) -> list[dict[str, Any]]:
return self._rows(
"ifind_snapshots",
{
"codes": codes,
"indicators": indicators,
"start_time": start_time,
"end_time": end_time,
"cache_ttl": cache_ttl,
},
)
def history(
self,
codes: str | list[str],
indicators: list[str],
start_date: str,
end_date: str,
cache_ttl: int = 300,
) -> list[dict[str, Any]]:
return self._rows(
"ifind_history",
{
"codes": codes,
"indicators": indicators,
"start_date": start_date,
"end_date": end_date,
"cache_ttl": cache_ttl,
},
)
def real_time(
self,
codes: str | list[str],
indicators: list[str],
cache_ttl: int = 10,
) -> list[dict[str, Any]]:
return self._rows(
"ifind_realtime",
{"codes": codes, "indicators": indicators, "cache_ttl": cache_ttl},
)
def intraday(
self,
code: str,
start_time: str,
end_time: str,
cache_ttl: int = 20,
) -> list[dict[str, Any]]:
return self._rows(
"ifind_intraday",
{
"code": code,
"start_time": start_time,
"end_time": end_time,
"cache_ttl": cache_ttl,
},
)
def test_connection(self) -> dict[str, Any]:
payload = self.real_time(
"000001.SH",
["open", "high", "low", "latest", "preClose"],
cache_ttl=0,
)
return {
"ok": bool(payload),
"sample_time": str(payload[0].get("time") or "") if payload else "",
}
def _rows(self, api_name: str, params: dict[str, Any]) -> list[dict[str, Any]]:
try:
response = self._datahub.client.query_api(api_name, params)
except DatahubError as exc:
raise IfindError(str(exc) or "iFinD 数据中枢暂不可用") from exc
data = response.data
if isinstance(data, list):
return [dict(item) for item in data if isinstance(item, dict)]
if isinstance(data, dict):
return [dict(data)]
return []
+51
View File
@@ -17,6 +17,13 @@ API_TO_DATASET = {
"index_daily": "index_daily", "index_daily": "index_daily",
"moneyflow": "moneyflow", "moneyflow": "moneyflow",
"stk_auction": "auction", "stk_auction": "auction",
"limit_list_d": "limit_events",
"ths_hot": "popularity",
"dc_hot": "popularity",
"hm_detail": "dragon_tiger",
"ths_daily": "sector_daily",
"dc_index": "sector_daily",
"sw_daily": "sector_daily",
} }
SCALE_FIELDS = { SCALE_FIELDS = {
@@ -35,6 +42,16 @@ SCALE_FIELDS = {
"net_mf_amount": AMOUNT_WAN_YUAN, "net_mf_amount": AMOUNT_WAN_YUAN,
}, },
"auction": {"vol": VOLUME_LOT, "float_share": AMOUNT_WAN_YUAN}, "auction": {"vol": VOLUME_LOT, "float_share": AMOUNT_WAN_YUAN},
"limit_events": {
"limit_amount": AMOUNT_WAN_YUAN,
"float_mv": AMOUNT_WAN_YUAN,
"total_mv": AMOUNT_WAN_YUAN,
},
"dragon_tiger": {
"buy_amount": AMOUNT_WAN_YUAN,
"sell_amount": AMOUNT_WAN_YUAN,
"net_amount": AMOUNT_WAN_YUAN,
},
} }
@@ -67,6 +84,16 @@ def to_native_row(dataset: str, row: dict[str, Any]) -> dict[str, Any]:
converted[field] = _unscale(converted.get(field), factor) converted[field] = _unscale(converted.get(field), factor)
if dataset == "stocks": if dataset == "stocks":
converted.pop("updated_at", None) converted.pop("updated_at", None)
if dataset == "popularity":
# keep hub source; callers filter ths/dc themselves when needed
if converted.get("ts_name") and not converted.get("name"):
converted["name"] = converted.get("ts_name")
if dataset == "dragon_tiger":
if converted.get("ts_name") and not converted.get("name"):
converted["name"] = converted.get("ts_name")
if dataset == "sector_daily":
if converted.get("pct_change") is not None and converted.get("pct_chg") is None:
converted["pct_chg"] = converted.get("pct_change")
return converted return converted
@@ -96,6 +123,30 @@ def row_key(dataset: str, row: dict[str, Any]) -> tuple[str, ...]:
return (str(row.get("ts_code") or "").upper(),) return (str(row.get("ts_code") or "").upper(),)
if dataset == "status": if dataset == "status":
return (str(row.get("dataset") or ""), yyyymmdd(row.get("trade_date"))) return (str(row.get("dataset") or ""), yyyymmdd(row.get("trade_date")))
if dataset == "limit_events":
return (
str(row.get("ts_code") or "").upper(),
yyyymmdd(row.get("trade_date")),
str(row.get("limit_type") or ""),
)
if dataset == "popularity":
return (
str(row.get("ts_code") or "").upper(),
yyyymmdd(row.get("trade_date")),
str(row.get("source") or ""),
)
if dataset == "dragon_tiger":
return (
str(row.get("ts_code") or "").upper(),
yyyymmdd(row.get("trade_date")),
str(row.get("hm_name") or ""),
)
if dataset == "sector_daily":
return (
str(row.get("ts_code") or "").upper(),
yyyymmdd(row.get("trade_date")),
str(row.get("family") or ""),
)
return (str(row.get("ts_code") or "").upper(), yyyymmdd(row.get("trade_date"))) return (str(row.get("ts_code") or "").upper(), yyyymmdd(row.get("trade_date")))
+180
View File
@@ -0,0 +1,180 @@
from __future__ import annotations
from datetime import datetime
from typing import Any
from backend.data.datahub.bridge import DatahubBridge
from backend.data.realtime import RealtimeAggregateError
class HubRealtimeProxy:
"""Realtime observation facade. Talks only to xiaobai-datahub."""
def __init__(self, datahub: DatahubBridge) -> None:
self._datahub = datahub
def health_snapshot(self, sector: str = "") -> dict[str, Any]:
started = datetime.now().astimezone()
indices: list[dict[str, Any]] = []
error = ""
try:
indices = self.tencent_indices()
except RealtimeAggregateError as exc:
error = str(exc)
epochs = [int(item.get("quote_time_epoch") or 0) for item in indices]
now = datetime.now().astimezone()
max_skew = 120 if now.hour >= 15 else 15
index_consistent = bool(epochs) and max(epochs) - min(epochs) <= max_skew
ready = len(indices) == 3 and index_consistent
return {
"ready": ready,
"isolated": True,
"generated_at": started.isoformat(timespec="seconds"),
"elapsed_ms": 0,
"indices": indices,
"index_consistent": index_consistent,
"sector": None,
"sources": {
"datahub_indices": {
"ok": ready,
"error": error,
"source": "datahub",
}
},
"observations": {},
"policy": {
"integration": "datahub_exclusive",
"max_index_time_skew_seconds": max_skew,
"notice": "实时观察只走数据中枢,主网站不再直连东财/腾讯。",
},
}
def tencent_indices(self) -> list[dict[str, Any]]:
rows = self._datahub.try_index_quotes() or []
result = [_as_index(item) for item in rows if _as_index(item)]
wanted = {"000001", "399001", "399006"}
result = [item for item in result if item.get("code") in wanted]
result.sort(key=lambda item: str(item.get("code") or ""))
if len(result) != 3:
raise RealtimeAggregateError(f"datahub returned {len(result)}/3 indices")
return result
def eastmoney_indices(self) -> list[dict[str, Any]]:
return self.tencent_indices()
def tencent_stock_quote(self, code: str, expected_date: str = "") -> dict[str, Any]:
return self._stock_quote(code, expected_date)
def eastmoney_stock_quote(self, code: str, expected_date: str = "") -> dict[str, Any]:
return self._stock_quote(code, expected_date)
def tencent_stock_quotes(
self,
codes: list[str],
expected_date: str = "",
minimum: int | None = None,
) -> list[dict[str, Any]]:
return self._stock_quotes(codes, expected_date, minimum)
def eastmoney_stock_quotes(
self,
codes: list[str],
expected_date: str = "",
) -> list[dict[str, Any]]:
return self._stock_quotes(codes, expected_date, None)
def eastmoney_shenwan_quote(self, ts_code: str, expected_date: str = "") -> dict[str, Any]:
quote = self._datahub.try_sector_quote(ts_code, expected_date)
if not quote:
raise RealtimeAggregateError(f"datahub shenwan quote unavailable for {ts_code}")
return quote
def _stock_quote(self, code: str, expected_date: str) -> dict[str, Any]:
rows = self._stock_quotes([code], expected_date, 1)
if not rows:
raise RealtimeAggregateError(f"datahub stock quote unavailable for {code}")
return rows[0]
def _stock_quotes(
self,
codes: list[str],
expected_date: str,
minimum: int | None,
) -> list[dict[str, Any]]:
cleaned = [str(item or "").strip() for item in codes if str(item or "").strip()]
rows = self._datahub.try_quotes(cleaned) if cleaned else (self._datahub.try_market_quotes(expected_date) or [])
quotes = [_as_stock(item) for item in (rows or []) if _as_stock(item)]
if expected_date:
compact = str(expected_date).replace("-", "")
quotes = [
item
for item in quotes
if not item.get("quote_date") or str(item.get("quote_date") or "").replace("-", "") == compact
]
if minimum is not None and len(quotes) < minimum:
raise RealtimeAggregateError(f"datahub returned {len(quotes)} quotes, need {minimum}")
return quotes
def _as_index(row: dict[str, Any]) -> dict[str, Any] | None:
code = str(row.get("code") or str(row.get("ts_code") or "").split(".")[0] or "")
price = _number(row.get("price") if row.get("price") not in (None, "") else row.get("close"))
if not code or price <= 0:
return None
epoch = int(_number(row.get("quote_time_epoch")))
amount = _number(row.get("amount_billion"))
if amount <= 0:
amount = round(_number(row.get("amount")) / 100_000_000, 2)
return {
"code": code,
"name": row.get("name") or code,
"price": price,
"change": _number(row.get("change") if row.get("change") not in (None, "") else row.get("pct_chg")),
"change_amount": _number(row.get("change_amount")),
"open": _number(row.get("open")),
"high": _number(row.get("high")),
"low": _number(row.get("low")),
"previous_close": _number(
row.get("previous_close") if row.get("previous_close") not in (None, "") else row.get("pre_close")
),
"amount_billion": amount,
"quote_time_epoch": epoch,
"quote_time": str(row.get("quote_time") or ""),
"source": str(row.get("source") or "datahub"),
"cache_age_seconds": 0,
}
def _as_stock(row: dict[str, Any]) -> dict[str, Any] | None:
close = _number(row.get("close") if row.get("close") not in (None, "") else row.get("price"))
if close <= 0:
return None
ts_code = str(row.get("ts_code") or "")
code = str(row.get("code") or ts_code.split(".")[0] or "")
return {
"ts_code": ts_code or code,
"code": code,
"name": row.get("name") or "",
"close": close,
"pre_close": _number(
row.get("pre_close") if row.get("pre_close") not in (None, "") else row.get("previous_close")
),
"open": _number(row.get("open")),
"high": _number(row.get("high")),
"low": _number(row.get("low")),
"volume": _number(row.get("volume") if row.get("volume") not in (None, "") else row.get("vol")),
"vol": _number(row.get("vol") if row.get("vol") not in (None, "") else row.get("volume")),
"amount": _number(row.get("amount")),
"quote_time_epoch": int(_number(row.get("quote_time_epoch"))),
"quote_time": str(row.get("quote_time") or ""),
"quote_date": str(row.get("quote_date") or ""),
"source": str(row.get("source") or "datahub"),
"delayed": bool(row.get("delayed")),
}
def _number(value: Any) -> float:
try:
return float(value or 0)
except (TypeError, ValueError):
return 0.0
+57
View File
@@ -0,0 +1,57 @@
from __future__ import annotations
from datetime import datetime
from threading import Lock
from typing import Any
from backend.data.datahub.settings import DATASETS
DATASET_LABELS = {
"calendar": "交易日历",
"stocks": "股票主档",
"daily": "个股日K",
"index_daily": "指数日K",
"valuation": "估值",
"moneyflow": "资金流",
"auction": "竞价",
"limit_events": "涨停池",
"popularity": "人气榜",
"dragon_tiger": "龙虎榜",
"sector_daily": "题材板块",
"quotes": "全市场实时行情",
"index_quotes": "指数实时行情",
"intraday": "分时",
"status": "数据集状态",
}
class DatahubRouteLedger:
def __init__(self) -> None:
self._lock = Lock()
self._rows: dict[str, dict[str, Any]] = {}
def record(self, dataset: str, route: str, source: str = "", error: str = "") -> None:
name = str(dataset or "").strip() or "unknown"
with self._lock:
self._rows[name] = {
"dataset": name,
"label": DATASET_LABELS.get(name, name),
"route": "legacy" if route == "legacy" else "datahub",
"source": str(source or "").strip(),
"error": str(error or "").strip(),
"at": datetime.now().astimezone().isoformat(timespec="seconds"),
}
def snapshot(self) -> list[dict[str, Any]]:
with self._lock:
rows = [dict(item) for item in self._rows.values()]
order = {name: index for index, name in enumerate(DATASETS)}
rows.sort(key=lambda item: (order.get(str(item.get("dataset")), 99), str(item.get("dataset"))))
return rows
def clear(self) -> None:
with self._lock:
self._rows.clear()
LEDGER = DatahubRouteLedger()
+14
View File
@@ -17,6 +17,13 @@ DATASETS = (
"valuation", "valuation",
"moneyflow", "moneyflow",
"auction", "auction",
"limit_events",
"popularity",
"dragon_tiger",
"sector_daily",
"quotes",
"index_quotes",
"intraday",
"status", "status",
) )
@@ -28,6 +35,13 @@ ENV_DATASET = {
"valuation": "VALUATION", "valuation": "VALUATION",
"moneyflow": "MONEYFLOW", "moneyflow": "MONEYFLOW",
"auction": "AUCTION", "auction": "AUCTION",
"limit_events": "LIMIT_EVENTS",
"popularity": "POPULARITY",
"dragon_tiger": "DRAGON_TIGER",
"sector_daily": "SECTOR_DAILY",
"quotes": "QUOTES",
"index_quotes": "INDEX_QUOTES",
"intraday": "INTRADAY",
"status": "STATUS", "status": "STATUS",
} }
+38 -23
View File
@@ -1,43 +1,40 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from typing import Any from typing import Any
from backend.data.contracts import DataUsage from backend.data.contracts import DataUsage
from backend.data.datahub import DatahubAwareTushareClient, DatahubBridge, DatahubClient, DatahubSettings from backend.data.datahub import DatahubAwareTushareClient, DatahubBridge, DatahubClient, DatahubSettings
from backend.data.datahub.ifind_proxy import HubIfindProxy
from backend.data.datahub.realtime_proxy import HubRealtimeProxy
from backend.data.policy import DataSourcePolicy from backend.data.policy import DataSourcePolicy
from backend.data.providers import IfindProvider, TushareProvider from backend.data.providers import IfindProvider
from backend.data.quality import DataQualityGate, QualityEvidence, QualityReport from backend.data.quality import DataQualityGate, QualityEvidence, QualityReport
from backend.data.providers.ifind_client import IfindHttpClient from backend.features.market.charts import MarketChartClient
from backend.data.providers.tushare_client import TushareClient
from backend.data.realtime import WebRealtimeAggregator
from backend.features.market.charts import EastmoneyChartClient, MarketChartClient
@dataclass(frozen=True) @dataclass(frozen=True)
class DataGateway: class DataGateway:
policy: DataSourcePolicy policy: DataSourcePolicy
quality: DataQualityGate quality: DataQualityGate
tushare_provider: TushareProvider
ifind_provider: IfindProvider ifind_provider: IfindProvider
chart_data: MarketChartClient chart_data: MarketChartClient
realtime_observer: WebRealtimeAggregator realtime_observer: HubRealtimeProxy
datahub: DatahubBridge datahub: DatahubBridge
@property @property
def ifind(self) -> IfindHttpClient: def ifind(self) -> HubIfindProxy:
return self.ifind_provider.client return self.ifind_provider.client
def tushare( def tushare(
self, self,
dataset_id: str = "", dataset_id: str = "",
usage: DataUsage = "calculation", usage: DataUsage = "calculation",
) -> TushareClient: ) -> DatahubAwareTushareClient:
if dataset_id: if dataset_id:
self.policy.assert_allowed(dataset_id, "tushare", usage) self.policy.assert_allowed(dataset_id, "tushare", usage)
return DatahubAwareTushareClient(self.tushare_provider.client(), self.datahub) return DatahubAwareTushareClient(self.datahub)
def dataset_status(self, trade_date: str) -> list[dict[str, Any]] | None: def dataset_status(self, trade_date: str) -> list[dict[str, Any]] | None:
return self.datahub.dataset_status(trade_date) return self.datahub.dataset_status(trade_date)
@@ -45,6 +42,31 @@ class DataGateway:
def batches(self, trade_date: str, dataset: str = "") -> list[dict[str, Any]] | None: def batches(self, trade_date: str, dataset: str = "") -> list[dict[str, Any]] | None:
return self.datahub.batches(trade_date, dataset) return self.datahub.batches(trade_date, dataset)
def datahub_status(self) -> dict[str, Any]:
from backend.data.datahub.route_state import DATASET_LABELS, LEDGER
from backend.data.datahub.settings import DATASETS
settings = self.datahub.settings
flags = []
enabled = 0
for name in DATASETS:
read = bool(settings.flags(name).read)
if read:
enabled += 1
flags.append({"dataset": name, "label": DATASET_LABELS.get(name, name), "read": read})
routes = LEDGER.snapshot()
fallbacks = [item for item in routes if item.get("route") == "legacy"]
return {
"configured": bool(settings.token and settings.base_url),
"base_url": settings.base_url,
"enabled_reads": enabled,
"total_reads": len(DATASETS),
"flags": flags,
"routes": routes,
"fallback_count": len(fallbacks),
"fallback_labels": [str(item.get("label") or item.get("dataset")) for item in fallbacks],
}
def assert_source(self, dataset_id: str, provider_id: str, usage: DataUsage) -> None: def assert_source(self, dataset_id: str, provider_id: str, usage: DataUsage) -> None:
self.policy.assert_allowed(dataset_id, provider_id, usage) self.policy.assert_allowed(dataset_id, provider_id, usage)
@@ -72,25 +94,18 @@ class DataGateway:
def build_data_gateway( def build_data_gateway(
credentials: dict[str, object], credentials: dict[str, object],
tushare_token_supplier: Callable[[], str] | None = None,
datahub_settings: DatahubSettings | None = None, datahub_settings: DatahubSettings | None = None,
) -> DataGateway: ) -> DataGateway:
ifind = IfindHttpClient(
str(credentials.get("ifind_refresh_token") or ""),
str(credentials.get("ifind_access_token") or ""),
)
token_supplier = tushare_token_supplier or (
lambda: str(credentials.get("tushare_token") or "")
)
policy = DataSourcePolicy.load() policy = DataSourcePolicy.load()
settings = datahub_settings or DatahubSettings.load(credentials=credentials) settings = datahub_settings or DatahubSettings.load(credentials=credentials)
datahub_client = DatahubClient(settings) datahub_client = DatahubClient(settings)
datahub = DatahubBridge(settings, datahub_client)
ifind = HubIfindProxy(datahub)
return DataGateway( return DataGateway(
policy=policy, policy=policy,
quality=DataQualityGate.load(policy), quality=DataQualityGate.load(policy),
tushare_provider=TushareProvider(token_supplier),
ifind_provider=IfindProvider(ifind), ifind_provider=IfindProvider(ifind),
chart_data=MarketChartClient(ifind, EastmoneyChartClient()), chart_data=MarketChartClient(datahub),
realtime_observer=WebRealtimeAggregator(), realtime_observer=HubRealtimeProxy(datahub),
datahub=DatahubBridge(settings, datahub_client), datahub=datahub,
) )
+5 -3
View File
@@ -1,11 +1,13 @@
from __future__ import annotations from __future__ import annotations
from backend.data.providers.ifind_client import IfindHttpClient from typing import Any
class IfindProvider: class IfindProvider:
def __init__(self, client: IfindHttpClient) -> None: def __init__(self, client: Any) -> None:
self.client = client self.client = client
def set_credentials(self, refresh_token: str, access_token: str = "") -> None: def set_credentials(self, refresh_token: str, access_token: str = "") -> None:
self.client.set_credentials(refresh_token, access_token) setter = getattr(self.client, "set_credentials", None)
if callable(setter):
setter(refresh_token, access_token)
+2
View File
@@ -66,3 +66,5 @@ class TushareClient(
_stock_listing_lock: ClassVar[Lock] = Lock() _stock_listing_lock: ClassVar[Lock] = Lock()
_suspension_cache: ClassVar[dict[str, dict[str, str] | None]] = {} _suspension_cache: ClassVar[dict[str, dict[str, str] | None]] = {}
_suspension_lock: ClassVar[Lock] = Lock() _suspension_lock: ClassVar[Lock] = Lock()
_sw_member_cache: ClassVar[dict[str, Any]] = {}
_sw_member_lock: ClassVar[Lock] = Lock()
+76 -4
View File
@@ -3,7 +3,12 @@ from __future__ import annotations
from typing import Any from typing import Any
from backend.data.numbers import finite_number as _number from backend.data.numbers import finite_number as _number
from backend.data.providers.tushare_helpers import _display_time, _prices_equal from backend.data.providers.tushare_helpers import (
_display_time,
_optional_number,
_prices_equal,
calendar_is_open,
)
class DailyMarketMixin: class DailyMarketMixin:
@@ -17,7 +22,11 @@ class DailyMarketMixin:
trade_date = requested trade_date = requested
else: else:
row = requested_rows[0] row = requested_rows[0]
trade_date = row["cal_date"] if row.get("is_open") == 1 else row.get("pretrade_date", requested) trade_date = (
row["cal_date"]
if calendar_is_open(row.get("is_open"))
else row.get("pretrade_date", requested)
)
resolved_rows = self.query( resolved_rows = self.query(
"trade_cal", "trade_cal",
@@ -129,7 +138,66 @@ class DailyMarketMixin:
) )
item["capital_trade_date"] = str(capital.get("trade_date") or "") item["capital_trade_date"] = str(capital.get("trade_date") or "")
result.append(item) result.append(item)
return result return self._overlay_board_fields(result, trade_date)
def _overlay_board_fields(
self,
rows: list[dict[str, Any]],
trade_date: str,
) -> list[dict[str, Any]]:
if not rows:
return rows
official = self._official_board_map(trade_date)
free = self._free_board_map(trade_date) if not official else {}
merged: list[dict[str, Any]] = []
for row in rows:
code = str(row.get("ts_code") or "")
extra = official.get(code) or free.get(code) or {}
if not extra:
merged.append(row)
continue
item = dict(row)
for key in (
"first_time",
"last_time",
"fd_amount",
"open_times",
"limit_times",
"turnover_ratio",
):
incoming = extra.get(key)
current = item.get(key)
if incoming in (None, "", "--"):
continue
if current in (None, "", "--", 0, 0.0):
item[key] = incoming
merged.append(item)
return merged
def _official_board_map(self, trade_date: str) -> dict[str, dict[str, Any]]:
mapped: dict[str, dict[str, Any]] = {}
try:
for row in self._load_limit_lists(trade_date):
code = str(row.get("ts_code") or "")
if code:
mapped[code] = row
except Exception:
return {}
return mapped
def _free_board_map(self, trade_date: str) -> dict[str, dict[str, Any]]:
loader = getattr(self, "try_limit_pool", None)
if not callable(loader):
return {}
try:
rows = loader(trade_date) or []
except Exception:
return {}
return {
str(row.get("ts_code") or ""): row
for row in rows
if row.get("ts_code")
}
@staticmethod @staticmethod
def _normalize_limit(row: dict[str, Any], status: str) -> dict[str, Any]: def _normalize_limit(row: dict[str, Any], status: str) -> dict[str, Any]:
@@ -154,7 +222,11 @@ class DailyMarketMixin:
"turnover_source": row.get("turnover_source") or "provider", "turnover_source": row.get("turnover_source") or "provider",
"capital_trade_date": row.get("capital_trade_date") or "", "capital_trade_date": row.get("capital_trade_date") or "",
"amount_billion": round(amount_billion, 2), "amount_billion": round(amount_billion, 2),
"seal_amount_million": round(_number(row.get("fd_amount")) / 10000, 0), "seal_amount_million": (
round(fd / 10000, 0)
if (fd := _optional_number(row.get("fd_amount"))) is not None
else None
),
"float_mv_billion": round(_number(row.get("float_mv")) / 100000000, 1), "float_mv_billion": round(_number(row.get("float_mv")) / 100000000, 1),
"status": status, "status": status,
} }
+139 -17
View File
@@ -16,6 +16,12 @@ from backend.data.providers.tushare_transport import TushareError
class DashboardMixin: class DashboardMixin:
def _now(self) -> datetime:
clock = getattr(self, "clock", None)
if callable(clock):
return clock()
return datetime.now().astimezone()
def dashboard(self, requested_date: str) -> dict[str, Any]: def dashboard(self, requested_date: str) -> dict[str, Any]:
trade_date, previous_trade_date = self.resolve_trade_context(requested_date) trade_date, previous_trade_date = self.resolve_trade_context(requested_date)
if self.should_use_realtime(requested_date, trade_date): if self.should_use_realtime(requested_date, trade_date):
@@ -26,11 +32,12 @@ class DashboardMixin:
) )
daily = self._load_daily(trade_date) daily = self._load_daily(trade_date)
now = self._now()
if ( if (
not daily not daily
and requested_date == datetime.now().astimezone().strftime("%Y%m%d") and requested_date == now.strftime("%Y%m%d")
and trade_date == requested_date and trade_date == requested_date
and datetime.now().astimezone().time().replace(tzinfo=None) >= dt_time(9, 15) and now.time().replace(tzinfo=None) >= dt_time(9, 15)
): ):
return self._realtime_dashboard( return self._realtime_dashboard(
requested_date, requested_date,
@@ -98,15 +105,14 @@ class DashboardMixin:
} }
return apply_sentiment_to_dashboard(dashboard) return apply_sentiment_to_dashboard(dashboard)
@staticmethod def should_use_realtime(self, requested_date: str, trade_date: str) -> bool:
def should_use_realtime(requested_date: str, trade_date: str) -> bool: """Use live quotes for today's open session until official daily settles."""
"""Use rt_k for today's open market until end-of-day datasets settle.""" now = self._now()
now = datetime.now().astimezone()
today = now.strftime("%Y%m%d") today = now.strftime("%Y%m%d")
return ( return (
requested_date == today requested_date == today
and trade_date == today and trade_date == today
and dt_time(9, 15) <= now.time().replace(tzinfo=None) < dt_time(16, 30) and dt_time(9, 15) <= now.time().replace(tzinfo=None) < dt_time(15, 5)
) )
def _realtime_dashboard( def _realtime_dashboard(
@@ -122,7 +128,7 @@ class DashboardMixin:
) )
if not codes: if not codes:
raise TushareError("No active stock codes available for rt_k") raise TushareError("No active stock codes available for rt_k")
quotes = self.query("rt_k", {"ts_code": codes}) quotes, quote_source = self._load_realtime_quotes(codes, trade_date)
if not quotes: if not quotes:
raise TushareError(f"No realtime data returned for {trade_date}") raise TushareError(f"No realtime data returned for {trade_date}")
@@ -178,14 +184,35 @@ class DashboardMixin:
) )
sectors = _build_sectors(limits) sectors = _build_sectors(limits)
previous_sectors = _build_sectors(previous_limits) previous_sectors = _build_sectors(previous_limits)
now = datetime.now().astimezone() now = self._now()
market_status = _realtime_market_status(now.time().replace(tzinfo=None)) market_status = _realtime_market_status(now.time().replace(tzinfo=None))
if quote_source == "datahub":
notice = (
"盘中行情由数据中枢统一提供;涨停原因、封板时间和开板次数以盘后榜单校正为准。"
)
source_name = "datahub"
elif quote_source == "eastmoney_clist":
notice = (
"盘中行情由东财免费实时快照计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。"
)
source_name = "eastmoney"
elif quote_source == "tencent_qt":
notice = (
"盘中行情由腾讯免费实时行情计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。"
)
source_name = "tencent"
else:
notice = (
"盘中行情由 Tushare rt_k 实时计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。"
)
source_name = "tushare"
dashboard = { dashboard = {
"meta": { "meta": {
"requested_date": _display_date(requested_date), "requested_date": _display_date(requested_date),
"trade_date": _display_date(trade_date), "trade_date": _display_date(trade_date),
"previous_trade_date": _display_date(previous_trade_date), "previous_trade_date": _display_date(previous_trade_date),
"source": "tushare", "source": source_name,
"quote_source": quote_source,
"mode": "realtime", "mode": "realtime",
"realtime": True, "realtime": True,
"market_status": market_status, "market_status": market_status,
@@ -193,7 +220,8 @@ class DashboardMixin:
"auto_refresh": False, "auto_refresh": False,
"quote_count": len(daily), "quote_count": len(daily),
"updated_at": now.isoformat(timespec="seconds"), "updated_at": now.isoformat(timespec="seconds"),
"notice": "盘中行情由 Tushare rt_k 实时计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。", "notice": notice,
"indices": self._free_realtime_indices() if quote_source != "tushare_rt_k" else [],
}, },
"overview": _build_overview(daily, up_rows, down_rows, broken_rows), "overview": _build_overview(daily, up_rows, down_rows, broken_rows),
"limits": limits, "limits": limits,
@@ -207,6 +235,62 @@ class DashboardMixin:
} }
return apply_sentiment_to_dashboard(dashboard) return apply_sentiment_to_dashboard(dashboard)
def _realtime_aggregator(self):
aggregator = getattr(self, "realtime_aggregator", None)
if aggregator is None:
raise TushareError("免费实时源未配置")
return aggregator
def _load_realtime_quotes(
self,
codes: str,
trade_date: str,
) -> tuple[list[dict[str, Any]], str]:
hub = getattr(self, "try_market_quotes", None)
if callable(hub):
quotes = hub(trade_date)
if quotes:
return list(quotes), "datahub"
named = getattr(self, "try_quotes", None)
code_list = [item for item in str(codes or "").split(",") if item]
if callable(named) and code_list:
collected: list[dict[str, Any]] = []
for index in range(0, len(code_list), 60):
collected.extend(named(code_list[index:index + 60]) or [])
if collected:
delayed = any(item.get("delayed") for item in collected)
return collected, "datahub_delayed" if delayed else "datahub"
try:
quotes = self.query("rt_k", {"ts_code": codes})
if quotes:
delayed = any(item.get("delayed") for item in quotes)
return list(quotes), "datahub_delayed" if delayed else "datahub"
except TushareError as exc:
raise TushareError(f"当天盘中实时行情不可用:{exc}") from exc
raise TushareError("当天盘中实时行情不可用:数据中枢未返回可用行情")
def _mark_quote_legacy(self, source: str, error: str = "") -> None:
marker = getattr(self, "record_datahub_legacy", None)
if callable(marker):
marker("quotes", source, error)
def _free_realtime_quotes(
self,
trade_date: str,
codes: str = "",
) -> tuple[list[dict[str, Any]], str]:
del trade_date, codes
raise TushareError("主网站不再直连免费行情源,请走数据中枢")
def _free_realtime_indices(self) -> list[dict[str, Any]]:
hub = getattr(self, "try_index_quotes", None)
if callable(hub):
rows = hub()
converted = [item for item in (_hub_index_quote(row) for row in rows or []) if item]
if converted:
return converted
return []
def _load_realtime_reference( def _load_realtime_reference(
self, self,
trade_date: str, trade_date: str,
@@ -234,7 +318,7 @@ class DashboardMixin:
{"trade_date": previous_trade_date}, {"trade_date": previous_trade_date},
"ts_code,trade_date,total_share,float_share,free_share,total_mv,circ_mv", "ts_code,trade_date,total_share,float_share,free_share,total_mv,circ_mv",
) )
if not basic_rows or not price_limits: if not basic_rows:
raise TushareError(f"Realtime reference data is incomplete for {trade_date}") raise TushareError(f"Realtime reference data is incomplete for {trade_date}")
result = { result = {
"basic_rows": basic_rows, "basic_rows": basic_rows,
@@ -254,10 +338,9 @@ class DashboardMixin:
ts_code: str, ts_code: str,
reference_date: str = "", reference_date: str = "",
) -> dict[str, Any]: ) -> dict[str, Any]:
rows = self.query("rt_k", {"ts_code": ts_code}) row = self._realtime_quote_row(ts_code, reference_date)
if not rows: if not row:
raise TushareError(f"No realtime quote returned for {ts_code}") raise TushareError(f"No realtime quote returned for {ts_code}")
row = rows[0]
close = _number(row.get("close")) close = _number(row.get("close"))
previous_close = _number(row.get("pre_close")) previous_close = _number(row.get("pre_close"))
if close <= 0 or previous_close <= 0: if close <= 0 or previous_close <= 0:
@@ -341,10 +424,24 @@ class DashboardMixin:
"float_share_10k": float_share, "float_share_10k": float_share,
"capital_trade_date": str(capital.get("trade_date") or ""), "capital_trade_date": str(capital.get("trade_date") or ""),
"turnover_source": "rt_volume/latest_float_share" if float_share else "unavailable", "turnover_source": "rt_volume/latest_float_share" if float_share else "unavailable",
"data_source": "tushare", "data_source": str(row.get("source") or "tushare"),
"realtime": True, "realtime": True,
} }
def _realtime_quote_row(self, ts_code: str, reference_date: str = "") -> dict[str, Any]:
hub = getattr(self, "try_quotes", None)
if callable(hub):
rows = hub([ts_code]) or []
if rows:
return dict(rows[0])
try:
rows = self.query("rt_k", {"ts_code": ts_code})
if rows:
return dict(rows[0])
except TushareError:
pass
return {}
def _stock_activity_metrics( def _stock_activity_metrics(
self, self,
ts_code: str, ts_code: str,
@@ -462,7 +559,7 @@ class DashboardMixin:
for row in reference.get("basic_rows") or [] for row in reference.get("basic_rows") or []
if row.get("ts_code") if row.get("ts_code")
] ]
quotes = self.query("rt_k", {"ts_code": ",".join(codes)}, "") quotes, quote_source = self._load_realtime_quotes(",".join(codes), trade_date)
rows = [ rows = [
row for row in quotes row for row in quotes
if _number(row.get("close")) > 0 and _number(row.get("pre_close")) > 0 if _number(row.get("close")) > 0 and _number(row.get("pre_close")) > 0
@@ -608,6 +705,31 @@ def _build_yesterday_performance(
return result return result
def _hub_index_quote(row: dict[str, Any]) -> dict[str, Any] | None:
ts_code = str(row.get("ts_code") or "")
code = str(row.get("code") or ts_code.split(".")[0])
close = _number(row.get("price") if row.get("price") not in (None, "") else row.get("close"))
previous = _number(
row.get("previous_close") if row.get("previous_close") not in (None, "") else row.get("pre_close")
)
if close <= 0 or previous <= 0:
return None
amount = _number(row.get("amount"))
amount_billion = _number(row.get("amount_billion"))
if not amount_billion and amount:
amount_billion = round(amount / 100_000_000, 2)
return {
"code": code,
"name": str(row.get("name") or code),
"price": close,
"change": _number(row.get("pct_chg") if row.get("pct_chg") not in (None, "") else row.get("change")),
"previous_close": previous,
"amount_billion": amount_billion,
"quote_time": str(row.get("quote_time") or ""),
"source": "datahub",
}
def _build_limit_performance(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: def _build_limit_performance(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
result = [] result = []
for level in sorted({int(row.get("prior_streak") or 1) for row in rows}, reverse=True): for level in sorted({int(row.get("prior_streak") or 1) for row in rows}, reverse=True):
+60
View File
@@ -6,12 +6,72 @@ from typing import Any
from backend.data.numbers import finite_number as _number from backend.data.numbers import finite_number as _number
def calendar_is_open(value: Any) -> bool:
if value in (True, 1, "1", "Y", "y"):
return True
if value in (False, 0, "0", "N", "n", None, ""):
return False
try:
return int(value) == 1
except (TypeError, ValueError):
return False
def _text(value: Any) -> str: def _text(value: Any) -> str:
if isinstance(value, (list, tuple, set)): if isinstance(value, (list, tuple, set)):
return "".join(str(item).strip() for item in value if str(item).strip()) return "".join(str(item).strip() for item in value if str(item).strip())
return str(value or "").strip() return str(value or "").strip()
def _optional_number(value: Any) -> float | None:
if value in (None, "", "-"):
return None
number = _number(value, default=float("nan"))
if number != number:
return None
return number
def _moneyflow_payload(flow: dict[str, Any] | None) -> dict[str, Any]:
if not flow:
return {
"available": False,
"net_million": None,
"large_million": None,
"medium_million": None,
"small_million": None,
}
net = _optional_number(flow.get("net_mf_amount"))
buy_lg = _optional_number(flow.get("buy_lg_amount"))
sell_lg = _optional_number(flow.get("sell_lg_amount"))
buy_elg = _optional_number(flow.get("buy_elg_amount"))
sell_elg = _optional_number(flow.get("sell_elg_amount"))
buy_md = _optional_number(flow.get("buy_md_amount"))
sell_md = _optional_number(flow.get("sell_md_amount"))
buy_sm = _optional_number(flow.get("buy_sm_amount"))
sell_sm = _optional_number(flow.get("sell_sm_amount"))
large = None
if None not in (buy_lg, sell_lg, buy_elg, sell_elg):
large = (buy_lg + buy_elg - sell_lg - sell_elg)
elif _optional_number(flow.get("large_amount")) is not None:
large = _optional_number(flow.get("large_amount"))
medium = None if None in (buy_md, sell_md) else (buy_md - sell_md)
if medium is None:
medium = _optional_number(flow.get("medium_amount"))
small = None if None in (buy_sm, sell_sm) else (buy_sm - sell_sm)
if small is None:
small = _optional_number(flow.get("small_amount"))
if net is None and large is None and medium is None and small is None:
return _moneyflow_payload(None)
return {
"available": True,
"net_million": None if net is None else round(net / 100, 2),
"large_million": None if large is None else round(large / 100, 2),
"medium_million": None if medium is None else round(medium / 100, 2),
"small_million": None if small is None else round(small / 100, 2),
}
def _prices_equal(left: Any, right: Any) -> bool: def _prices_equal(left: Any, right: Any) -> bool:
if left is None or right is None: if left is None or right is None:
return False return False
+71
View File
@@ -59,6 +59,73 @@ class IndexMixin:
} }
def realtime_market_indices(self, requested_date: str) -> dict[str, Any]: def realtime_market_indices(self, requested_date: str) -> dict[str, Any]:
hub = getattr(self, "try_index_quotes", None)
if callable(hub):
rows = hub()
if rows:
return self._hub_realtime_market_indices(requested_date, rows)
raise TushareError("Realtime index quotes are incomplete")
def _hub_realtime_market_indices(
self,
requested_date: str,
rows: list[dict[str, Any]],
) -> dict[str, Any]:
trade_date, _ = self.resolve_trade_context(requested_date)
index_names = {
"000001.SH": "上证指数",
"399001.SZ": "深证成指",
"399006.SZ": "创业板指",
}
by_code = {str(row.get("ts_code") or ""): row for row in rows}
by_symbol = {str(row.get("code") or ""): row for row in rows}
indices = []
for ts_code, name in index_names.items():
row = by_code.get(ts_code) or by_symbol.get(ts_code.split(".")[0])
if not row:
continue
close = _number(row.get("price") if row.get("price") not in (None, "") else row.get("close"))
previous_close = _number(
row.get("previous_close") if row.get("previous_close") not in (None, "") else row.get("pre_close")
)
if close <= 0 or previous_close <= 0:
continue
amount = _number(row.get("amount"))
amount_billion = _number(row.get("amount_billion"))
if not amount_billion and amount:
amount_billion = round(amount / 100_000_000, 2)
indices.append(
{
"ts_code": ts_code,
"name": str(row.get("name") or name).strip(),
"trade_date": trade_date,
"close": close,
"pct_chg": round(
_number(row.get("pct_chg")) or (close / previous_close - 1) * 100,
3,
),
"return_5d": 0,
"amount_billion": amount_billion,
"quote_time": str(row.get("quote_time") or ""),
"source": "datahub",
}
)
if len(indices) != 3:
raise TushareError("Realtime index quotes are incomplete")
return {
"trade_date": trade_date,
"source": "datahub",
"realtime": True,
"precise": True,
"indices": indices,
"aggregate": {
"average_pct_chg": round(sum(item["pct_chg"] for item in indices) / len(indices), 3),
"average_return_5d": 0,
"average_return_20d": 0,
},
}
def _tushare_realtime_market_indices(self, requested_date: str) -> dict[str, Any]:
trade_date, _ = self.resolve_trade_context(requested_date) trade_date, _ = self.resolve_trade_context(requested_date)
index_names = { index_names = {
"000001.SH": "上证指数", "000001.SH": "上证指数",
@@ -116,3 +183,7 @@ class IndexMixin:
"average_return_20d": 0, "average_return_20d": 0,
}, },
} }
def _free_realtime_market_indices(self, requested_date: str) -> dict[str, Any]:
del requested_date
raise TushareError("主网站不再直连免费行情源,请走数据中枢")
+318 -75
View File
@@ -1,11 +1,16 @@
from __future__ import annotations from __future__ import annotations
import json
from datetime import datetime, timedelta from datetime import datetime, timedelta
from pathlib import Path
from typing import Any from typing import Any
from backend.data.numbers import finite_number as _number from backend.data.numbers import finite_number as _number
from backend.data.providers.tushare_transport import TushareError from backend.data.providers.tushare_transport import TushareError
_SW_MEMBER_TTL = timedelta(hours=24)
_SW_MEMBER_DIR = Path(__file__).resolve().parents[3] / "data" / "cache" / "sw_members"
class ShenwanIndustryMixin: class ShenwanIndustryMixin:
def sw_stock_industry(self, ts_code: str, trade_date: str) -> dict[str, Any]: def sw_stock_industry(self, ts_code: str, trade_date: str) -> dict[str, Any]:
@@ -132,21 +137,63 @@ class ShenwanIndustryMixin:
actual_trade_date = str(daily.get("trade_date") or "") actual_trade_date = str(daily.get("trade_date") or "")
outer_precise = actual_trade_date == trade_date outer_precise = actual_trade_date == trade_date
outer_error = "" if outer_precise else ( outer_error = "" if outer_precise else (
f"No Shenwan daily returned for {sector_code} on {trade_date}" f"申万行业 {sector_code} 当日盘后正式数据尚未入库"
) )
outer_source = "tushare_sw_daily" if outer_precise else "unavailable"
if not outer_precise and allow_realtime_close: if not outer_precise and allow_realtime_close:
try: inner_ok = bool(member_rows) and not coverage_issue
return self._sw_realtime_sector_snapshot( if inner_ok:
industry, sw_row, rt_source, rt_error = self._sw_outer_realtime(
members, sector_code,
str(industry.get("l2_name") or ""),
trade_date, trade_date,
previous_trade_date,
finalized=True, finalized=True,
) )
except TushareError as exc: if sw_row:
outer_error = f"{outer_error}; realtime close fallback failed: {exc}" daily = sw_row
actual_trade_date = str(
sw_row.get("quote_date") or sw_row.get("trade_date") or ""
)
trade_time = str(sw_row.get("trade_time") or sw_row.get("quote_time") or "")
quote_clock = (
trade_time[11:19]
if len(trade_time) >= 19
else str(sw_row.get("quote_clock") or "")
)
outer_precise = actual_trade_date == trade_date
if quote_clock and quote_clock < "15:00:00":
outer_precise = False
outer_source = rt_source or "eastmoney_sw"
outer_error = "" if outer_precise else (
rt_error or f"申万行业 {sector_code} 免费实时尚未形成收盘快照"
)
else:
outer_error = rt_error or outer_error
else:
try:
snapshot = self._sw_realtime_sector_snapshot(
industry,
members,
trade_date,
previous_trade_date,
finalized=True,
)
snapshot.update({
"raw_member_count": raw_member_count,
"excluded_member_count": len(excluded_members),
"excluded_members": excluded_members,
})
return snapshot
except TushareError:
outer_error = f"{outer_error}; 免费实时成分暂不可用"
official_change = _number(daily.get("pct_change")) if outer_precise else None official_change = None
if outer_precise:
official_change = _number(
daily.get("pct_change")
if daily.get("pct_change") not in (None, "")
else daily.get("change")
)
return { return {
"code": sector_code, "code": sector_code,
"name": industry.get("l2_name") or daily.get("name") or sector_code, "name": industry.get("l2_name") or daily.get("name") or sector_code,
@@ -173,9 +220,9 @@ class ShenwanIndustryMixin:
"amount_billion": round(amount_billion, 2), "amount_billion": round(amount_billion, 2),
"count": 0, "count": 0,
"max_streak": 0, "max_streak": 0,
"source": "tushare_sw_daily+member_daily" if outer_precise else "tushare_member_daily", "source": f"{outer_source}+tushare_member_daily" if outer_precise else "tushare_member_daily",
"inner_source": "tushare_member_daily", "inner_source": "tushare_member_daily",
"outer_source": "tushare_sw_daily" if outer_precise else "unavailable", "outer_source": outer_source,
"taxonomy": "sw_l2", "taxonomy": "sw_l2",
"industry": industry, "industry": industry,
"trade_date": trade_date, "trade_date": trade_date,
@@ -189,7 +236,7 @@ class ShenwanIndustryMixin:
"inner_error": inner_error, "inner_error": inner_error,
"outer_error": outer_error, "outer_error": outer_error,
"schema_version": 6, "schema_version": 6,
"methodology": "外显使用申万二级行业官方日线;内核独立使用当日成分日线宽度与等权涨跌聚合", "methodology": "外显使用已发布 sw_daily 或免费申万实时;内核优先使用当日成分日线,不调用 rt_sw_k",
} }
def _sw_sector_members( def _sw_sector_members(
@@ -197,23 +244,100 @@ class ShenwanIndustryMixin:
sector_code: str, sector_code: str,
trade_date: str, trade_date: str,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
rows = [] cached_rows = self._read_local_sw_members(sector_code)
for is_new in ("Y", "N"): if cached_rows is not None:
rows.extend( return _active_members(cached_rows, trade_date)
self.query( rows: list[dict[str, Any]] = []
"index_member_all", try:
{"l2_code": sector_code, "is_new": is_new}, for is_new in ("Y", "N"):
"l2_code,l2_name,ts_code,name,in_date,out_date,is_new", rows.extend(
self.query(
"index_member_all",
{"l2_code": sector_code, "is_new": is_new},
"l2_code,l2_name,ts_code,name,in_date,out_date,is_new",
)
) )
except TushareError:
stale = self._read_local_sw_members(sector_code, allow_stale=True) or []
if stale:
return _active_members(stale, trade_date)
raise
reconciled = _reconcile_membership_rows(rows)
self._write_local_sw_members(sector_code, reconciled)
return _active_members(reconciled, trade_date)
def _read_local_sw_members(
self,
sector_code: str,
allow_stale: bool = False,
) -> list[dict[str, Any]] | None:
now = datetime.now().astimezone()
cache = getattr(self, "_sw_member_cache", None)
lock = getattr(self, "_sw_member_lock", None)
if isinstance(cache, dict) and lock is not None:
with lock:
packed = cache.get(sector_code)
if isinstance(packed, dict):
loaded_at = packed.get("loaded_at")
rows = packed.get("rows")
fresh = (
isinstance(loaded_at, datetime)
and now - loaded_at < _SW_MEMBER_TTL
)
if isinstance(rows, list) and (fresh or allow_stale):
return [dict(item) for item in rows]
path = _sw_member_path(sector_code)
if not path.exists():
return None
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
rows = list(payload.get("rows") or [])
updated = str(payload.get("updated_at") or "")
fresh = False
try:
stamped = datetime.fromisoformat(updated)
if stamped.tzinfo is None:
stamped = stamped.replace(tzinfo=now.tzinfo)
fresh = now - stamped.astimezone(now.tzinfo) < _SW_MEMBER_TTL
except ValueError:
fresh = False
if rows and (fresh or allow_stale):
self._remember_sw_members(sector_code, rows)
return rows
return None
def _write_local_sw_members(self, sector_code: str, rows: list[dict[str, Any]]) -> None:
packed = [dict(item) for item in rows]
self._remember_sw_members(sector_code, packed)
path = _sw_member_path(sector_code)
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(
{
"sector_code": sector_code,
"updated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"rows": packed,
},
ensure_ascii=False,
),
encoding="utf-8",
) )
deduped: dict[str, dict[str, Any]] = {} except OSError:
for row in _reconcile_membership_rows(rows): pass
code = str(row.get("ts_code") or "")
if code and _membership_active_on(row, trade_date): def _remember_sw_members(self, sector_code: str, rows: list[dict[str, Any]]) -> None:
current = deduped.get(code) cache = getattr(self, "_sw_member_cache", None)
if current is None or str(row.get("in_date") or "") > str(current.get("in_date") or ""): lock = getattr(self, "_sw_member_lock", None)
deduped[code] = row if not isinstance(cache, dict) or lock is None:
return list(deduped.values()) return
with lock:
cache[sector_code] = {
"loaded_at": datetime.now().astimezone(),
"rows": [dict(item) for item in rows],
}
def sw_sector_members(self, sector_code: str, trade_date: str) -> list[dict[str, Any]]: def sw_sector_members(self, sector_code: str, trade_date: str) -> list[dict[str, Any]]:
"""Return constituents active in a Shenwan L2 industry on the target date.""" """Return constituents active in a Shenwan L2 industry on the target date."""
@@ -311,37 +435,37 @@ class ShenwanIndustryMixin:
finalized: bool = False, finalized: bool = False,
) -> dict[str, Any]: ) -> dict[str, Any]:
sector_code = str(industry.get("l2_code") or "") sector_code = str(industry.get("l2_code") or "")
sw_rows = self.query( sw_row, outer_source, outer_error = self._sw_outer_realtime(
"rt_sw_k", sector_code,
{"ts_code": sector_code}, str(industry.get("l2_name") or ""),
"ts_code,name,trade_time,close,pre_close,high,open,low,vol,amount,pct_change", trade_date,
finalized=finalized,
) )
sw_row = sw_rows[0] if sw_rows else {} trade_time = str(sw_row.get("trade_time") or sw_row.get("quote_time") or "")
trade_time = str(sw_row.get("trade_time") or "") quote_date = str(sw_row.get("quote_date") or trade_time[:10].replace("-", ""))
quote_date = trade_time[:10].replace("-", "") quote_clock = trade_time[11:19] if len(trade_time) >= 19 else str(sw_row.get("quote_clock") or "")
quote_clock = trade_time[11:19] if len(trade_time) >= 19 else ""
outer_precise = bool(sw_row and quote_date == trade_date) outer_precise = bool(sw_row and quote_date == trade_date)
if finalized and (not quote_clock or quote_clock < "15:00:00"): if finalized and quote_clock and quote_clock < "15:00:00":
outer_precise = False outer_precise = False
official_change = _number(sw_row.get("pct_change")) official_change = _number(sw_row.get("pct_change") if sw_row.get("pct_change") not in (None, "") else sw_row.get("change"))
if not official_change: if not official_change:
close = _number(sw_row.get("close")) close = _number(sw_row.get("close") if sw_row.get("close") not in (None, "") else sw_row.get("price"))
pre_close = _number(sw_row.get("pre_close")) pre_close = _number(sw_row.get("pre_close") if sw_row.get("pre_close") not in (None, "") else sw_row.get("previous_close"))
official_change = (close / pre_close - 1) * 100 if close and pre_close else 0 official_change = (close / pre_close - 1) * 100 if close and pre_close else 0
if not outer_precise: if not outer_precise:
official_change = None official_change = None
outer_error = "" if not sw_row and not outer_error:
if not sw_row: outer_error = f"申万行业 {sector_code} 当日外显待盘后正式数据或免费实时源"
outer_error = f"No Shenwan realtime index returned for {sector_code}" elif quote_date and quote_date != trade_date:
elif quote_date != trade_date: outer_error = f"申万实时行业日期是 {quote_date},期望 {trade_date}"
outer_error = f"Shenwan realtime index date is {quote_date or 'unknown'}, expected {trade_date}" elif finalized and quote_clock and quote_clock < "15:00:00":
elif finalized and (not quote_clock or quote_clock < "15:00:00"): outer_error = f"申万行业尚未形成收盘快照({trade_time}"
outer_error = f"Shenwan realtime index is not a close snapshot ({trade_time})"
valid: list[dict[str, Any]] = [] valid: list[dict[str, Any]] = []
codes: list[str] = [] codes: list[str] = []
reference: dict[str, Any] = {} reference: dict[str, Any] = {}
inner_error = "" inner_error = ""
inner_source = "unavailable"
try: try:
reference = self._load_realtime_reference(trade_date, previous_trade_date) reference = self._load_realtime_reference(trade_date, previous_trade_date)
active_codes = { active_codes = {
@@ -352,20 +476,27 @@ class ShenwanIndustryMixin:
codes = [ codes = [
str(row.get("ts_code") or "") str(row.get("ts_code") or "")
for row in members for row in members
if str(row.get("ts_code") or "") in active_codes if str(row.get("ts_code") or "")
] ]
if codes: if active_codes:
quotes = self.query("rt_k", {"ts_code": ",".join(codes)}, "") listed = [code for code in codes if code in active_codes]
for row in quotes: if listed:
close = _number(row.get("close")) codes = listed
previous_close = _number(row.get("pre_close")) quotes, inner_source = self._load_member_realtime_quotes(codes, trade_date)
if close <= 0 or previous_close <= 0: for row in quotes:
continue close = _number(row.get("close"))
valid.append({**row, "change": (close / previous_close - 1) * 100}) previous_close = _number(row.get("pre_close"))
else: if close <= 0 or previous_close <= 0:
continue
valid.append({**row, "change": (close / previous_close - 1) * 100})
if not codes:
inner_error = f"No active Shenwan members returned for {sector_code}" inner_error = f"No active Shenwan members returned for {sector_code}"
elif not quotes:
inner_error = f"申万成分实时行情暂不可用:{sector_code}"
except TushareError as exc: except TushareError as exc:
inner_error = str(exc) inner_error = str(exc)
if "rt_k" in inner_error or "权限" in inner_error:
inner_error = "申万成分实时行情暂不可用,已避开无权限接口"
coverage = len(valid) / max(len(codes), 1) * 100 coverage = len(valid) / max(len(codes), 1) * 100
valid_codes = {str(item.get("ts_code") or "") for item in valid} valid_codes = {str(item.get("ts_code") or "") for item in valid}
@@ -390,16 +521,18 @@ class ShenwanIndustryMixin:
} }
equal_change = sum(item["change"] for item in valid) / len(valid) if valid else 0 equal_change = sum(item["change"] for item in valid) / len(valid) if valid else 0
amount_billion = sum(_number(item.get("amount")) for item in valid) / 100000000 amount_billion = sum(_number(item.get("amount")) for item in valid) / 100000000
market_rows: list[dict[str, Any]] = []
try: try:
self._ensure_realtime_market_cache(trade_date) market_rows = self._ensure_realtime_market_cache(trade_date)
with self._realtime_reference_lock:
market_rows = list(
(self._latest_realtime_market.get(trade_date) or {}).get("rows") or []
)
except TushareError as exc: except TushareError as exc:
market_rows = [] message = str(exc)
inner_precise = False if "rt_k" in message or "权限" in message:
inner_error = inner_error or str(exc) market_error = "全市场实时行情暂不可用,已避开无权限接口"
else:
market_error = message
if not valid:
inner_precise = False
inner_error = inner_error or market_error
capital_map = { capital_map = {
str(item.get("ts_code") or ""): item str(item.get("ts_code") or ""): item
for item in reference.get("capital_rows") or [] for item in reference.get("capital_rows") or []
@@ -408,20 +541,28 @@ class ShenwanIndustryMixin:
for item in valid: for item in valid:
capital = capital_map.get(str(item.get("ts_code") or ""), {}) capital = capital_map.get(str(item.get("ts_code") or ""), {})
float_share = _number(capital.get("float_share")) float_share = _number(capital.get("float_share"))
if float_share: volume = _number(item.get("vol"))
sector_turnovers.append(_number(item.get("vol")) / float_share / 100) if float_share and volume:
# 免费源成交量为股;daily_basic.float_share 为万股。
sector_turnovers.append(volume / float_share / 100)
market_turnovers = [] market_turnovers = []
for item in market_rows: for item in market_rows:
capital = capital_map.get(str(item.get("ts_code") or ""), {}) capital = capital_map.get(str(item.get("ts_code") or ""), {})
float_share = _number(capital.get("float_share")) float_share = _number(capital.get("float_share"))
if float_share: volume = _number(item.get("vol"))
market_turnovers.append(_number(item.get("vol")) / float_share / 100) if float_share and volume:
market_turnovers.append(volume / float_share / 100)
average_turnover = sum(sector_turnovers) / len(sector_turnovers) if sector_turnovers else 0 average_turnover = sum(sector_turnovers) / len(sector_turnovers) if sector_turnovers else 0
market_turnover = sum(market_turnovers) / len(market_turnovers) if market_turnovers else 0 market_turnover = sum(market_turnovers) / len(market_turnovers) if market_turnovers else 0
relative_turnover = average_turnover / market_turnover if market_turnover else 0 relative_turnover = average_turnover / market_turnover if market_turnover else 0
if not relative_turnover: delayed = "delayed" in str(inner_source) or any(item.get("delayed") for item in valid)
inner_precise = False delay_seconds = max((int(item.get("delay_seconds") or 0) for item in valid), default=0)
inner_error = inner_error or "Shenwan member relative turnover is unavailable" delay_notice = ""
if delayed:
delay_notice = next(
(str(item.get("delay_notice") or "") for item in valid if item.get("delay_notice")),
"",
) or f"主备免费行情均暂不可用,显示最近一次真实快照(延迟 {delay_seconds} 秒)"
return { return {
"code": sector_code, "code": sector_code,
"name": str(industry.get("l2_name") or sw_row.get("name") or ""), "name": str(industry.get("l2_name") or sw_row.get("name") or ""),
@@ -447,9 +588,9 @@ class ShenwanIndustryMixin:
"amount_billion": round(amount_billion, 2), "amount_billion": round(amount_billion, 2),
"count": sum(item["change"] >= 9.5 for item in valid), "count": sum(item["change"] >= 9.5 for item in valid),
"max_streak": 0, "max_streak": 0,
"source": "tushare_rt_sw_k+sw_members_rt_k", "source": f"{outer_source or 'unavailable'}+{inner_source}",
"inner_source": "tushare_sw_members+rt_k", "inner_source": inner_source,
"outer_source": "tushare_rt_sw_k", "outer_source": outer_source or "unavailable",
"taxonomy": "sw_l2", "taxonomy": "sw_l2",
"industry": industry, "industry": industry,
"trade_date": trade_date, "trade_date": trade_date,
@@ -463,10 +604,96 @@ class ShenwanIndustryMixin:
"precise": inner_precise and outer_precise, "precise": inner_precise and outer_precise,
"inner_error": inner_error, "inner_error": inner_error,
"outer_error": outer_error, "outer_error": outer_error,
"delayed": delayed,
"delay_seconds": delay_seconds,
"delay_notice": delay_notice,
"schema_version": 6, "schema_version": 6,
"methodology": "外显使用申万官方 rt_sw_k;内核独立使用申万成分 rt_k 宽度与相对换手聚合", "methodology": "外显使用已发布 sw_daily 或免费申万实时;内核使用数据中枢/免费实时成分,不调用 rt_sw_k",
} }
def _sw_outer_realtime(
self,
sector_code: str,
sector_name: str,
trade_date: str,
finalized: bool = False,
) -> tuple[dict[str, Any], str, str]:
hub = getattr(self, "try_sector_quote", None)
if callable(hub):
try:
row = hub(sector_code, "" if finalized else trade_date)
except Exception as exc:
message = str(exc)
if finalized:
return {}, "", f"申万行业 {sector_code} 盘后正式数据待入库"
return {}, "", f"数据中枢申万实时暂不可用:{message[:180]}"
if row:
return dict(row), str(row.get("source") or "datahub"), ""
if finalized:
return {}, "", f"申万行业 {sector_code} 当日盘后正式数据尚未入库"
return {}, "", f"申万行业 {sector_code} 当日外显待补充"
def _load_member_realtime_quotes(
self,
codes: list[str],
trade_date: str,
) -> tuple[list[dict[str, Any]], str]:
wanted = [str(code).strip() for code in codes if str(code or "").strip()]
if not wanted:
return [], "unavailable"
best_rows: list[dict[str, Any]] = []
best_source = "unavailable"
def consider(rows: list[dict[str, Any]] | None, source: str) -> list[dict[str, Any]]:
nonlocal best_rows, best_source
filtered = _filter_quotes_for_codes(rows, wanted)
if len(filtered) > len(best_rows):
best_rows = filtered
best_source = source
return filtered
hub_market = getattr(self, "try_market_quotes", None)
if callable(hub_market):
filtered = consider(hub_market(trade_date) or [], "datahub")
if len(filtered) >= max(1, int(len(wanted) * 0.9)):
delayed = any(item.get("delayed") for item in filtered)
return filtered, "datahub_delayed" if delayed else "datahub"
hub = getattr(self, "try_quotes", None)
if callable(hub):
collected: list[dict[str, Any]] = []
for index in range(0, len(wanted), _QUOTE_BATCH):
collected.extend(hub(wanted[index:index + _QUOTE_BATCH]) or [])
filtered = consider(collected, "datahub")
if len(filtered) >= max(1, int(len(wanted) * 0.9)):
delayed = any(item.get("delayed") for item in filtered)
return filtered, "datahub_delayed" if delayed else "datahub"
if best_rows:
delayed = any(item.get("delayed") for item in best_rows)
if delayed and not str(best_source).endswith("_delayed"):
return best_rows, f"{best_source}_delayed"
return best_rows, best_source
return [], "unavailable"
_QUOTE_BATCH = 60
def _filter_quotes_for_codes(
rows: list[dict[str, Any]] | None,
codes: list[str],
) -> list[dict[str, Any]]:
wanted = {str(code) for code in codes if code}
filtered: list[dict[str, Any]] = []
seen: set[str] = set()
for row in rows or []:
ts_code = str(row.get("ts_code") or "")
if ts_code in wanted and ts_code not in seen:
seen.add(ts_code)
filtered.append(row)
return filtered
def _filter_members_by_listing( def _filter_members_by_listing(
members: list[dict[str, Any]], members: list[dict[str, Any]],
@@ -568,6 +795,22 @@ def _reconcile_membership_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any
return list(reconciled.values()) return list(reconciled.values())
def _sw_member_path(sector_code: str) -> Path:
safe = "".join(ch if ch.isalnum() or ch in "._-" else "_" for ch in str(sector_code or ""))
return _SW_MEMBER_DIR / f"{safe or 'unknown'}.json"
def _active_members(rows: list[dict[str, Any]], trade_date: str) -> list[dict[str, Any]]:
deduped: dict[str, dict[str, Any]] = {}
for row in rows:
code = str(row.get("ts_code") or "")
if code and _membership_active_on(row, trade_date):
current = deduped.get(code)
if current is None or str(row.get("in_date") or "") > str(current.get("in_date") or ""):
deduped[code] = dict(row)
return list(deduped.values())
def _match_sector_row(rows: list[dict[str, Any]], identifier: str) -> dict[str, Any] | None: def _match_sector_row(rows: list[dict[str, Any]], identifier: str) -> dict[str, Any] | None:
if not rows: if not rows:
return None return None
+5 -19
View File
@@ -5,13 +5,14 @@ from typing import Any
from backend.bootstrap.config import display_compact_date as _display_date from backend.bootstrap.config import display_compact_date as _display_date
from backend.data.numbers import finite_number as _number from backend.data.numbers import finite_number as _number
from backend.data.providers.tushare_helpers import _moneyflow_payload
class StockMixin: class StockMixin:
def stock_detail(self, ts_code: str, requested_date: str) -> dict[str, Any]: def stock_detail(self, ts_code: str, requested_date: str) -> dict[str, Any]:
trade_date, _ = self.resolve_trade_context(requested_date) trade_date, _ = self.resolve_trade_context(requested_date)
end = datetime.strptime(trade_date, "%Y%m%d") end = datetime.strptime(trade_date, "%Y%m%d")
start_date = (end - timedelta(days=190)).strftime("%Y%m%d") start_date = (end - timedelta(days=400)).strftime("%Y%m%d")
daily = self.query( daily = self.query(
"daily", "daily",
{"ts_code": ts_code, "start_date": start_date, "end_date": trade_date}, {"ts_code": ts_code, "start_date": start_date, "end_date": trade_date},
@@ -41,7 +42,7 @@ class StockMixin:
factor_map = {row["trade_date"]: _number(row.get("adj_factor"), 1) for row in factors} factor_map = {row["trade_date"]: _number(row.get("adj_factor"), 1) for row in factors}
latest_factor = max(factor_map.values(), default=1) or 1 latest_factor = max(factor_map.values(), default=1) or 1
prices = [] prices = []
for row in sorted(daily, key=lambda item: item.get("trade_date", ""))[-90:]: for row in sorted(daily, key=lambda item: item.get("trade_date", ""))[-250:]:
factor = factor_map.get(row.get("trade_date"), latest_factor) factor = factor_map.get(row.get("trade_date"), latest_factor)
ratio = factor / latest_factor ratio = factor / latest_factor
prices.append( prices.append(
@@ -56,7 +57,7 @@ class StockMixin:
"amount_billion": round(_number(row.get("amount")) / 100000, 2), "amount_billion": round(_number(row.get("amount")) / 100000, 2),
} }
) )
flow = moneyflow[0] if moneyflow else {} flow = moneyflow[0] if moneyflow else None
basic = basics[0] if basics else {} basic = basics[0] if basics else {}
daily_basic = daily_basics[0] if daily_basics else {} daily_basic = daily_basics[0] if daily_basics else {}
latest = prices[-1] if prices else {} latest = prices[-1] if prices else {}
@@ -87,22 +88,7 @@ class StockMixin:
"amount_billion": latest.get("amount_billion", 0), "amount_billion": latest.get("amount_billion", 0),
}, },
"prices": prices, "prices": prices,
"moneyflow": { "moneyflow": _moneyflow_payload(flow),
"net_million": round(_number(flow.get("net_mf_amount")) / 100, 2),
"large_million": round(
(_number(flow.get("buy_lg_amount")) + _number(flow.get("buy_elg_amount"))
- _number(flow.get("sell_lg_amount")) - _number(flow.get("sell_elg_amount"))) / 100,
2,
),
"medium_million": round(
(_number(flow.get("buy_md_amount")) - _number(flow.get("sell_md_amount"))) / 100,
2,
),
"small_million": round(
(_number(flow.get("buy_sm_amount")) - _number(flow.get("sell_sm_amount"))) / 100,
2,
),
},
} }
def stock_intraday(self, ts_code: str, requested_date: str) -> dict[str, Any]: def stock_intraday(self, ts_code: str, requested_date: str) -> dict[str, Any]:
@@ -20,6 +20,8 @@ class TushareTransportMixin:
params: dict[str, Any] | None = None, params: dict[str, Any] | None = None,
fields: str = "", fields: str = "",
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
if api_name == "rt_sw_k":
raise TushareError("rt_sw_k is disabled; use published sw_daily or free Shenwan realtime")
payload = json.dumps( payload = json.dumps(
{ {
"api_name": api_name, "api_name": api_name,
+521 -4
View File
@@ -19,8 +19,22 @@ class RealtimeAggregateError(RuntimeError):
EASTMONEY_INDEX_URL = "https://push2.eastmoney.com/api/qt/ulist.np/get" EASTMONEY_INDEX_URL = "https://push2.eastmoney.com/api/qt/ulist.np/get"
EASTMONEY_STOCK_URL = "https://push2.eastmoney.com/api/qt/stock/get"
EASTMONEY_STOCK_FIELDS = "f43,f44,f45,f46,f47,f48,f57,f58,f60,f86,f168,f62,f66,f72,f78,f84"
EASTMONEY_SECTOR_URL = "https://push2.eastmoney.com/api/qt/clist/get" EASTMONEY_SECTOR_URL = "https://push2.eastmoney.com/api/qt/clist/get"
EASTMONEY_ZT_POOL_URL = "https://push2ex.eastmoney.com/getTopicZTPool"
EASTMONEY_ZB_POOL_URL = "https://push2ex.eastmoney.com/getTopicZBPool"
EASTMONEY_A_SHARE_BOARDS = (
"m:0+t:6",
"m:0+t:80",
"m:1+t:2",
"m:1+t:23",
"m:0+t:81",
)
EASTMONEY_QUOTE_FIELDS = "f12,f13,f14,f2,f3,f4,f5,f6,f15,f16,f17,f18,f8,f124"
EASTMONEY_MARKET_PAGE_SIZE = 100
TENCENT_INDEX_URL = "https://qt.gtimg.cn/q=sh000001,sz399001,sz399006" TENCENT_INDEX_URL = "https://qt.gtimg.cn/q=sh000001,sz399001,sz399006"
TENCENT_QUOTE_URL = "https://qt.gtimg.cn/q="
THS_LIMIT_URL = "https://data.10jqka.com.cn/dataapi/limit_up/limit_up_pool" THS_LIMIT_URL = "https://data.10jqka.com.cn/dataapi/limit_up/limit_up_pool"
XGB_POOL_URL = "https://flash-api.xuangubao.cn/api/pool/detail" XGB_POOL_URL = "https://flash-api.xuangubao.cn/api/pool/detail"
BROWSER_USER_AGENT = ( BROWSER_USER_AGENT = (
@@ -134,6 +148,312 @@ class WebRealtimeAggregator:
raise RealtimeAggregateError(f"Eastmoney returned {len(result)}/3 indices") raise RealtimeAggregateError(f"Eastmoney returned {len(result)}/3 indices")
return result return result
def eastmoney_market_quotes(self, expected_date: str = "") -> list[dict[str, Any]]:
"""Full A-share snapshot via Eastmoney clist, used when Tushare rt_k is unavailable."""
now = time.time()
cache_key = "assembled:eastmoney_market"
with self._response_cache_lock:
cached = self._response_cache.get(cache_key)
cache_age = now - float((cached or {}).get("created_at") or 0)
if cached and cache_age <= min(20, self.response_cache_ttl_seconds):
quotes = list(cached.get("payload") or [])
return self._filter_quotes_by_date(quotes, expected_date)
rows: list[dict[str, Any]] = []
board_errors: list[str] = []
for board in EASTMONEY_A_SHARE_BOARDS:
try:
rows.extend(self._eastmoney_board_quotes(board))
except Exception as exc:
board_errors.append(f"{board}:{exc}")
quotes = []
seen: set[str] = set()
for row in rows:
quote = _normalize_eastmoney_quote(row)
ts_code = str((quote or {}).get("ts_code") or "")
if not quote or ts_code in seen:
continue
seen.add(ts_code)
quotes.append(quote)
if len(quotes) < 200:
detail = f"{'; '.join(board_errors)}" if board_errors else ""
raise RealtimeAggregateError(
f"Eastmoney market snapshot too small: {len(quotes)}{detail}"
)
quotes = self._filter_quotes_by_date(quotes, expected_date)
with self._response_cache_lock:
self._response_cache[cache_key] = {"created_at": now, "payload": quotes}
return quotes
def _eastmoney_board_quotes(self, board: str) -> list[dict[str, Any]]:
first = self._eastmoney_market_page(board, 1)
data = first.get("data") or {}
rows = _diff_rows(data)
total = int(_number(data.get("total")))
page_count = 1
if total > 0:
page_count = max(1, (total + EASTMONEY_MARKET_PAGE_SIZE - 1) // EASTMONEY_MARKET_PAGE_SIZE)
for page in range(2, min(page_count, 40) + 1):
payload = self._eastmoney_market_page(board, page)
rows.extend(_diff_rows(payload.get("data") or {}))
return rows
def _eastmoney_market_page(self, board: str, page: int) -> dict[str, Any]:
return self._get_json(
EASTMONEY_SECTOR_URL,
{
"pn": str(page),
"pz": str(EASTMONEY_MARKET_PAGE_SIZE),
"po": "1",
"np": "1",
"fltt": "2",
"invt": "2",
"fid": "f12",
"fs": board,
"fields": EASTMONEY_QUOTE_FIELDS,
},
referer="https://quote.eastmoney.com/center/gridlist.html",
)
def _filter_quotes_by_date(
self,
quotes: list[dict[str, Any]],
expected_date: str,
) -> list[dict[str, Any]]:
want = str(expected_date or "").replace("-", "")
if not want or not quotes:
return quotes
dated = [item for item in quotes if str(item.get("quote_date") or "") == want]
if dated and len(dated) >= max(100, int(len(quotes) * 0.2)):
return dated
if dated:
return dated
if all(not item.get("quote_date") for item in quotes):
return quotes
raise RealtimeAggregateError(f"Eastmoney quotes are not for {want}")
def tencent_market_quotes(
self,
codes: list[str],
expected_date: str = "",
) -> list[dict[str, Any]]:
quotes = self.tencent_stock_quotes(codes, expected_date="", minimum=200)
return self._filter_quotes_by_date(quotes, expected_date)
def tencent_stock_quotes(
self,
codes: list[str],
expected_date: str = "",
minimum: int | None = None,
) -> list[dict[str, Any]]:
symbols: list[str] = []
seen: set[str] = set()
for raw in codes:
ts = str(raw or "").strip().upper()
if not ts:
continue
symbol = ts.split(".")[0]
if not symbol.isdigit() or len(symbol) != 6 or symbol in seen:
continue
seen.add(symbol)
if ts.endswith(".SH") or symbol.startswith(("5", "6", "9")):
symbols.append(f"sh{symbol}")
elif ts.endswith(".BJ") or symbol.startswith(("4", "8")):
symbols.append(f"bj{symbol}")
else:
symbols.append(f"sz{symbol}")
if not symbols:
raise RealtimeAggregateError("No stock codes available for Tencent quotes")
quotes: list[dict[str, Any]] = []
batch_size = 80
def load_batch(batch: list[str]) -> list[dict[str, Any]]:
raw, _cache_age = self._get_text(
f"{TENCENT_QUOTE_URL}{','.join(batch)}",
referer="https://gu.qq.com/",
encoding="gb18030",
)
return [
quote
for line in raw.splitlines()
if (quote := _parse_tencent_stock_quote(line))
]
batches = [symbols[index:index + batch_size] for index in range(0, len(symbols), batch_size)]
errors: list[str] = []
with ThreadPoolExecutor(max_workers=4) as executor:
for result in executor.map(self._capture, [lambda batch=batch: load_batch(batch) for batch in batches]):
rows, status = result
if status.get("ok") and rows:
quotes.extend(rows)
elif not status.get("ok"):
errors.append(str(status.get("error") or "batch failed"))
floor = minimum if minimum is not None else max(1, int(len(symbols) * 0.5))
if len(quotes) < floor:
detail = f"{'; '.join(errors[:3])}" if errors else ""
raise RealtimeAggregateError(
f"Tencent quotes too small: {len(quotes)}/{len(symbols)}{detail}"
)
return self._filter_quotes_by_date(quotes, expected_date)
def tencent_stock_quote(self, code: str, expected_date: str = "") -> dict[str, Any]:
symbol, _secid, ts_code = _a_share_identity(code)
raw, _cache_age = self._get_text(
f"{TENCENT_QUOTE_URL}{symbol}",
referer="https://gu.qq.com/",
encoding="gb18030",
)
quote = next(
(
item
for line in raw.splitlines()
if (item := _parse_tencent_stock_quote(line))
),
None,
)
if not quote:
raise RealtimeAggregateError(f"Tencent stock quote unavailable for {ts_code}")
return _require_quote_date(quote, expected_date)
def eastmoney_stock_quote(self, code: str, expected_date: str = "") -> dict[str, Any]:
_symbol, secid, ts_code = _a_share_identity(code)
payload = self._get_json(
EASTMONEY_STOCK_URL,
{
"secid": secid,
"invt": "2",
"fltt": "2",
"fields": EASTMONEY_STOCK_FIELDS,
},
referer="https://quote.eastmoney.com/",
)
quote = _normalize_eastmoney_stock_quote(payload.get("data") or {}, ts_code)
if not quote:
raise RealtimeAggregateError(f"Eastmoney stock quote unavailable for {ts_code}")
return _require_quote_date(quote, expected_date)
def eastmoney_stock_quotes(
self,
codes: list[str],
expected_date: str = "",
) -> list[dict[str, Any]]:
secids = []
for code in codes:
try:
_symbol, secid, _ts = _a_share_identity(code)
except RealtimeAggregateError:
continue
secids.append(secid)
quotes: list[dict[str, Any]] = []
for index in range(0, len(secids), 60):
payload = self._get_json(
EASTMONEY_INDEX_URL,
{
"secids": ",".join(secids[index:index + 60]),
"fltt": "2",
"invt": "2",
"fields": EASTMONEY_QUOTE_FIELDS,
},
referer="https://quote.eastmoney.com/",
)
for row in _diff_rows(payload.get("data") or {}):
quote = _normalize_eastmoney_quote(row)
if quote:
quotes.append(quote)
return self._filter_quotes_by_date(quotes, expected_date)
def eastmoney_shenwan_quote(
self,
ts_code: str,
expected_date: str = "",
) -> dict[str, Any]:
code = str(ts_code or "").split(".")[0]
if not code:
raise RealtimeAggregateError("Invalid Shenwan code")
payload = self._get_json(
EASTMONEY_INDEX_URL,
{
"secids": f"90.{code}",
"fltt": "2",
"invt": "2",
"fields": "f12,f14,f2,f3,f4,f15,f16,f17,f18,f6,f8,f104,f105,f128,f136,f140,f124",
},
referer="https://quote.eastmoney.com/",
)
row = next((item for item in _diff_rows(payload.get("data") or {}) if item), None)
if not row:
raise RealtimeAggregateError(f"Eastmoney Shenwan quote missing for {code}")
epoch = int(_number(row.get("f124")))
quote_time = (
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
if epoch
else ""
)
close = _number(row.get("f2"))
previous = _number(row.get("f18"))
if close <= 0 or previous <= 0:
raise RealtimeAggregateError(f"Eastmoney Shenwan quote empty for {code}")
result = {
"ts_code": f"{code}.SI",
"code": f"{code}.SI",
"name": row.get("f14") or code,
"price": close,
"close": close,
"pre_close": previous,
"previous_close": previous,
"open": _number(row.get("f17")),
"high": _number(row.get("f15")),
"low": _number(row.get("f16")),
"change": _number(row.get("f3")),
"pct_change": _number(row.get("f3")),
"amount": _number(row.get("f6")),
"leader": row.get("f128") or "--",
"leader_code": row.get("f140") or "",
"leading_pct": _number(row.get("f136")),
"up_count": int(_number(row.get("f104"))),
"down_count": int(_number(row.get("f105"))),
"quote_time": quote_time,
"trade_time": quote_time,
"quote_date": datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d") if epoch else "",
"quote_time_epoch": epoch,
"source": "eastmoney_sw",
}
return _require_quote_date(result, expected_date) if expected_date else result
def eastmoney_limit_pool(self, trade_date: str = "") -> list[dict[str, Any]]:
day = str(trade_date or "").replace("-", "")
rows: list[dict[str, Any]] = []
for url, limit_type in (
(EASTMONEY_ZT_POOL_URL, "U"),
(EASTMONEY_ZB_POOL_URL, "Z"),
):
try:
payload = self._get_json(
url,
{
"ut": "7eea3edcaed734bea9cbfc24409ed989",
"dpt": "wz.ztzt",
"PageIndex": "0",
"PageSize": "200",
"sort": "fbt:asc",
"date": day,
},
referer="https://quote.eastmoney.com/ztb/detail",
)
except RealtimeAggregateError:
continue
pool = (payload.get("data") or {}).get("pool") or []
if isinstance(pool, dict):
pool = list(pool.values())
for item in pool:
if not isinstance(item, dict):
continue
parsed = _normalize_eastmoney_limit_row(item, limit_type)
if parsed:
rows.append(parsed)
return rows
def tencent_indices(self) -> list[dict[str, Any]]: def tencent_indices(self) -> list[dict[str, Any]]:
raw, cache_age = self._get_text( raw, cache_age = self._get_text(
TENCENT_INDEX_URL, TENCENT_INDEX_URL,
@@ -185,11 +505,17 @@ class WebRealtimeAggregator:
if not matched: if not matched:
raise RealtimeAggregateError(f"Eastmoney sector not found: {query}") raise RealtimeAggregateError(f"Eastmoney sector not found: {query}")
epoch = int(_number(matched.get("f124"))) epoch = int(_number(matched.get("f124")))
quote_time = (
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
if epoch else ""
)
return { return {
"code": matched.get("f12") or "", "code": matched.get("f12") or "",
"name": matched.get("f14") or query, "name": matched.get("f14") or query,
"price": _number(matched.get("f2")), "price": _number(matched.get("f2")),
"close": _number(matched.get("f2")),
"change": _number(matched.get("f3")), "change": _number(matched.get("f3")),
"pct_change": _number(matched.get("f3")),
"change_amount": _number(matched.get("f4")), "change_amount": _number(matched.get("f4")),
"turnover_rate": _number(matched.get("f8")), "turnover_rate": _number(matched.get("f8")),
"up_count": int(_number(matched.get("f104"))), "up_count": int(_number(matched.get("f104"))),
@@ -198,10 +524,9 @@ class WebRealtimeAggregator:
"leader_code": matched.get("f140") or "", "leader_code": matched.get("f140") or "",
"leading_pct": _number(matched.get("f136")), "leading_pct": _number(matched.get("f136")),
"quote_time_epoch": epoch, "quote_time_epoch": epoch,
"quote_time": ( "quote_time": quote_time,
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds") "trade_time": quote_time,
if epoch else "" "quote_date": datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d") if epoch else "",
),
"source": "eastmoney_push2", "source": "eastmoney_push2",
"match_query": query, "match_query": query,
} }
@@ -397,6 +722,198 @@ class WebRealtimeAggregator:
) from last_error ) from last_error
def _diff_rows(data: dict[str, Any]) -> list[dict[str, Any]]:
diff = data.get("diff") or []
if isinstance(diff, dict):
return [row for row in diff.values() if isinstance(row, dict)]
return [row for row in diff if isinstance(row, dict)]
def _a_share_identity(code: str) -> tuple[str, str, str]:
raw = str(code or "").strip().upper()
symbol = raw.split(".")[0]
if not symbol.isdigit() or len(symbol) != 6:
raise RealtimeAggregateError("Invalid stock code")
if raw.endswith(".SH") or symbol.startswith(("5", "6", "9")):
return f"sh{symbol}", f"1.{symbol}", f"{symbol}.SH"
if raw.endswith(".BJ") or symbol.startswith(("4", "8")):
return f"bj{symbol}", f"0.{symbol}", f"{symbol}.BJ"
return f"sz{symbol}", f"0.{symbol}", f"{symbol}.SZ"
def _require_quote_date(quote: dict[str, Any], expected_date: str) -> dict[str, Any]:
want = str(expected_date or "").replace("-", "")
got = str(quote.get("quote_date") or "")
if want and got != want:
raise RealtimeAggregateError(f"quote date {got or 'empty'} is not {want}")
return quote
def _normalize_eastmoney_stock_quote(
row: dict[str, Any], ts_code: str
) -> dict[str, Any] | None:
close = _number(row.get("f43"))
previous_close = _number(row.get("f60"))
if close <= 0 or previous_close <= 0:
return None
epoch = int(_number(row.get("f86")))
quote_date = ""
if epoch > 0:
quote_date = datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d")
return {
"ts_code": ts_code,
"name": row.get("f58") or ts_code.split(".")[0],
"pre_close": previous_close,
"open": _number(row.get("f46")),
"high": _number(row.get("f44")),
"low": _number(row.get("f45")),
"close": close,
"vol": _number(row.get("f47")) * 100,
"amount": _number(row.get("f48")),
"num": 0,
"quote_date": quote_date,
"quote_time_epoch": epoch,
"turnover_rate": _number(row.get("f168")),
"net_mf_amount": _eastmoney_flow_wan(row.get("f62")),
"large_amount": _eastmoney_flow_wan(row.get("f62")),
"medium_amount": _eastmoney_flow_wan(row.get("f78")),
"small_amount": _eastmoney_flow_wan(row.get("f84")),
"source": "eastmoney_stock",
}
def _parse_tencent_stock_quote(line: str) -> dict[str, Any] | None:
if '="' not in line:
return None
prefix, payload = line.split('="', 1)
fields = payload.rsplit('";', 1)[0].split("~")
if len(fields) < 38:
return None
symbol = fields[2]
if not symbol.isdigit() or len(symbol) != 6:
return None
close = _number(fields[3])
previous_close = _number(fields[4])
if close <= 0 or previous_close <= 0:
return None
marker = prefix.lower()
if "sh" in marker:
ts_code = f"{symbol}.SH"
elif "bj" in marker:
ts_code = f"{symbol}.BJ"
else:
ts_code = f"{symbol}.SZ"
try:
quote_time = datetime.strptime(fields[30], "%Y%m%d%H%M%S")
quote_date = quote_time.strftime("%Y%m%d")
epoch = int(quote_time.timestamp())
except ValueError:
quote_date = ""
epoch = 0
return {
"ts_code": ts_code,
"name": fields[1] or symbol,
"pre_close": previous_close,
"open": _number(fields[5]),
"high": _number(fields[33]),
"low": _number(fields[34]),
"close": close,
"vol": _number(fields[6]) * 100,
"amount": _number(fields[37]) * 10000,
"num": 0,
"quote_date": quote_date,
"quote_time_epoch": epoch,
"source": "tencent_qt",
}
def _normalize_eastmoney_quote(row: dict[str, Any]) -> dict[str, Any] | None:
symbol = str(row.get("f12") or "").strip()
if not symbol.isdigit() or len(symbol) != 6:
return None
close = _number(row.get("f2"))
previous_close = _number(row.get("f18"))
if close <= 0 or previous_close <= 0:
return None
market = int(_number(row.get("f13")))
if market == 1 or symbol.startswith(("5", "6", "9")):
ts_code = f"{symbol}.SH"
elif symbol.startswith(("4", "8")):
ts_code = f"{symbol}.BJ"
else:
ts_code = f"{symbol}.SZ"
epoch = int(_number(row.get("f124")))
quote_date = ""
if epoch > 0:
quote_date = datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d")
return {
"ts_code": ts_code,
"name": row.get("f14") or symbol,
"pre_close": previous_close,
"open": _number(row.get("f17")),
"high": _number(row.get("f15")),
"low": _number(row.get("f16")),
"close": close,
"vol": _number(row.get("f5")) * 100,
"amount": _number(row.get("f6")),
"num": 0,
"quote_date": quote_date,
"quote_time_epoch": epoch,
"source": "eastmoney_clist",
}
def _eastmoney_flow_wan(value: Any) -> float | None:
if value in (None, "", "-"):
return None
amount = _number(value, default=float("nan"))
if amount != amount:
return None
return amount / 10000
def _board_clock(value: Any) -> str:
digits = "".join(character for character in str(value or "") if character.isdigit())
if len(digits) >= 6:
return f"{digits[:2]}:{digits[2:4]}:{digits[4:6]}"
if len(digits) == 5:
digits = digits.zfill(6)
return f"{digits[:2]}:{digits[2:4]}:{digits[4:6]}"
if len(digits) == 4:
return f"{digits[:2]}:{digits[2:]}:00"
return ""
def _normalize_eastmoney_limit_row(row: dict[str, Any], limit_type: str) -> dict[str, Any] | None:
symbol = str(row.get("c") or row.get("code") or "").strip()
if not symbol.isdigit() or len(symbol) != 6:
return None
market = int(_number(row.get("m") if row.get("m") not in (None, "") else row.get("market")))
if market == 1 or symbol.startswith(("5", "6", "9")):
ts_code = f"{symbol}.SH"
elif symbol.startswith(("4", "8")):
ts_code = f"{symbol}.BJ"
else:
ts_code = f"{symbol}.SZ"
first_time = _board_clock(row.get("fbt") if row.get("fbt") not in (None, "") else row.get("first_time"))
last_time = _board_clock(row.get("lbt") if row.get("lbt") not in (None, "") else row.get("last_time"))
fund = row.get("fund")
if fund in (None, ""):
fund = row.get("fd_amount")
return {
"ts_code": ts_code,
"name": row.get("n") or row.get("name") or symbol,
"limit_type": limit_type,
"first_time": first_time or None,
"last_time": last_time or None,
"open_times": int(_number(row.get("zbc") if row.get("zbc") not in (None, "") else row.get("open_times"))),
"limit_times": max(1, int(_number(row.get("lbc") if row.get("lbc") not in (None, "") else 1))),
"turnover_ratio": _number(row.get("hs") if row.get("hs") not in (None, "") else row.get("turnover_ratio")),
"fd_amount": _number(fund) if fund not in (None, "", "-") else None,
"source": "eastmoney_zt_pool",
}
def _normalize_sector(value: Any) -> str: def _normalize_sector(value: Any) -> str:
text = str(value or "").strip().replace(" ", "") text = str(value or "").strip().replace(" ", "")
for suffix in ("板块", "概念", "行业", "", "", "(A股)", "A股)"): for suffix in ("板块", "概念", "行业", "", "", "(A股)", "A股)"):
+25 -9
View File
@@ -283,9 +283,9 @@ class HeavenMarketContextMixin:
) -> dict[str, Any] | None: ) -> dict[str, Any] | None:
"""Return the Shenwan L2 sector context for heaven trend. """Return the Shenwan L2 sector context for heaven trend.
观势行业层只使用申万二级行业。外显盘中使用 rt_sw_k、历史使用 观势行业层只使用申万二级行业。外显优先使用已发布的 sw_daily
sw_daily;内核独立使用目标日期成分股行情聚合。收盘过渡期在 盘中及收盘过渡期使用免费申万实时行情;内核使用数据中枢或免费
sw_daily 入库前接受同日15:00后的 rt_sw_k 收盘快照 实时成分行情。不再调用无权限的 rt_sw_k / rt_k
""" """
cache_key = f"{trade_date}:{identifier.strip().lower()}" cache_key = f"{trade_date}:{identifier.strip().lower()}"
cached = self.database.get_data_snapshot("heaven_sector", cache_key) cached = self.database.get_data_snapshot("heaven_sector", cache_key)
@@ -299,6 +299,14 @@ class HeavenMarketContextMixin:
and not cached.get("realtime") and not cached.get("realtime")
and int(cached.get("schema_version") or 0) >= 6 and int(cached.get("schema_version") or 0) >= 6
) )
cached_quotes = int((cached or {}).get("quote_count") or 0)
cached_lkg = bool(
cached
and cached_date == trade_date
and cached.get("taxonomy") == "sw_l2"
and cached_quotes > 0
and int(cached.get("schema_version") or 0) >= 6
)
if market_mode != "intraday" and cached_valid: if market_mode != "intraday" and cached_valid:
return cached return cached
if not self.configured: if not self.configured:
@@ -311,8 +319,12 @@ class HeavenMarketContextMixin:
allow_realtime_close=market_mode == "closed", allow_realtime_close=market_mode == "closed",
) )
except TushareError as exc: except TushareError as exc:
if cached_valid: if cached_lkg:
return cached delayed = dict(cached)
delayed["delayed"] = True
delayed["delay_notice"] = "主备免费行情均暂不可用,显示最近一次真实快照"
delayed["realtime"] = market_mode == "intraday"
return delayed
return { return {
"name": "", "name": "",
"code": "", "code": "",
@@ -323,12 +335,16 @@ class HeavenMarketContextMixin:
"precise": False, "precise": False,
"inner_precise": False, "inner_precise": False,
"outer_precise": False, "outer_precise": False,
"coverage": 0,
"member_count": 0,
"quote_count": 0,
"error": f"申万二级行业数据获取失败:{exc}", "error": f"申万二级行业数据获取失败:{exc}",
} }
if not payload.get("realtime") and payload.get("precise"): if int(payload.get("quote_count") or 0) > 0:
self.database.save_data_snapshot(
"heaven_sector",
cache_key,
str(payload.get("source") or "tushare"),
payload,
)
elif not payload.get("realtime") and payload.get("precise"):
self.database.save_data_snapshot( self.database.save_data_snapshot(
"heaven_sector", "heaven_sector",
cache_key, cache_key,
+4 -1
View File
@@ -243,6 +243,7 @@ class HeavenTrendMixin:
"detail": ( "detail": (
f"申万二级 {sector.get('name') or '--'} {sector.get('code') or '--'} " f"申万二级 {sector.get('name') or '--'} {sector.get('code') or '--'} "
f"成分覆盖 {int(sector.get('quote_count') or 0)}/{int(sector.get('member_count') or 0)}" f"成分覆盖 {int(sector.get('quote_count') or 0)}/{int(sector.get('member_count') or 0)}"
+ (";延迟快照" if sector.get("delayed") or sector.get("delay_notice") else "")
), ),
}, },
{ {
@@ -341,7 +342,9 @@ class HeavenTrendMixin:
issues.append("行业外显缺少申万官方行情") issues.append("行业外显缺少申万官方行情")
if sector and sector_coverage_issue: if sector and sector_coverage_issue:
issues.append(sector_coverage_issue) issues.append(sector_coverage_issue)
if sector.get("realtime") and not sector.get("relative_turnover"): if sector.get("delay_notice"):
issues.append(str(sector.get("delay_notice")))
if sector.get("realtime") and not sector.get("relative_turnover") and not sector.get("delayed"):
issues.append("行业内核缺少相对全市场换手活跃度") issues.append("行业内核缺少相对全市场换手活跃度")
stock = stock or {} stock = stock or {}
+196 -39
View File
@@ -2,6 +2,7 @@ from __future__ import annotations
import http.client import http.client
import json import json
import logging
import re import re
import time import time
import urllib.error import urllib.error
@@ -13,14 +14,20 @@ from threading import Lock
from typing import Any, ClassVar from typing import Any, ClassVar
from backend.bootstrap.config import tushare_code as _stock_market_code from backend.bootstrap.config import tushare_code as _stock_market_code
from backend.data.providers.ifind_client import IfindError, IfindHttpClient from backend.data.providers.ifind_client import IfindError
LOGGER = logging.getLogger("xiaobai.charts")
class ChartDataError(RuntimeError): class ChartDataError(RuntimeError):
pass pass
DAILY_CHART_LIMIT = 45
TRENDS_URL = "https://push2delay.eastmoney.com/api/qt/stock/trends2/get" TRENDS_URL = "https://push2delay.eastmoney.com/api/qt/stock/trends2/get"
HIS_TRENDS_URL = "https://push2his.eastmoney.com/api/qt/stock/trends2/get"
BOARD_LIST_URL = "https://push2delay.eastmoney.com/api/qt/clist/get" BOARD_LIST_URL = "https://push2delay.eastmoney.com/api/qt/clist/get"
BROWSER_USER_AGENT = ( BROWSER_USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
@@ -35,55 +42,186 @@ INDEX_SECIDS = {
class MarketChartClient: class MarketChartClient:
"""Prefer iFinD for display charts and retain Eastmoney as a last resort.""" """Display charts are served by the data hub only."""
def __init__(self, ifind: IfindHttpClient, fallback: "EastmoneyChartClient") -> None: def __init__(self, datahub: Any = None) -> None:
self.ifind = ifind self.datahub = datahub
self.fallback = fallback self.ifind = None
self.fallback = None
def stock_intraday(self, code: str) -> dict[str, Any]: def stock_intraday(self, code: str) -> dict[str, Any]:
normalized = str(code or "").strip() normalized = str(code or "").strip()
if not re.fullmatch(r"\d{6}", normalized): if not re.fullmatch(r"\d{6}", normalized):
raise ChartDataError("Invalid stock code") raise ChartDataError("Invalid stock code")
ifind_code = _stock_market_code(normalized) hub_chart = self._datahub_intraday(normalized)
try: if hub_chart is not None:
return self._ifind_intraday(ifind_code, "stock", normalized) return hub_chart
except (IfindError, ChartDataError): raise ChartDataError("分时图数据中枢暂不可用")
return self.fallback.stock_intraday(normalized)
def stock_daily(self, code: str, end_date: str, limit: int = 90) -> list[dict[str, Any]]: def stock_daily(self, code: str, end_date: str, limit: int = DAILY_CHART_LIMIT) -> list[dict[str, Any]]:
normalized = str(code or "").strip() normalized = str(code or "").strip()
if not re.fullmatch(r"\d{6}", normalized): if not re.fullmatch(r"\d{6}", normalized):
raise ChartDataError("Invalid stock code") raise ChartDataError("Invalid stock code")
return self._ifind_daily(_stock_market_code(normalized), end_date, limit) hub_rows = self._datahub_daily(normalized, end_date, limit, "daily")
if hub_rows:
return hub_rows
raise ChartDataError("日K数据中枢暂不可用")
def index_daily(self, identifier: str, end_date: str, limit: int = 90) -> list[dict[str, Any]]: def index_daily(self, identifier: str, end_date: str, limit: int = DAILY_CHART_LIMIT) -> list[dict[str, Any]]:
normalized = str(identifier or "").strip().upper() normalized = str(identifier or "").strip().upper()
if normalized not in INDEX_SECIDS: if normalized not in INDEX_SECIDS:
raise ChartDataError("Unsupported index") raise ChartDataError("Unsupported index")
return self._ifind_daily(normalized, end_date, limit) hub_rows = self._datahub_daily(normalized, end_date, limit, "index_daily")
if hub_rows:
return hub_rows
raise ChartDataError("指数日K数据中枢暂不可用")
def board_daily(self, identifier: str, end_date: str, limit: int = 90) -> list[dict[str, Any]]: def board_daily(self, identifier: str, end_date: str, limit: int = 90) -> list[dict[str, Any]]:
normalized = str(identifier or "").strip().upper() normalized = str(identifier or "").strip().upper()
if not normalized: if not normalized:
raise ChartDataError("Invalid board code") raise ChartDataError("Invalid board code")
return self._ifind_daily(normalized, end_date, limit) hub_rows = self._datahub_daily(normalized, end_date, limit, "sector_daily")
if hub_rows:
return hub_rows
raise ChartDataError("板块日K数据中枢暂不可用")
def index_intraday(self, identifier: str) -> dict[str, Any]: def index_intraday(self, identifier: str) -> dict[str, Any]:
normalized = str(identifier or "").strip().upper() normalized = str(identifier or "").strip().upper()
if normalized not in INDEX_SECIDS: if normalized not in INDEX_SECIDS:
raise ChartDataError("Unsupported index") raise ChartDataError("Unsupported index")
hub_chart = self._datahub_intraday(normalized)
if hub_chart is not None:
return hub_chart
raise ChartDataError("指数分时数据中枢暂不可用")
def _datahub_intraday(self, code: str) -> dict[str, Any] | None:
if self.datahub is None:
return None
try: try:
return self._ifind_intraday(normalized, "index", normalized) chart = self.datahub.try_intraday(code)
except (IfindError, ChartDataError): except Exception as exc:
return self.fallback.index_intraday(normalized) LOGGER.warning("datahub intraday unexpected error: %s", exc)
return None
if not chart:
return None
points = list(chart.get("points") or [])
if not points:
return None
return chart
def _datahub_daily(
self,
code: str,
end_date: str,
limit: int,
dataset: str,
) -> list[dict[str, Any]] | None:
if self.datahub is None or not hasattr(self.datahub, "try_daily_chart"):
return None
try:
rows = self.datahub.try_daily_chart(code, end_date, limit, dataset)
except Exception as exc:
LOGGER.warning("datahub daily unexpected error: %s", exc)
rows = None
if not rows:
return None
compact_end = str(end_date or "").replace("-", "")
market_now = datetime.now().astimezone()
today = market_now.strftime("%Y%m%d")
market_open = (
market_now.weekday() < 5
and market_now.time().replace(tzinfo=None) >= dt_time(9, 30)
)
if compact_end == today and market_open:
overlay = self._datahub_today_bar(code, dataset, rows)
if overlay:
if rows and rows[-1]["trade_date"] == overlay["trade_date"]:
rows[-1] = overlay
else:
rows.append(overlay)
return rows
def _datahub_today_bar(
self,
code: str,
dataset: str,
history: list[dict[str, Any]],
) -> dict[str, Any] | None:
today_display = datetime.now().astimezone().date().isoformat()
previous = history[-1]["close"] if history and history[-1]["trade_date"] != today_display else (
history[-2]["close"] if len(history) >= 2 else 0.0
)
quote = None
if dataset == "index_daily" and hasattr(self.datahub, "try_index_quotes"):
quotes = self.datahub.try_index_quotes() or []
quote = next(
(
item for item in quotes
if str(item.get("ts_code") or "") == code or str(item.get("code") or "") == code.split(".")[0]
),
None,
)
elif hasattr(self.datahub, "try_quotes"):
quotes = self.datahub.try_quotes([code]) or []
quote = quotes[0] if quotes else None
if quote:
close = _number(quote.get("close") if quote.get("close") not in (None, "") else quote.get("price"))
open_price = _number(quote.get("open"))
high = _number(quote.get("high"))
low = _number(quote.get("low"))
previous_close = _number(
quote.get("pre_close") if quote.get("pre_close") not in (None, "") else quote.get("previous_close")
) or previous
volume = _number(quote.get("vol") if quote.get("vol") not in (None, "") else quote.get("volume"))
amount = _number(quote.get("amount"))
if close > 0 and open_price > 0:
return {
"trade_date": today_display,
"open": open_price,
"high": high or close,
"low": low or close,
"close": close,
"change": round((close / previous_close - 1) * 100, 4) if previous_close else 0.0,
"volume": volume,
"amount_billion": amount / 100_000_000,
"realtime": True,
}
chart = self._datahub_intraday(code)
points = list((chart or {}).get("points") or [])
if not points:
return None
closes = [_number(point.get("close")) for point in points if _number(point.get("close")) > 0]
if not closes:
return None
opens = [_number(point.get("open")) for point in points if _number(point.get("open")) > 0]
highs = [_number(point.get("high")) for point in points if _number(point.get("high")) > 0]
lows = [_number(point.get("low")) for point in points if _number(point.get("low")) > 0]
volume = sum(_number(point.get("volume")) for point in points)
amount = sum(_number(point.get("amount")) for point in points)
previous_close = _number((chart or {}).get("previous_close")) or previous
close = closes[-1]
open_price = opens[0] if opens else closes[0]
return {
"trade_date": today_display,
"open": open_price,
"high": max(highs or closes),
"low": min(lows or closes),
"close": close,
"change": round((close / previous_close - 1) * 100, 4) if previous_close else 0.0,
"volume": volume,
"amount_billion": amount / 100_000_000,
"realtime": True,
}
def board_intraday(self, identifier: str, name: str = "") -> dict[str, Any]: def board_intraday(self, identifier: str, name: str = "") -> dict[str, Any]:
normalized = str(identifier or "").strip().upper() normalized = str(identifier or "").strip().upper()
try: hub_chart = self._datahub_intraday(normalized)
return self._ifind_intraday(normalized, "board", normalized, name) if hub_chart is not None:
except (IfindError, ChartDataError): if name:
return self.fallback.board_intraday(normalized, name) hub_chart = dict(hub_chart)
hub_chart["name"] = name
return hub_chart
raise ChartDataError("板块分时数据中枢暂不可用")
def _ifind_intraday( def _ifind_intraday(
self, self,
@@ -92,7 +230,7 @@ class MarketChartClient:
identifier: str, identifier: str,
name: str = "", name: str = "",
) -> dict[str, Any]: ) -> dict[str, Any]:
if not self.ifind.configured: if not self.ifind or not self.ifind.configured:
raise ChartDataError("iFinD is not configured") raise ChartDataError("iFinD is not configured")
now = datetime.now().astimezone() now = datetime.now().astimezone()
rows: list[dict[str, Any]] = [] rows: list[dict[str, Any]] = []
@@ -129,7 +267,7 @@ class MarketChartClient:
def _ifind_daily( def _ifind_daily(
self, ifind_code: str, end_date: str, limit: int self, ifind_code: str, end_date: str, limit: int
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
if not self.ifind.configured: if not self.ifind or not self.ifind.configured:
raise ChartDataError("iFinD is not configured") raise ChartDataError("iFinD is not configured")
compact_end = str(end_date or "").replace("-", "") compact_end = str(end_date or "").replace("-", "")
if not re.fullmatch(r"\d{8}", compact_end): if not re.fullmatch(r"\d{8}", compact_end):
@@ -231,9 +369,11 @@ class MarketChartClient:
pass pass
if not normalized: if not normalized:
raise ChartDataError("No iFinD daily chart data returned") raise ChartDataError("No iFinD daily chart data returned")
return normalized[-max(20, min(180, int(limit))):] return normalized[-max(1, int(limit)):]
def _previous_close(self, code: str, trade_date: str, fallback: float) -> float: def _previous_close(self, code: str, trade_date: str, fallback: float) -> float:
if not self.ifind:
return fallback
today = datetime.now().astimezone().date().isoformat() today = datetime.now().astimezone().date().isoformat()
if trade_date == today: if trade_date == today:
try: try:
@@ -305,21 +445,29 @@ class EastmoneyChartClient:
if cached is not None: if cached is not None:
return cached return cached
payload = self._request_json( params = {
TRENDS_URL, "secid": secid,
{ "fields1": "f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,f13",
"secid": secid, "fields2": "f51,f52,f53,f54,f55,f56,f57,f58",
"fields1": "f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,f13", "iscr": "0",
"fields2": "f51,f52,f53,f54,f55,f56,f57,f58", }
"iscr": "0", last_error: Exception | None = None
"ndays": "1", data: dict[str, Any] = {}
}, points: list[dict[str, Any]] = []
"https://quote.eastmoney.com/", for url, ndays in ((TRENDS_URL, "1"), (TRENDS_URL, "5"), (HIS_TRENDS_URL, "5")):
) request_params = {**params, "ndays": ndays}
data = payload.get("data") or {} try:
points = [point for raw in data.get("trends") or [] if (point := _parse_trend(raw))] payload = self._request_json(url, request_params, "https://quote.eastmoney.com/")
except ChartDataError as exc:
last_error = exc
continue
data = payload.get("data") or {}
parsed = [point for raw in data.get("trends") or [] if (point := _parse_trend(raw))]
points = _latest_session(parsed)
if points:
break
if not points: if not points:
raise ChartDataError("No intraday chart data returned") raise ChartDataError("No intraday chart data returned") from last_error
result = { result = {
"entity_type": entity_type, "entity_type": entity_type,
@@ -433,6 +581,15 @@ class EastmoneyChartClient:
raise ChartDataError("Intraday chart request failed") from last_error raise ChartDataError("Intraday chart request failed") from last_error
def _latest_session(points: list[dict[str, Any]]) -> list[dict[str, Any]]:
if not points:
return []
latest = max(str(point.get("date") or "") for point in points)
if not latest:
return points
return [point for point in points if str(point.get("date") or "") == latest]
def _parse_trend(raw: Any) -> dict[str, Any] | None: def _parse_trend(raw: Any) -> dict[str, Any] | None:
fields = str(raw or "").split(",") fields = str(raw or "").split(",")
if len(fields) < 8 or " " not in fields[0]: if len(fields) < 8 or " " not in fields[0]:
+319 -49
View File
@@ -14,7 +14,9 @@ from backend.bootstrap.config import (
validate_text, validate_text,
) )
from backend.data.providers.ifind_client import IfindError from backend.data.providers.ifind_client import IfindError
from backend.data.providers.tushare_client import TushareClient, TushareError from backend.data.providers.tushare_client import TushareError
from backend.data.providers.tushare_helpers import _moneyflow_payload, _optional_number
from backend.data.realtime import RealtimeAggregateError
from backend.features.market.backfill_history import ( from backend.features.market.backfill_history import (
DEFAULT_RECENT_TRADING_DAYS, DEFAULT_RECENT_TRADING_DAYS,
MAX_RANGE_TRADING_DAYS, MAX_RANGE_TRADING_DAYS,
@@ -26,7 +28,7 @@ from backend.features.market.backfill_history import (
select_open_trade_dates, select_open_trade_dates,
select_open_trade_dates_in_range, select_open_trade_dates_in_range,
) )
from backend.features.market.charts import ChartDataError from backend.features.market.charts import ChartDataError, DAILY_CHART_LIMIT
from backend.features.market.insights import MarketInsightsService from backend.features.market.insights import MarketInsightsService
from backend.features.sentiment.engine import SENTIMENT_ENGINE_VERSION from backend.features.sentiment.engine import SENTIMENT_ENGINE_VERSION
@@ -42,6 +44,7 @@ SEARCH_TYPE_LABELS = {
"theme": "题材", "theme": "题材",
"index": "指数", "index": "指数",
} }
TODAY_DAILY_UNAVAILABLE_NOTICE = "今日日K暂不可用,仍显示最近收盘K线。"
THS_SEARCH_TYPES = { THS_SEARCH_TYPES = {
"I": ("sector", "行业板块"), "I": ("sector", "行业板块"),
"R": ("sector", "地域板块"), "R": ("sector", "地域板块"),
@@ -58,16 +61,40 @@ class MarketServiceMixin:
self._tushare_client(), self._tushare_client(),
ifind=self.ifind, ifind=self.ifind,
) )
def _tushare_client(self) -> TushareClient: def _tushare_client(self) -> Any:
override = getattr(self, "_market_client_override", None)
if override is not None:
return override
gateway = getattr(self, "data_gateway", None) gateway = getattr(self, "data_gateway", None)
if gateway is not None: if gateway is None:
return gateway.tushare() raise RuntimeError("数据中枢尚未装配。")
# Compatibility for isolated legacy unit-test service stubs. return gateway.tushare()
return TushareClient(self.token)
def _now(self) -> datetime:
clock = getattr(self, "clock", None)
if callable(clock):
return clock()
return datetime.now().astimezone()
def _is_requested_open_session(self, requested_date: str) -> bool:
now = self._now()
if requested_date != now.strftime("%Y%m%d"):
return False
if now.time().replace(tzinfo=None) < dt_time(9, 15):
return False
client = self._tushare_client() if self.configured else None
resolve = getattr(client, "resolve_trade_context", None) if client else None
if resolve is None:
return now.weekday() < 5
try:
trade_date, _ = resolve(requested_date)
except Exception:
return now.weekday() < 5
return str(trade_date or "") == requested_date
def get_dashboard(self, trade_date: str, force: bool = False) -> dict[str, Any]: def get_dashboard(self, trade_date: str, force: bool = False) -> dict[str, Any]:
normalized_date = normalize_date(trade_date) normalized_date = normalize_date(trade_date)
now = datetime.now().astimezone() now = self._now()
if ( if (
normalized_date == now.strftime("%Y%m%d") normalized_date == now.strftime("%Y%m%d")
and now.time().replace(tzinfo=None) < datetime.strptime("09:15", "%H:%M").time() and now.time().replace(tzinfo=None) < datetime.strptime("09:15", "%H:%M").time()
@@ -149,11 +176,10 @@ class MarketServiceMixin:
return "最近可用交易日" return "最近可用交易日"
return f"{int(compact[4:6])}{int(compact[6:8])}" return f"{int(compact[4:6])}{int(compact[6:8])}"
@classmethod def _preparing_display_notice(self, actual_date: str, requested_date: str) -> str:
def _preparing_display_notice(cls, actual_date: str, requested_date: str) -> str: shown = self._chinese_month_day(actual_date)
shown = cls._chinese_month_day(actual_date)
requested = str(requested_date or "").replace("-", "") requested = str(requested_date or "").replace("-", "")
if requested == date.today().strftime("%Y%m%d"): if requested == self._now().strftime("%Y%m%d"):
return f"今日数据正在准备,当前展示 {shown}" return f"今日数据正在准备,当前展示 {shown}"
return f"所选日期数据尚未到齐,当前展示 {shown}" return f"所选日期数据尚未到齐,当前展示 {shown}"
@@ -174,14 +200,14 @@ class MarketServiceMixin:
def _should_retry_incomplete_snapshot( def _should_retry_incomplete_snapshot(
self, snapshot: dict[str, Any], requested_date: str self, snapshot: dict[str, Any], requested_date: str
) -> bool: ) -> bool:
if requested_date != date.today().strftime("%Y%m%d"): if requested_date != self._now().strftime("%Y%m%d"):
return False return False
meta = snapshot.get("meta") or {} meta = snapshot.get("meta") or {}
incomplete = ( actual = str(meta.get("trade_date") or "").replace("-", "")
meta.get("limit_data_source") == "derived" stale_carry = bool(meta.get("carried_forward") or actual != requested_date)
or bool(meta.get("carried_forward")) if stale_carry and self._is_requested_open_session(requested_date):
or str(meta.get("trade_date") or "").replace("-", "") != requested_date return True
) incomplete = meta.get("limit_data_source") == "derived" or stale_carry
return incomplete and self._snapshot_age_seconds(meta) >= 60 return incomplete and self._snapshot_age_seconds(meta) >= 60
def _annotate_data_status(self, dashboard: dict[str, Any]) -> dict[str, Any]: def _annotate_data_status(self, dashboard: dict[str, Any]) -> dict[str, Any]:
@@ -199,6 +225,9 @@ class MarketServiceMixin:
else: else:
meta["data_status"] = "preparing" meta["data_status"] = "preparing"
meta["display_notice"] = self._preparing_display_notice(actual, requested) meta["display_notice"] = self._preparing_display_notice(actual, requested)
elif meta.get("realtime"):
meta["data_status"] = "intraday"
meta.setdefault("display_notice", "")
else: else:
meta["data_status"] = "official" meta["data_status"] = "official"
meta.setdefault("display_notice", "") meta.setdefault("display_notice", "")
@@ -225,9 +254,9 @@ class MarketServiceMixin:
normalized_date: str, normalized_date: str,
snapshot: dict[str, Any], snapshot: dict[str, Any],
) -> bool: ) -> bool:
if not self.configured or normalized_date != date.today().strftime("%Y%m%d"): if not self.configured or normalized_date != self._now().strftime("%Y%m%d"):
return False return False
now = datetime.now().astimezone() now = self._now()
local_time = now.time().replace(tzinfo=None) local_time = now.time().replace(tzinfo=None)
realtime_start = datetime.strptime("09:15", "%H:%M").time() realtime_start = datetime.strptime("09:15", "%H:%M").time()
morning_end = datetime.strptime("11:35", "%H:%M").time() morning_end = datetime.strptime("11:35", "%H:%M").time()
@@ -264,7 +293,10 @@ class MarketServiceMixin:
raise TushareError("公共行情尚未配置") raise TushareError("公共行情尚未配置")
dashboard = self._tushare_client().dashboard(normalized_date) dashboard = self._tushare_client().dashboard(normalized_date)
meta = dashboard.setdefault("meta", {}) meta = dashboard.setdefault("meta", {})
quote_source = str(meta.get("quote_source") or "")
meta["source"] = source meta["source"] = source
if quote_source:
meta["quote_source"] = quote_source
meta["requested_date"] = self._display_compact_date(normalized_date) meta["requested_date"] = self._display_compact_date(normalized_date)
if meta.get("limit_data_source") == "derived": if meta.get("limit_data_source") == "derived":
meta.setdefault( meta.setdefault(
@@ -276,6 +308,12 @@ class MarketServiceMixin:
actual_date = normalize_date( actual_date = normalize_date(
str(dashboard.get("meta", {}).get("trade_date") or normalized_date) str(dashboard.get("meta", {}).get("trade_date") or normalized_date)
) )
if actual_date != normalized_date and self._is_requested_open_session(
normalized_date
):
raise TushareError(
f"Intraday dashboard resolved {actual_date} instead of {normalized_date}"
)
self.database.save_snapshot(actual_date, source, dashboard) self.database.save_snapshot(actual_date, source, dashboard)
if actual_date != normalized_date: if actual_date != normalized_date:
dashboard.setdefault("meta", {}).update( dashboard.setdefault("meta", {}).update(
@@ -297,6 +335,30 @@ class MarketServiceMixin:
) )
return self._apply_reason_overrides(self._with_storage(dashboard, cached=False)) return self._apply_reason_overrides(self._with_storage(dashboard, cached=False))
except TushareError as exc: except TushareError as exc:
if self._is_requested_open_session(normalized_date):
existing = self.database.get_snapshot(normalized_date)
existing_date = str(
((existing or {}).get("meta") or {}).get("trade_date") or ""
).replace("-", "")
if existing and existing_date == normalized_date:
kept = copy.deepcopy(existing)
kept.setdefault("meta", {}).update(
{
"requested_date": self._display_compact_date(normalized_date),
}
)
self.database.finish_sync(
sync_id,
"fallback",
self._record_count(kept),
str(exc),
"tushare",
)
return self._apply_reason_overrides(
self._with_storage(kept, cached=True)
)
self.database.finish_sync(sync_id, "failed", message=str(exc))
raise ValueError("当天盘中行情暂时不可用,请稍后重试。") from exc
fallback = self.database.get_latest_real_snapshot(normalized_date) fallback = self.database.get_latest_real_snapshot(normalized_date)
if fallback: if fallback:
actual = str((fallback.get("meta") or {}).get("trade_date") or "") actual = str((fallback.get("meta") or {}).get("trade_date") or "")
@@ -613,7 +675,7 @@ class MarketServiceMixin:
"index_daily", "index_daily",
{ {
"ts_code": basic["id"], "ts_code": basic["id"],
"start_date": (end - timedelta(days=190)).strftime("%Y%m%d"), "start_date": (end - timedelta(days=400)).strftime("%Y%m%d"),
"end_date": resolved_date, "end_date": resolved_date,
}, },
"ts_code,trade_date,open,high,low,close,pct_chg,vol,amount", "ts_code,trade_date,open,high,low,close,pct_chg,vol,amount",
@@ -629,10 +691,10 @@ class MarketServiceMixin:
"change": float(row.get("pct_chg") or 0), "change": float(row.get("pct_chg") or 0),
"volume": float(row.get("vol") or 0), "volume": float(row.get("vol") or 0),
} }
for row in rows[-90:] for row in rows[-DAILY_CHART_LIMIT:]
] ]
try: try:
chart_series = self.chart_data.index_daily(str(basic["id"]), resolved_date, 90) chart_series = self.chart_data.index_daily(str(basic["id"]), resolved_date, DAILY_CHART_LIMIT)
if chart_series: if chart_series:
series = chart_series series = chart_series
except (AttributeError, ChartDataError): except (AttributeError, ChartDataError):
@@ -740,7 +802,7 @@ class MarketServiceMixin:
result = copy.deepcopy(payload) result = copy.deepcopy(payload)
now = datetime.now().astimezone() now = datetime.now().astimezone()
try: try:
result["prices"] = self.chart_data.stock_daily(code, requested_date, 90) result["prices"] = self.chart_data.stock_daily(code, requested_date, DAILY_CHART_LIMIT)
result["meta"] = {**(result.get("meta") or {}), "chart_source": "market_chart"} result["meta"] = {**(result.get("meta") or {}), "chart_source": "market_chart"}
except (AttributeError, ChartDataError): except (AttributeError, ChartDataError):
pass pass
@@ -752,27 +814,28 @@ class MarketServiceMixin:
"trade_date": f"{actual_date[:4]}-{actual_date[4:6]}-{actual_date[6:]}", "trade_date": f"{actual_date[:4]}-{actual_date[4:6]}-{actual_date[6:]}",
} }
today = now.strftime("%Y%m%d") today = now.strftime("%Y%m%d")
latest_bar = (result.get("prices") or [{}])[-1] if result.get("prices") else {}
official_today = (
actual_date == today and not bool(latest_bar.get("realtime"))
)
after_close = now.time().replace(tzinfo=None) >= dt_time(15, 0)
should_merge = ( should_merge = (
requested_date == today requested_date == today
and actual_date <= today and actual_date <= today
and now.weekday() < 5 and now.weekday() < 5
and now.time().replace(tzinfo=None) >= dt_time(9, 30) and now.time().replace(tzinfo=None) >= dt_time(9, 30)
and not (official_today and after_close)
) )
if should_merge: if should_merge:
quote = self._ifind_realtime_stock_quote(code) quote = self._resolve_today_daily_quote(code, today, result)
if quote and self._valid_realtime_stock_quote(quote, today): if quote and self._valid_realtime_stock_quote(quote, today):
self._merge_realtime_stock_detail(result, quote, requested_date) self._merge_realtime_stock_detail(result, quote, requested_date)
elif self.configured and actual_date < today: elif actual_date < today:
client = self._tushare_client() result["meta"] = {
try: **(result.get("meta") or {}),
resolved_date, _ = client.resolve_trade_context(requested_date) "notice": TODAY_DAILY_UNAVAILABLE_NOTICE,
if resolved_date == today: }
quote = client.realtime_stock_quote(tushare_code(code), requested_date) return self._enrich_stock_detail(result, requested_date)
if self._valid_realtime_stock_quote(quote, today):
self._merge_realtime_stock_detail(result, quote, requested_date)
except TushareError:
pass
return self._enrich_stock_detail(result)
@staticmethod @staticmethod
def _sanitize_stock_detail_prices( def _sanitize_stock_detail_prices(
@@ -886,6 +949,138 @@ class MarketServiceMixin:
"quote_time": str(row.get("time") or ""), "quote_time": str(row.get("time") or ""),
} }
def _resolve_today_daily_quote(
self, code: str, today: str, payload: dict[str, Any]
) -> dict[str, Any] | None:
quote = self._ifind_realtime_stock_quote(code)
if quote and self._valid_realtime_stock_quote(quote, today):
return quote
if self.configured:
try:
client = self._tushare_client()
resolve = getattr(client, "resolve_trade_context", None)
resolved = today
if callable(resolve):
resolved, _ = resolve(today)
if str(resolved or "") == today:
quote = client.realtime_stock_quote(tushare_code(code), today)
if self._valid_realtime_stock_quote(quote, today):
return quote
except TushareError:
pass
quote = self._free_realtime_stock_quote(code, today)
if quote and self._valid_realtime_stock_quote(quote, today):
return quote
return self._intraday_realtime_stock_quote(code, today, payload)
def _free_realtime_stock_quote(self, code: str, today: str) -> dict[str, Any] | None:
aggregator = getattr(self, "realtime_aggregator", None)
if aggregator is None:
return None
ts_code = tushare_code(code)
for loader in (
getattr(aggregator, "tencent_stock_quote", None),
getattr(aggregator, "eastmoney_stock_quote", None),
):
if not callable(loader):
continue
try:
row = loader(ts_code, expected_date=today)
except (RealtimeAggregateError, Exception):
continue
quote = self._quote_from_free_row(code, today, row)
if quote:
return quote
return None
def _quote_from_free_row(
self, code: str, today: str, row: dict[str, Any]
) -> dict[str, Any] | None:
price = float(row.get("close") or 0)
previous_close = float(row.get("pre_close") or 0)
if price <= 0 or previous_close <= 0:
return None
try:
name, sector = self._stock_identity(code, today)
except Exception:
name, sector = "--", "其他"
epoch = int(row.get("quote_time_epoch") or 0)
if epoch > 0:
quote_time = datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
else:
quote_date = str(row.get("quote_date") or today)
quote_time = f"{quote_date[:4]}-{quote_date[4:6]}-{quote_date[6:]}"
quote = {
"name": str(row.get("name") or name or "--"),
"sector": sector,
"price": price,
"open": float(row.get("open") or 0),
"high": float(row.get("high") or 0),
"low": float(row.get("low") or 0),
"change": round((price / previous_close - 1) * 100, 4),
"volume": float(row.get("vol") or 0),
"amount_billion": float(row.get("amount") or 0) / 100_000_000,
"turnover_rate": float(row.get("turnover_rate") or 0),
"quote_time": quote_time,
}
flow = _moneyflow_payload(row)
if flow.get("available"):
quote["moneyflow"] = flow
return quote
def _intraday_realtime_stock_quote(
self, code: str, today: str, payload: dict[str, Any]
) -> dict[str, Any] | None:
chart_data = getattr(self, "chart_data", None)
if chart_data is None:
return None
try:
chart = chart_data.stock_intraday(code)
except (AttributeError, ChartDataError, Exception):
return None
points = [
point
for point in list(chart.get("points") or [])
if str(point.get("date") or "").replace("-", "") == today
]
if not points:
return None
opens = [float(point.get("open") or 0) for point in points if float(point.get("open") or 0) > 0]
highs = [float(point.get("high") or 0) for point in points if float(point.get("high") or 0) > 0]
lows = [float(point.get("low") or 0) for point in points if float(point.get("low") or 0) > 0]
closes = [float(point.get("close") or 0) for point in points if float(point.get("close") or 0) > 0]
if not opens or not highs or not lows or not closes:
return None
price = closes[-1]
previous_close = float(chart.get("previous_close") or 0)
if previous_close <= 0:
history = list(payload.get("prices") or [])
previous_close = float((history[-1] if history else {}).get("close") or 0)
if previous_close <= 0:
return None
volume = sum(float(point.get("volume") or 0) for point in points)
amount = sum(float(point.get("amount") or 0) for point in points)
if volume <= 0 and amount <= 0:
return None
try:
name, sector = self._stock_identity(code, today)
except Exception:
name, sector = "--", "其他"
return {
"name": name,
"sector": sector,
"price": price,
"open": opens[0],
"high": max(highs),
"low": min(lows),
"change": round((price / previous_close - 1) * 100, 4),
"volume": volume,
"volume_unit": "lots",
"amount_billion": amount / 100_000_000,
"turnover_rate": 0.0,
"quote_time": str(points[-1].get("date") or today),
}
@staticmethod @staticmethod
def _merge_realtime_stock_detail( def _merge_realtime_stock_detail(
payload: dict[str, Any], quote: dict[str, Any], trade_date: str payload: dict[str, Any], quote: dict[str, Any], trade_date: str
@@ -907,23 +1102,29 @@ class MarketServiceMixin:
prices[-1] = realtime_bar prices[-1] = realtime_bar
else: else:
prices.append(realtime_bar) prices.append(realtime_bar)
payload["prices"] = prices[-90:] payload["prices"] = prices[-DAILY_CHART_LIMIT:]
stock = dict(payload.get("stock") or {}) stock = dict(payload.get("stock") or {})
stock.update( updates = {
{ "name": quote["name"],
"name": quote["name"], "industry": quote["sector"],
"industry": quote["sector"], "price": quote["price"],
"price": quote["price"], "change": quote["change"],
"change": quote["change"], "amount_billion": quote["amount_billion"],
"amount_billion": quote["amount_billion"], }
"turnover_rate": quote["turnover_rate"], quote_turnover = _optional_number(quote.get("turnover_rate"))
} if quote_turnover:
) updates["turnover_rate"] = quote_turnover
stock.update(updates)
payload["stock"] = stock payload["stock"] = stock
quote_flow = quote.get("moneyflow")
current_flow = payload.get("moneyflow") or {}
if isinstance(quote_flow, dict) and quote_flow.get("available") and not current_flow.get("available"):
payload["moneyflow"] = quote_flow
payload["meta"] = { payload["meta"] = {
**(payload.get("meta") or {}), **(payload.get("meta") or {}),
"trade_date": display_date, "trade_date": display_date,
"realtime": True, "realtime": True,
"notice": "",
"updated_at": datetime.now().astimezone().isoformat(timespec="seconds"), "updated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
} }
@@ -957,7 +1158,7 @@ class MarketServiceMixin:
intraday_status = "unavailable" intraday_status = "unavailable"
intraday_notice = "分时行情暂不可用,请稍后重试。" intraday_notice = "分时行情暂不可用,请稍后重试。"
prices = list(detail.get("prices") or [])[-60:] prices = list(detail.get("prices") or [])[-DAILY_CHART_LIMIT:]
stock = dict(detail.get("stock") or {"code": code}) stock = dict(detail.get("stock") or {"code": code})
realtime = bool(detail_meta.get("realtime")) realtime = bool(detail_meta.get("realtime"))
return { return {
@@ -1209,10 +1410,40 @@ class MarketServiceMixin:
return item["name"], item["sector"] or "其他" return item["name"], item["sector"] or "其他"
return "--", "其他" return "--", "其他"
def _enrich_stock_detail(self, payload: dict[str, Any]) -> dict[str, Any]: def _enrich_stock_detail(
self, payload: dict[str, Any], trade_date: str = ""
) -> dict[str, Any]:
result = dict(payload) result = dict(payload)
stock = dict(payload.get("stock") or {}) stock = dict(payload.get("stock") or {})
code = str(stock.get("code") or "") code = str(stock.get("code") or "")
compact_date = normalize_date(
str((payload.get("meta") or {}).get("trade_date") or trade_date)
)
board = self._limit_event_for_stock(code, compact_date)
if board:
if not stock.get("first_time") or stock.get("first_time") == "--":
stock["first_time"] = board.get("first_time") or "--"
if not stock.get("last_time") or stock.get("last_time") == "--":
stock["last_time"] = board.get("last_time") or "--"
if not stock.get("open_times"):
stock["open_times"] = board.get("open_times") or 0
if _optional_number(stock.get("seal_amount_million")) is None:
stock["seal_amount_million"] = board.get("seal_amount_million")
if not _optional_number(stock.get("turnover_rate")) and _optional_number(board.get("turnover_rate")):
stock["turnover_rate"] = board.get("turnover_rate")
flow = result.get("moneyflow") or {}
if not flow.get("available"):
live_flow = self._live_moneyflow_for_stock(code, compact_date)
if live_flow.get("available"):
result["moneyflow"] = live_flow
else:
result["moneyflow"] = {
"available": False,
"net_million": None,
"large_million": None,
"medium_million": None,
"small_million": None,
}
watched = { watched = {
item["code"]: item item["code"]: item
for item in self.database.list_watchlist(self.current_user_id) for item in self.database.list_watchlist(self.current_user_id)
@@ -1222,6 +1453,45 @@ class MarketServiceMixin:
result["notes"] = self.database.list_notes(self.current_user_id, code=code) result["notes"] = self.database.list_notes(self.current_user_id, code=code)
return result return result
def _limit_event_for_stock(self, code: str, trade_date: str) -> dict[str, Any]:
if not code or not trade_date:
return {}
ts_code = tushare_code(code)
client = self._tushare_client() if self.configured else None
rows: list[dict[str, Any]] = []
if client is not None:
try:
rows = client._load_limit_type(trade_date, "U") + client._load_limit_type(trade_date, "Z")
except Exception:
rows = []
if not rows:
try:
rows = list((client._free_board_map(trade_date) or {}).values())
except Exception:
rows = []
match = next((row for row in rows if str(row.get("ts_code") or "") == ts_code), None)
if not match:
return {}
fd = _optional_number(match.get("fd_amount"))
return {
"first_time": match.get("first_time") or "--",
"last_time": match.get("last_time") or "--",
"open_times": match.get("open_times") or 0,
"seal_amount_million": None if fd is None else round(fd / 10000, 0),
"turnover_rate": _optional_number(match.get("turnover_ratio")),
}
def _live_moneyflow_for_stock(self, code: str, trade_date: str) -> dict[str, Any]:
aggregator = getattr(self, "realtime_aggregator", None)
loader = getattr(aggregator, "eastmoney_stock_quote", None) if aggregator else None
if not callable(loader) or not code:
return _moneyflow_payload(None)
try:
quote = loader(tushare_code(code), expected_date=trade_date)
except Exception:
return _moneyflow_payload(None)
return _moneyflow_payload(quote)
def _with_storage(self, dashboard: dict[str, Any], cached: bool) -> dict[str, Any]: def _with_storage(self, dashboard: dict[str, Any], cached: bool) -> dict[str, Any]:
result = dict(dashboard) result = dict(dashboard)
result["meta"] = { result["meta"] = {
+42 -31
View File
@@ -4,7 +4,14 @@ import re
import secrets import secrets
from typing import Any from typing import Any
from backend.bootstrap.config import TOKEN_PATTERN, validate_text from backend.bootstrap.config import validate_text
MARKET_SOURCE_SECRET_KEYS = {
"tushare_token",
"ifind_refresh_token",
"ifind_access_token",
}
class SystemServiceMixin: class SystemServiceMixin:
@@ -18,9 +25,6 @@ class SystemServiceMixin:
first_encrypted = self.database.get_user_credentials(first_user_id) first_encrypted = self.database.get_user_credentials(first_user_id)
first_personal = self.vault.decrypt_json(first_encrypted) if first_encrypted else {} first_personal = self.vault.decrypt_json(first_encrypted) if first_encrypted else {}
defaults = { defaults = {
"tushare_token": environment.get("tushare_token") or first_personal.get("tushare_token") or "",
"ifind_refresh_token": environment.get("ifind_refresh_token") or "",
"ifind_access_token": environment.get("ifind_access_token") or "",
"platform_llm_primary_api_key": environment.get("platform_llm_primary_api_key") or first_personal.get("llm_primary_api_key") or "", "platform_llm_primary_api_key": environment.get("platform_llm_primary_api_key") or first_personal.get("llm_primary_api_key") or "",
"platform_llm_primary_base_url": environment.get("platform_llm_primary_base_url") or first_personal.get("llm_primary_base_url") or "https://api.openai.com/v1", "platform_llm_primary_base_url": environment.get("platform_llm_primary_base_url") or first_personal.get("llm_primary_base_url") or "https://api.openai.com/v1",
"platform_llm_primary_model": environment.get("platform_llm_primary_model") or first_personal.get("llm_primary_model") or "", "platform_llm_primary_model": environment.get("platform_llm_primary_model") or first_personal.get("llm_primary_model") or "",
@@ -34,6 +38,10 @@ class SystemServiceMixin:
if key not in current: if key not in current:
current[key] = value current[key] = value
changed = True changed = True
for key in MARKET_SOURCE_SECRET_KEYS:
if key in current:
current.pop(key, None)
changed = True
if not isinstance(current.get("llm_models"), list): if not isinstance(current.get("llm_models"), list):
migrated_models: list[dict[str, str]] = [] migrated_models: list[dict[str, str]] = []
for role, label in (("primary", "原主模型"), ("fallback", "原辅助模型")): for role, label in (("primary", "原主模型"), ("fallback", "原辅助模型")):
@@ -56,26 +64,27 @@ class SystemServiceMixin:
self.database.save_system_setting("credentials", self.vault.encrypt_json(current)) self.database.save_system_setting("credentials", self.vault.encrypt_json(current))
for row in self.database.list_user_credentials(): for row in self.database.list_user_credentials():
personal = self.vault.decrypt_json(str(row.get("encrypted_payload") or "")) personal = self.vault.decrypt_json(str(row.get("encrypted_payload") or ""))
if "tushare_token" in personal: if any(key in personal for key in MARKET_SOURCE_SECRET_KEYS):
personal.pop("tushare_token", None) for key in MARKET_SOURCE_SECRET_KEYS:
personal.pop(key, None)
self.database.save_user_credentials( self.database.save_user_credentials(
int(row["user_id"]), self.vault.encrypt_json(personal) int(row["user_id"]), self.vault.encrypt_json(personal)
) )
return current return current
def _save_system_credentials(self, credentials: dict[str, Any]) -> None: def _save_system_credentials(self, credentials: dict[str, Any]) -> None:
sanitized = {
key: value
for key, value in credentials.items()
if key not in MARKET_SOURCE_SECRET_KEYS
}
with self.system_lock: with self.system_lock:
self.database.save_system_setting("credentials", self.vault.encrypt_json(credentials)) self.database.save_system_setting("credentials", self.vault.encrypt_json(sanitized))
self._system_credentials = dict(credentials) self._system_credentials = dict(sanitized)
if hasattr(self, "ifind"):
self.ifind.set_credentials(
str(credentials.get("ifind_refresh_token") or ""),
str(credentials.get("ifind_access_token") or ""),
)
@property @property
def configured(self) -> bool: def configured(self) -> bool:
return bool(self.token) return bool(self._datahub_status().get("configured"))
def _credentials(self) -> dict[str, str]: def _credentials(self) -> dict[str, str]:
credentials = getattr(self._request_context, "credentials", {}) credentials = getattr(self._request_context, "credentials", {})
@@ -99,7 +108,7 @@ class SystemServiceMixin:
@property @property
def token(self) -> str: def token(self) -> str:
return str(self._system_credentials.get("tushare_token") or "") return "datahub" if self.configured else ""
def system_status(self) -> dict[str, Any]: def system_status(self) -> dict[str, Any]:
platform = self._platform_llm_profile() platform = self._platform_llm_profile()
@@ -130,6 +139,7 @@ class SystemServiceMixin:
), ),
**self.database.status(), **self.database.status(),
"jobs": self.jobs.repository.recent(12), "jobs": self.jobs.repository.recent(12),
"datahub": self._datahub_status(),
}, },
"llm": { "llm": {
"primary_configured": self._profile_configured(platform["primary"]), "primary_configured": self._profile_configured(platform["primary"]),
@@ -145,21 +155,24 @@ class SystemServiceMixin:
}, },
} }
def _datahub_status(self) -> dict[str, Any]:
gateway = getattr(self, "data_gateway", None)
reporter = getattr(gateway, "datahub_status", None)
if callable(reporter):
return reporter()
return {
"configured": False,
"base_url": "",
"enabled_reads": 0,
"total_reads": 0,
"flags": [],
"routes": [],
"fallback_count": 0,
"fallback_labels": [],
}
def save_system_settings(self, payload: dict[str, Any]) -> dict[str, Any]: def save_system_settings(self, payload: dict[str, Any]) -> dict[str, Any]:
current = dict(self._system_credentials) current = dict(self._system_credentials)
token = str(payload.get("tushare_token") or current.get("tushare_token") or "").strip()
if token and not TOKEN_PATTERN.fullmatch(token):
raise ValueError("Tushare Token 格式不正确。")
ifind_refresh_token = str(
payload.get("ifind_refresh_token")
or current.get("ifind_refresh_token")
or ""
).strip()
if ifind_refresh_token and (
len(ifind_refresh_token) > 2048
or any(character.isspace() for character in ifind_refresh_token)
):
raise ValueError("iFinD Refresh Token 格式不正确。")
existing_models = { existing_models = {
str(item.get("id") or ""): item str(item.get("id") or ""): item
for item in current.get("llm_models") or [] for item in current.get("llm_models") or []
@@ -221,8 +234,6 @@ class SystemServiceMixin:
raise ValueError("会员每日额度应为 1 至 1000。") from exc raise ValueError("会员每日额度应为 1 至 1000。") from exc
current.update( current.update(
{ {
"tushare_token": token,
"ifind_refresh_token": ifind_refresh_token,
"llm_models": models, "llm_models": models,
"primary_model_id": primary_model_id, "primary_model_id": primary_model_id,
"fallback_model_id": fallback_model_id, "fallback_model_id": fallback_model_id,
@@ -242,7 +253,7 @@ class SystemServiceMixin:
llm_access = self.llm_access_status() llm_access = self.llm_access_status()
return { return {
"configured": self.configured, "configured": self.configured,
"mode": "tushare" if self.configured else "unavailable", "mode": "datahub" if self.configured else "unavailable",
"llm_configured": self.llm_configured, "llm_configured": self.llm_configured,
"llm_model": self.llm_primary_model if self.llm_configured else "", "llm_model": self.llm_primary_model if self.llm_configured else "",
"llm_fallback_configured": self.llm_fallback_configured, "llm_fallback_configured": self.llm_fallback_configured,
+2
View File
@@ -41,6 +41,8 @@ def official_catchup_due(today: str, snapshot: dict[str, object]) -> bool:
actual == today actual == today
and meta.get("limit_data_source") != "derived" and meta.get("limit_data_source") != "derived"
and not meta.get("carried_forward") and not meta.get("carried_forward")
and not meta.get("realtime")
and meta.get("mode") != "realtime"
): ):
return False return False
return True return True
+2
View File
@@ -20,6 +20,8 @@ services:
DATAHUB_TOKEN: "${DATAHUB_TOKEN:?DATAHUB_TOKEN must be set}" DATAHUB_TOKEN: "${DATAHUB_TOKEN:?DATAHUB_TOKEN must be set}"
DATAHUB_ADMIN_PASSWORD: "${DATAHUB_ADMIN_PASSWORD:?DATAHUB_ADMIN_PASSWORD must be set}" DATAHUB_ADMIN_PASSWORD: "${DATAHUB_ADMIN_PASSWORD:?DATAHUB_ADMIN_PASSWORD must be set}"
TUSHARE_TOKEN: "${TUSHARE_TOKEN:-}" TUSHARE_TOKEN: "${TUSHARE_TOKEN:-}"
IFIND_REFRESH_TOKEN: "${IFIND_REFRESH_TOKEN:-}"
IFIND_ACCESS_TOKEN: "${IFIND_ACCESS_TOKEN:-}"
DATAHUB_DB_PATH: /app/data/datahub.db DATAHUB_DB_PATH: /app/data/datahub.db
DATAHUB_BACKUP_DIR: /app/data/backups DATAHUB_BACKUP_DIR: /app/data/backups
TZ: Asia/Shanghai TZ: Asia/Shanghai
+20
View File
@@ -13,6 +13,26 @@ services:
- ./.env - ./.env
environment: environment:
APP_ENCRYPTION_KEY: "${APP_ENCRYPTION_KEY:?APP_ENCRYPTION_KEY must be set in .env}" APP_ENCRYPTION_KEY: "${APP_ENCRYPTION_KEY:?APP_ENCRYPTION_KEY must be set in .env}"
# Provider credentials are consumed only by xiaobai-datahub.
TUSHARE_TOKEN: ""
IFIND_REFRESH_TOKEN: ""
IFIND_ACCESS_TOKEN: ""
DATAHUB_BASE_URL: "${DATAHUB_BASE_URL:-http://192.168.200.11:8766}"
DATAHUB_READ_CALENDAR: "1"
DATAHUB_READ_STOCKS: "1"
DATAHUB_READ_DAILY: "1"
DATAHUB_READ_INDEX_DAILY: "1"
DATAHUB_READ_VALUATION: "1"
DATAHUB_READ_MONEYFLOW: "1"
DATAHUB_READ_AUCTION: "1"
DATAHUB_READ_LIMIT_EVENTS: "1"
DATAHUB_READ_POPULARITY: "1"
DATAHUB_READ_DRAGON_TIGER: "1"
DATAHUB_READ_SECTOR_DAILY: "1"
DATAHUB_READ_QUOTES: "1"
DATAHUB_READ_INDEX_QUOTES: "1"
DATAHUB_READ_INTRADAY: "1"
DATAHUB_READ_STATUS: "1"
TZ: Asia/Shanghai TZ: Asia/Shanghai
PYTHONUTF8: "1" PYTHONUTF8: "1"
volumes: volumes:
+6 -3
View File
@@ -12,9 +12,12 @@ These registries describe the approved product surface of the standalone applica
providers, model entry points, CSS layers, and remaining code hotspots. providers, model entry points, CSS layers, and remaining code hotspots.
- `data-fields.config.json`: canonical data products, provider eligibility, intended use, and - `data-fields.config.json`: canonical data products, provider eligibility, intended use, and
known blocked datasets. known blocked datasets.
- `datahub.config.json`: optional read-only client for `xiaobai-datahub`. Each dataset has its - `datahub.config.json`: official read-only client for `xiaobai-datahub`. Each dataset has its
own `read` / `shadow` flag, all default off. Environment variables `DATAHUB_READ_*` and own `read` / `shadow` flag; official reads default on. `compose.yaml` pins every
`DATAHUB_SHADOW_*` can override a single dataset without a master switch. `DATAHUB_READ_*` to `"1"` so a leftover `.env` `=0` cannot silently keep official
pages on the old APIs. Environment variables can still override a single
`DATAHUB_SHADOW_*` without a master switch. The old website APIs stay as
emergency fallback only.
- `data-quality.config.json`: freshness, coverage, units, adjustment, point-in-time, and - `data-quality.config.json`: freshness, coverage, units, adjustment, point-in-time, and
fail-closed rules for every canonical data product. fail-closed rules for every canonical data product.
- `jobs.config.json`: background schedules, dependencies, lock keys, retry policy, timeouts, - `jobs.config.json`: background schedules, dependencies, lock keys, retry policy, timeouts,
+87 -92
View File
@@ -207,27 +207,22 @@
{ {
"provider": "datahub", "provider": "datahub",
"path": "backend/data/datahub/client.py", "path": "backend/data/datahub/client.py",
"runtime_role": "optional official EOD read path behind per-dataset flags" "runtime_role": "website-only read path; official EOD, live quotes, and licensed iFinD"
}, },
{ {
"provider": "ifind", "provider": "ifind",
"path": "backend/data/providers/ifind_client.py", "path": "xiaobai-datahub/datahub/adapters/ifind.py",
"runtime_role": "realtime, charts, snapshots, enrichment" "runtime_role": "licensed iFinD source inside the data hub"
}, },
{ {
"provider": "eastmoney", "provider": "eastmoney",
"path": "backend/features/market/charts.py", "path": "xiaobai-datahub/datahub/adapters/eastmoney.py",
"runtime_role": "display chart fallback" "runtime_role": "free realtime quotes and shenwan inside the data hub"
},
{
"provider": "eastmoney",
"path": "backend/data/realtime.py",
"runtime_role": "isolated realtime observation"
}, },
{ {
"provider": "tencent", "provider": "tencent",
"path": "backend/data/realtime.py", "path": "xiaobai-datahub/datahub/adapters/tencent.py",
"runtime_role": "index observation fallback" "runtime_role": "free index and stock quotes inside the data hub"
} }
], ],
"provider_domains": [ "provider_domains": [
@@ -296,16 +291,16 @@
"owner": "backend/data/gateway.py" "owner": "backend/data/gateway.py"
}, },
{ {
"client": "IfindHttpClient", "client": "HubIfindProxy",
"owner": "backend/data/gateway.py"
},
{
"client": "HubRealtimeProxy",
"owner": "backend/data/gateway.py" "owner": "backend/data/gateway.py"
}, },
{ {
"client": "MarketChartClient", "client": "MarketChartClient",
"owner": "backend/data/gateway.py" "owner": "backend/data/gateway.py"
},
{
"client": "WebRealtimeAggregator",
"owner": "backend/data/gateway.py"
} }
], ],
"heaven_service_owners": { "heaven_service_owners": {
@@ -468,8 +463,8 @@
}, },
{ {
"path": "frontend/pages/heaven/page.js", "path": "frontend/pages/heaven/page.js",
"bytes": 97268, "bytes": 97770,
"lines": 2070 "lines": 2079
}, },
{ {
"path": "frontend/shared/shell.css", "path": "frontend/shared/shell.css",
@@ -483,8 +478,13 @@
}, },
{ {
"path": "frontend/index.html", "path": "frontend/index.html",
"bytes": 48254, "bytes": 48403,
"lines": 664 "lines": 665
},
{
"path": "backend/data/providers/tushare_industries.py",
"bytes": 37168,
"lines": 859
}, },
{ {
"path": "backend/features/screener/catalog.py", "path": "backend/features/screener/catalog.py",
@@ -496,6 +496,11 @@
"bytes": 35247, "bytes": 35247,
"lines": 2416 "lines": 2416
}, },
{
"path": "backend/data/providers/tushare_dashboard.py",
"bytes": 33230,
"lines": 770
},
{ {
"path": "database.py", "path": "database.py",
"bytes": 32073, "bytes": 32073,
@@ -506,16 +511,6 @@
"bytes": 31756, "bytes": 31756,
"lines": 562 "lines": 562
}, },
{
"path": "backend/data/providers/tushare_dashboard.py",
"bytes": 28234,
"lines": 648
},
{
"path": "backend/data/providers/tushare_industries.py",
"bytes": 26540,
"lines": 616
},
{ {
"path": "backend/features/heaven/manual.py", "path": "backend/features/heaven/manual.py",
"bytes": 24521, "bytes": 24521,
@@ -533,13 +528,13 @@
}, },
{ {
"path": "frontend/pages/market/preview.js", "path": "frontend/pages/market/preview.js",
"bytes": 18178, "bytes": 18230,
"lines": 446 "lines": 447
}, },
{ {
"path": "backend/features/heaven/trend.py", "path": "backend/features/heaven/trend.py",
"bytes": 16772, "bytes": 17005,
"lines": 370 "lines": 373
}, },
{ {
"path": "backend/features/market/insights_auction_scoring.py", "path": "backend/features/market/insights_auction_scoring.py",
@@ -548,33 +543,33 @@
}, },
{ {
"path": "frontend/pages/market/charts.js", "path": "frontend/pages/market/charts.js",
"bytes": 15311, "bytes": 15743,
"lines": 387 "lines": 401
},
{
"path": "frontend/shared/dashboard.js",
"bytes": 15063,
"lines": 321
}, },
{ {
"path": "frontend/pages/pools/page.html", "path": "frontend/pages/pools/page.html",
"bytes": 14942, "bytes": 14942,
"lines": 235 "lines": 235
}, },
{
"path": "frontend/shared/admin.js",
"bytes": 14836,
"lines": 283
},
{ {
"path": "backend/features/screener/data_sync.py", "path": "backend/features/screener/data_sync.py",
"bytes": 14743, "bytes": 14743,
"lines": 342 "lines": 342
}, },
{
"path": "frontend/shared/dashboard.js",
"bytes": 14740,
"lines": 316
},
{
"path": "frontend/shared/admin.js",
"bytes": 14410,
"lines": 268
},
{ {
"path": "backend/features/heaven/market_context.py", "path": "backend/features/heaven/market_context.py",
"bytes": 13681, "bytes": 14409,
"lines": 338 "lines": 354
}, },
{ {
"path": "frontend/shared/session.js", "path": "frontend/shared/session.js",
@@ -588,8 +583,8 @@
}, },
{ {
"path": "backend/features/system/service.py", "path": "backend/features/system/service.py",
"bytes": 12392, "bytes": 12180,
"lines": 254 "lines": 265
}, },
{ {
"path": "backend/features/market/insights_auction.py", "path": "backend/features/market/insights_auction.py",
@@ -613,9 +608,14 @@
}, },
{ {
"path": "frontend/pages/market/entity-detail.js", "path": "frontend/pages/market/entity-detail.js",
"bytes": 9119, "bytes": 9139,
"lines": 199 "lines": 199
}, },
{
"path": "backend/data/providers/tushare_daily.py",
"bytes": 9076,
"lines": 232
},
{ {
"path": "backend/data/providers/tushare_dragon_tiger.py", "path": "backend/data/providers/tushare_dragon_tiger.py",
"bytes": 9059, "bytes": 9059,
@@ -626,6 +626,11 @@
"bytes": 8562, "bytes": 8562,
"lines": 238 "lines": 238
}, },
{
"path": "backend/data/providers/tushare_indices.py",
"bytes": 8447,
"lines": 189
},
{ {
"path": "frontend/pages/mentor/page.html", "path": "frontend/pages/mentor/page.html",
"bytes": 8357, "bytes": 8357,
@@ -636,16 +641,6 @@
"bytes": 6983, "bytes": 6983,
"lines": 146 "lines": 146
}, },
{
"path": "backend/data/providers/tushare_daily.py",
"bytes": 6837,
"lines": 160
},
{
"path": "backend/application.py",
"bytes": 6751,
"lines": 178
},
{ {
"path": "backend/features/market/insights_popularity.py", "path": "backend/features/market/insights_popularity.py",
"bytes": 6739, "bytes": 6739,
@@ -662,8 +657,13 @@
"lines": 81 "lines": 81
}, },
{ {
"path": "backend/data/providers/tushare_stocks.py", "path": "backend/application.py",
"bytes": 6244, "bytes": 6399,
"lines": 161
},
{
"path": "frontend/pages/market/stock-detail.js",
"bytes": 6325,
"lines": 137 "lines": 137
}, },
{ {
@@ -682,14 +682,9 @@
"lines": 85 "lines": 85
}, },
{ {
"path": "frontend/pages/market/stock-detail.js", "path": "backend/data/providers/tushare_stocks.py",
"bytes": 5690, "bytes": 5592,
"lines": 124 "lines": 123
},
{
"path": "backend/data/providers/tushare_indices.py",
"bytes": 5451,
"lines": 118
}, },
{ {
"path": "frontend/pages.config.js", "path": "frontend/pages.config.js",
@@ -721,6 +716,11 @@
"bytes": 4712, "bytes": 4712,
"lines": 106 "lines": 106
}, },
{
"path": "backend/data/providers/tushare_helpers.py",
"bytes": 4406,
"lines": 124
},
{ {
"path": "backend/features/market/routes.py", "path": "backend/features/market/routes.py",
"bytes": 4276, "bytes": 4276,
@@ -796,26 +796,21 @@
"bytes": 2299, "bytes": 2299,
"lines": 57 "lines": 57
}, },
{
"path": "backend/data/providers/tushare_client.py",
"bytes": 2263,
"lines": 70
},
{ {
"path": "backend/features/screener/regime.py", "path": "backend/features/screener/regime.py",
"bytes": 2202, "bytes": 2202,
"lines": 53 "lines": 53
}, },
{
"path": "backend/data/providers/tushare_client.py",
"bytes": 2166,
"lines": 68
},
{ {
"path": "frontend/pages/popularity/page.html", "path": "frontend/pages/popularity/page.html",
"bytes": 2165, "bytes": 2165,
"lines": 35 "lines": 35
}, },
{
"path": "backend/data/providers/tushare_helpers.py",
"bytes": 2083,
"lines": 64
},
{ {
"path": "frontend/pages/market/breadth.js", "path": "frontend/pages/market/breadth.js",
"bytes": 2071, "bytes": 2071,
@@ -827,13 +822,13 @@
"lines": 45 "lines": 45
}, },
{ {
"path": "backend/features/system/routes.py", "path": "backend/jobs/refresh.py",
"bytes": 1791, "bytes": 1808,
"lines": 46 "lines": 48
}, },
{ {
"path": "backend/jobs/refresh.py", "path": "backend/features/system/routes.py",
"bytes": 1728, "bytes": 1791,
"lines": 46 "lines": 46
}, },
{ {
@@ -846,6 +841,11 @@
"bytes": 1642, "bytes": 1642,
"lines": 53 "lines": 53
}, },
{
"path": "backend/data/providers/tushare_transport.py",
"bytes": 1592,
"lines": 50
},
{ {
"path": "backend/features/market/insights.py", "path": "backend/features/market/insights.py",
"bytes": 1580, "bytes": 1580,
@@ -856,11 +856,6 @@
"bytes": 1535, "bytes": 1535,
"lines": 39 "lines": 39
}, },
{
"path": "backend/data/providers/tushare_transport.py",
"bytes": 1455,
"lines": 48
},
{ {
"path": "backend/features/themes/routes.py", "path": "backend/features/themes/routes.py",
"bytes": 1337, "bytes": 1337,
+15 -8
View File
@@ -6,13 +6,20 @@
"page_limit": 5000, "page_limit": 5000,
"stale_seconds_max": 86400, "stale_seconds_max": 86400,
"datasets": { "datasets": {
"calendar": { "read": false, "shadow": false }, "calendar": { "read": true, "shadow": false },
"stocks": { "read": false, "shadow": false }, "stocks": { "read": true, "shadow": false },
"daily": { "read": false, "shadow": false }, "daily": { "read": true, "shadow": false },
"index_daily": { "read": false, "shadow": false }, "index_daily": { "read": true, "shadow": false },
"valuation": { "read": false, "shadow": false }, "valuation": { "read": true, "shadow": false },
"moneyflow": { "read": false, "shadow": false }, "moneyflow": { "read": true, "shadow": false },
"auction": { "read": false, "shadow": false }, "auction": { "read": true, "shadow": false },
"status": { "read": false, "shadow": false } "limit_events": { "read": true, "shadow": false },
"popularity": { "read": true, "shadow": false },
"dragon_tiger": { "read": true, "shadow": false },
"sector_daily": { "read": true, "shadow": false },
"quotes": { "read": true, "shadow": false },
"index_quotes": { "read": true, "shadow": false },
"intraday": { "read": true, "shadow": false },
"status": { "read": true, "shadow": false }
} }
} }
+2 -2
View File
@@ -213,12 +213,12 @@
{ {
"provider": "eastmoney", "provider": "eastmoney",
"path": "realtime_aggregator.py", "path": "realtime_aggregator.py",
"runtime_role": "isolated realtime observation" "runtime_role": "isolated realtime observation and intraday dashboard fallback"
}, },
{ {
"provider": "tencent", "provider": "tencent",
"path": "realtime_aggregator.py", "path": "realtime_aggregator.py",
"runtime_role": "index observation fallback" "runtime_role": "index observation and intraday quote fallback"
} }
], ],
"llm_entrypoints": [ "llm_entrypoints": [
@@ -320,6 +320,11 @@ PC端统一采用以下固定骨架:
### 6.1 数据源职责 ### 6.1 数据源职责
运行边界:下表中的职责全部由独立的 `xiaobai-datahub` 数据中枢执行。主网站只按固定业务
协议请求“行情、日K、分时、申万、竞价”等数据,不接触任何提供方参数或凭据,也不决定优先级、
重试、降级和回填。数据中枢是主网站唯一的行情出口;中枢暂时取不到新数据时,网站只能读取
已经归档的真实快照,不能绕回旧提供方直连接口。
| 数据源 | 可用于正式计算 | 主要职责 | | 数据源 | 可用于正式计算 | 主要职责 |
|---|:---:|---| |---|:---:|---|
| Tushare | 是 | 交易日历、股票主表、日线、估值、财务、资金流、申万行业、涨跌停、9:25竞价、热榜、龙虎榜 | | Tushare | 是 | 交易日历、股票主表、日线、估值、财务、资金流、申万行业、涨跌停、9:25竞价、热榜、龙虎榜 |
+4 -3
View File
@@ -607,12 +607,13 @@
<div class="admin-panel" data-admin-panel="market"> <div class="admin-panel" data-admin-panel="market">
<form id="systemMarketForm" class="settings-section"> <form id="systemMarketForm" class="settings-section">
<div class="settings-section-heading"><h3>公共行情</h3><span id="systemDataStatus">待检查</span></div> <div class="settings-section-heading"><h3>公共行情</h3><span id="systemDataStatus">待检查</span></div>
<label class="form-field"><span>Tushare Token</span><input id="systemTokenInput" type="password" autocomplete="off" minlength="20" placeholder="留空保留现有 Token"></label> <label class="form-field"><span>行情来源凭据</span><input id="systemTokenInput" type="text" value="请在数据中枢后台统一管理" disabled></label>
<label class="form-field"><span>iFinD Refresh Token</span><input id="systemIfindTokenInput" type="password" autocomplete="off" maxlength="2048" placeholder="留空保留现有 Token"></label> <label class="form-field"><span>实时来源凭据</span><input id="systemIfindTokenInput" type="text" value="请在数据中枢后台统一管理" disabled></label>
<label class="switch-control"><input id="systemBackgroundRefresh" type="checkbox"><span>启用交易时段后台刷新</span></label> <label class="switch-control"><input id="systemBackgroundRefresh" type="checkbox"><span>启用交易时段后台刷新</span></label>
<p class="form-hint">所有用户读取同一份后台快照,页面不会随后台任务自动重绘。</p> <p class="form-hint">所有用户读取同一份后台快照,页面不会随后台任务自动重绘。</p>
<div id="datahubRouteStatus" class="admin-refresh-status" data-tone="idle" role="status" aria-live="polite"><i data-lucide="database"></i><span>数据中枢线路待检查</span></div>
<div id="adminRefreshStatus" class="admin-refresh-status" data-tone="idle" role="status" aria-live="polite"><i data-lucide="circle-dot"></i><span>尚未手动刷新</span></div> <div id="adminRefreshStatus" class="admin-refresh-status" data-tone="idle" role="status" aria-live="polite"><i data-lucide="circle-dot"></i><span>尚未手动刷新</span></div>
<div class="dialog-actions admin-inline-actions"><button id="adminRefreshButton" class="button" type="button"><i data-lucide="refresh-cw"></i>立即后台刷新</button><button class="button primary" type="submit">保存行情配</button></div> <div class="dialog-actions admin-inline-actions"><button id="adminRefreshButton" class="button" type="button"><i data-lucide="refresh-cw"></i>立即后台刷新</button><button class="button primary" type="submit">保存刷新设</button></div>
</form> </form>
<section class="settings-section"> <section class="settings-section">
<div class="settings-section-heading"><h3>历史数据回补</h3><span>管理员任务</span></div> <div class="settings-section-heading"><h3>历史数据回补</h3><span>管理员任务</span></div>
+20 -14
View File
@@ -3402,9 +3402,9 @@
const payload = detail && detail.payload ? detail.payload : {}; const payload = detail && detail.payload ? detail.payload : {};
const meta = payload.meta || {}; const meta = payload.meta || {};
if (tab === "daily") { if (tab === "daily") {
const bars = (payload.prices || []).slice(-48); const bars = (payload.prices || []).slice(-45);
const last = bars.length ? bars[bars.length - 1].trade_date : ""; const last = bars.length ? bars[bars.length - 1].trade_date : "";
return "日线 · 近48根 · 至 " + (displayCompactDate(last) || "--"); return "日线 · 近45根 · 至 " + (displayCompactDate(last) || "--");
} }
const d = displayCompactDate(meta.intraday_trade_date) || displayCompactDate(meta.trade_date); const d = displayCompactDate(meta.intraday_trade_date) || displayCompactDate(meta.trade_date);
return "分时 · " + (d || "--"); return "分时 · " + (d || "--");
@@ -3702,7 +3702,7 @@
const W = 360, H = 240, padL = 8, padR = 52, padT = 10, padB = 22; const W = 360, H = 240, padL = 8, padR = 52, padT = 10, padB = 22;
const pw = W - padL - padR; const pw = W - padL - padR;
const ph = H - padT - padB; const ph = H - padT - padB;
const prices = (payload.prices || []).slice(-48); const prices = (payload.prices || []).slice(-45);
if (prices.length < 2) return emptyChart("日线数据暂不可用"); if (prices.length < 2) return emptyChart("日线数据暂不可用");
@@ -5219,6 +5219,15 @@
return '<span class="m-sys-dot' + (ok ? " m-sys-dot--ok" : "") + '"></span>'; return '<span class="m-sys-dot' + (ok ? " m-sys-dot--ok" : "") + '"></span>';
} }
function datahubStatusText(hub) {
const enabled = number(hub.enabled_reads);
const total = number(hub.total_reads) || enabled;
const fallbacks = hub.fallback_labels || [];
if (fallbacks.length) return " 备用 " + fallbacks.join("、");
if (hub.configured) return " 主线路 " + enabled + "/" + total;
return " 未配置";
}
function renderSystemAdmin(key) { function renderSystemAdmin(key) {
if (key === "system/members") { if (key === "system/members") {
renderSystemMembers(); renderSystemMembers();
@@ -5233,15 +5242,16 @@
'<div class="m-sys-body" data-system-admin-panel="market">' + '<div class="m-sys-body" data-system-admin-panel="market">' +
'<div class="m-card m-sys-section"><strong>数据源状态</strong>' + '<div class="m-card m-sys-section"><strong>数据源状态</strong>' +
'<div class="m-sys-status-list">' + '<div class="m-sys-status-list">' +
'<div class="m-sys-status-item"><span>Tushare</span><span>' + statusDot(data.configured) + (data.configured ? " 已配置" : " 未配置") + "</span></div>" + '<div class="m-sys-status-item"><span>数据中枢</span><span>' + statusDot(data.configured) + (data.configured ? " 已连接" : " 未连接") + "</span></div>" +
'<div class="m-sys-status-item"><span>iFinD</span><span>' + statusDot(ifind.configured) + (ifind.configured ? " 已配置" : " 未配置") + "</span></div>" + '<div class="m-sys-status-item"><span>iFinD</span><span>' + statusDot(ifind.configured) + (ifind.configured ? " 已配置" : " 未配置") + "</span></div>" +
'<div class="m-sys-status-item"><span>行情快照</span><strong>' + number(data.snapshot_dates) + " 个交易日</strong></div>" + '<div class="m-sys-status-item"><span>行情快照</span><strong>' + number(data.snapshot_dates) + " 个交易日</strong></div>" +
'<div class="m-sys-status-item"><span>后台刷新</span><span>' + statusDot(data.background_refresh_enabled) + (data.background_refresh_enabled ? " 已启用" : " 已暂停") + "</span></div>" + '<div class="m-sys-status-item"><span>后台刷新</span><span>' + statusDot(data.background_refresh_enabled) + (data.background_refresh_enabled ? " 已启用" : " 已暂停") + "</span></div>" +
'<div class="m-sys-status-item"><span>数据中枢</span><span>' + statusDot(Boolean((data.datahub || {}).configured) && !((data.datahub || {}).fallback_count)) + datahubStatusText(data.datahub || {}) + "</span></div>" +
"</div></div>" + "</div></div>" +
'<div class="m-card m-sys-section"><strong>数据源密钥</strong>' + '<div class="m-card m-sys-section"><strong>数据源密钥</strong>' +
formFieldHtml("Tushare Token", '<input id="m-sys-token" type="password" autocomplete="off" minlength="20" placeholder="留空则保留现有 Token">', false) + formFieldHtml("行情来源凭据", '<input id="m-sys-token" type="text" value="请在数据中枢后台统一管理" disabled>', false) +
formFieldHtml("iFinD Refresh Token", '<input id="m-sys-ifind" type="password" autocomplete="off" maxlength="2048" placeholder="留空则保留现有 Token">', false) + formFieldHtml("实时来源凭据", '<input id="m-sys-ifind" type="text" value="请在数据中枢后台统一管理" disabled>', false) +
'<button class="m-btn-primary" type="button" data-system-save-market>保存密钥</button></div>' + '<button class="m-btn-primary" type="button" data-system-save-market>刷新状态</button></div>' +
'<div class="m-card m-sys-section"><strong>后台刷新</strong>' + '<div class="m-card m-sys-section"><strong>后台刷新</strong>' +
'<div class="m-sys-switch-row"><div><strong>交易时段自动刷新</strong><p class="m-sys-hint">开启后后台定时更新快照</p></div>' + '<div class="m-sys-switch-row"><div><strong>交易时段自动刷新</strong><p class="m-sys-hint">开启后后台定时更新快照</p></div>' +
'<button class="m-theme-switch" type="button" data-system-toggle-refresh role="switch" aria-checked="' + (data.background_refresh_enabled ? "true" : "false") + '" aria-label="交易时段自动刷新"><span class="m-theme-switch-thumb"></span></button></div>' + '<button class="m-theme-switch" type="button" data-system-toggle-refresh role="switch" aria-checked="' + (data.background_refresh_enabled ? "true" : "false") + '" aria-label="交易时段自动刷新"><span class="m-theme-switch-thumb"></span></button></div>' +
@@ -5540,14 +5550,10 @@
function saveSystemMarket() { function saveSystemMarket() {
const button = document.querySelector("[data-system-save-market]"); const button = document.querySelector("[data-system-save-market]");
if (button) button.disabled = true; if (button) button.disabled = true;
global.MobileAPI.request("/api/admin/settings", "POST", { loadSystem().then(function () {
tushare_token: ((document.getElementById("m-sys-token") || {}).value || "").trim(), showToast("数据中枢状态已刷新");
ifind_refresh_token: ((document.getElementById("m-sys-ifind") || {}).value || "").trim(),
}).then(function () {
showToast("行情密钥已保存");
loadSystem();
}).catch(function (error) { }).catch(function (error) {
showToast(error && error.message ? error.message : "系统配置保存失败"); showToast(error && error.message ? error.message : "数据中枢状态读取失败");
}).then(function () { }).then(function () {
if (button) button.disabled = false; if (button) button.disabled = false;
}); });
+4 -4
View File
@@ -68,10 +68,10 @@
"/pages/sentiment/page.js?v=20260729-1", "/pages/sentiment/page.js?v=20260729-1",
"/pages/pools/page.js?v=20260820-1", "/pages/pools/page.js?v=20260820-1",
"/pages/market/breadth.js?v=20260803-1", "/pages/market/breadth.js?v=20260803-1",
"/pages/market/charts.js?v=20260803-1", "/pages/market/charts.js?v=20260908-1",
"/pages/market/entity-detail.js?v=20260803-1", "/pages/market/entity-detail.js?v=20260908-1",
"/pages/market/stock-detail.js?v=20260803-1", "/pages/market/stock-detail.js?v=20260908-1",
"/pages/market/preview.js?v=20260806-1", "/pages/market/preview.js?v=20260908-1",
"/pages/market/search.js?v=20260803-1", "/pages/market/search.js?v=20260803-1",
"/pages/market/bindings.js?v=20260803-1", "/pages/market/bindings.js?v=20260803-1",
"/pages/ladder/page.js?v=20260820-1", "/pages/ladder/page.js?v=20260820-1",
+13 -4
View File
@@ -113,9 +113,12 @@ async function loadHeavenSetup(force = false, sector = "", stockCode = "") {
document.querySelector("#resetHeavenCalibrationButton"), document.querySelector("#resetHeavenCalibrationButton"),
].filter(Boolean); ].filter(Boolean);
cancelHeavenPerformance(); cancelHeavenPerformance();
heavenView?.classList.add("heaven-data-loading"); const blocking = !state.heavenSetup;
if (blocking) heavenView?.classList.add("heaven-data-loading");
if (loadButton) loadButton.disabled = true; if (loadButton) loadButton.disabled = true;
calibrationButtons.forEach((button) => { button.disabled = true; }); calibrationButtons.forEach((button) => { button.disabled = true; });
const controller = new AbortController();
const timeoutId = window.setTimeout(() => controller.abort(), 25_000);
try { try {
if (state.heavenSetup?.requestedKey && state.heavenSetup.requestedKey !== requestedKey) { if (state.heavenSetup?.requestedKey && state.heavenSetup.requestedKey !== requestedKey) {
state.personalField = null; state.personalField = null;
@@ -126,7 +129,7 @@ async function loadHeavenSetup(force = false, sector = "", stockCode = "") {
if (sector) query.set("sector", sector); if (sector) query.set("sector", sector);
if (stockCode) query.set("stock_code", stockCode); if (stockCode) query.set("stock_code", stockCode);
if (manualData) query.set("manual_data", JSON.stringify(manualData)); if (manualData) query.set("manual_data", JSON.stringify(manualData));
const payload = await apiRequest(`/api/heaven/setup?${query}`); const payload = await apiRequest(`/api/heaven/setup?${query}`, "GET", null, { signal: controller.signal });
if ( if (
requestSequence !== state.heavenRequestSequence requestSequence !== state.heavenRequestSequence
|| calendarDate !== document.querySelector("#qiObservationDate")?.value || calendarDate !== document.querySelector("#qiObservationDate")?.value
@@ -152,9 +155,15 @@ async function loadHeavenSetup(force = false, sector = "", stockCode = "") {
if (payload.chart.selection_notice) showHeavenNotice(payload.chart.selection_notice); if (payload.chart.selection_notice) showHeavenNotice(payload.chart.selection_notice);
} catch (error) { } catch (error) {
if (requestSequence !== state.heavenRequestSequence) return; if (requestSequence !== state.heavenRequestSequence) return;
showHeavenNotice(error.message || "问天数据加载失败"); const aborted = error?.payload?.aborted || /abort|超时|cancel/i.test(String(error?.message || ""));
showToast(error.message || "问天数据加载失败"); const message = aborted
? "问天数据仍在准备,页面可继续输入和操作"
: (error.message || "问天数据加载失败");
showHeavenNotice(message);
if (!aborted) showToast(message);
if (!state.heavenSetup) renderHeavenWorkspace();
} finally { } finally {
window.clearTimeout(timeoutId);
if (requestSequence === state.heavenRequestSequence) { if (requestSequence === state.heavenRequestSequence) {
heavenView?.classList.remove("heaven-data-loading"); heavenView?.classList.remove("heaven-data-loading");
if (loadButton) loadButton.disabled = false; if (loadButton) loadButton.disabled = false;
+23 -9
View File
@@ -1,3 +1,16 @@
const DAILY_CHART_BARS = 45;
function visibleDailyPrices(prices) {
return (prices || []).slice(-DAILY_CHART_BARS);
}
function dailyChartSourceLabel(prices, notice) {
const count = visibleDailyPrices(prices).length;
const base = `日 K 行情 · ${count} 个交易日`;
const text = String(notice || "").trim();
return text ? `${base} · ${text}` : base;
}
function currentChartPalette() { function currentChartPalette() {
const style = getComputedStyle(document.documentElement); const style = getComputedStyle(document.documentElement);
const color = (token, fallback) => style.getPropertyValue(token).trim() || fallback; const color = (token, fallback) => style.getPropertyValue(token).trim() || fallback;
@@ -56,7 +69,8 @@ function drawCandlestick(context, x, item, priceY, candleWidth, palette = curren
function drawPriceChart(prices) { function drawPriceChart(prices) {
const canvas = elements.priceChart; const canvas = elements.priceChart;
if (!prices?.length) { const visible = visibleDailyPrices(prices);
if (!visible.length) {
clearPriceChart("暂无日 K 数据"); clearPriceChart("暂无日 K 数据");
return; return;
} }
@@ -81,15 +95,15 @@ function drawPriceChart(prices) {
const gap = 12; const gap = 12;
const priceBottom = height - bottom - volumeHeight - gap; const priceBottom = height - bottom - volumeHeight - gap;
const plotWidth = width - left - right; const plotWidth = width - left - right;
const highs = prices.map((item) => number(item.high)); const highs = visible.map((item) => number(item.high));
const lows = prices.map((item) => number(item.low)); const lows = visible.map((item) => number(item.low));
const maximum = Math.max(...highs); const maximum = Math.max(...highs);
const minimum = Math.min(...lows); const minimum = Math.min(...lows);
const range = Math.max(maximum - minimum, maximum * 0.01, 0.01); const range = Math.max(maximum - minimum, maximum * 0.01, 0.01);
const volumes = prices.map((item) => number(item.volume)); const volumes = visible.map((item) => number(item.volume));
const maxVolume = Math.max(...volumes, 1); const maxVolume = Math.max(...volumes, 1);
const priceY = (value) => top + (maximum - value) / range * (priceBottom - top); const priceY = (value) => top + (maximum - value) / range * (priceBottom - top);
const step = plotWidth / prices.length; const step = plotWidth / visible.length;
const candleWidth = clamp(step * 0.62, 2, 8); const candleWidth = clamp(step * 0.62, 2, 8);
context.strokeStyle = palette.grid; context.strokeStyle = palette.grid;
@@ -105,7 +119,7 @@ function drawPriceChart(prices) {
context.fillText((maximum - range * line / 4).toFixed(2), left - 5, y + 4); context.fillText((maximum - range * line / 4).toFixed(2), left - 5, y + 4);
} }
prices.forEach((item, index) => { visible.forEach((item, index) => {
const x = left + step * index + step / 2; const x = left + step * index + step / 2;
const color = drawCandlestick(context, x, item, priceY, candleWidth, palette); const color = drawCandlestick(context, x, item, priceY, candleWidth, palette);
const volumeBarHeight = number(item.volume) / maxVolume * volumeHeight; const volumeBarHeight = number(item.volume) / maxVolume * volumeHeight;
@@ -117,10 +131,10 @@ function drawPriceChart(prices) {
context.textAlign = "center"; context.textAlign = "center";
context.fillStyle = palette.axis; context.fillStyle = palette.axis;
const labelIndexes = [0, Math.floor((prices.length - 1) / 2), prices.length - 1]; const labelIndexes = [0, Math.floor((visible.length - 1) / 2), visible.length - 1];
labelIndexes.forEach((index) => { labelIndexes.forEach((index) => {
const x = left + step * index + step / 2; const x = left + step * index + step / 2;
context.fillText(String(prices[index].trade_date).slice(5), x, height - 5); context.fillText(String(visible[index].trade_date).slice(5), x, height - 5);
}); });
} }
@@ -301,7 +315,7 @@ function drawIntradayPreviewChart(points, dailyPrices, referenceClose = 0) {
function drawDailyPreviewChart(prices) { function drawDailyPreviewChart(prices) {
const { context, width, height, palette } = prepareStockPreviewCanvas(); const { context, width, height, palette } = prepareStockPreviewCanvas();
const visible = prices.slice(-45); const visible = visibleDailyPrices(prices);
const visibleStart = prices.length - visible.length; const visibleStart = prices.length - visible.length;
const left = 45; const left = 45;
const right = 10; const right = 10;
+2 -2
View File
@@ -113,13 +113,13 @@ function renderEntityDetailMetrics(metrics) {
} }
function drawEntityDetailChart(series, canvas = elements.entityDetailChart) { function drawEntityDetailChart(series, canvas = elements.entityDetailChart) {
const candles = (series || []).filter((item) => number(item.close) > 0).map((item) => { const candles = visibleDailyPrices((series || []).filter((item) => number(item.close) > 0).map((item) => {
const close = number(item.close); const close = number(item.close);
const open = number(item.open) || close; const open = number(item.open) || close;
const high = Math.max(number(item.high) || close, open, close); const high = Math.max(number(item.high) || close, open, close);
const low = Math.min(number(item.low) || close, open, close); const low = Math.min(number(item.low) || close, open, close);
return { ...item, open, high, low, close }; return { ...item, open, high, low, close };
}); }));
if (!candles.length) { if (!candles.length) {
clearEntityDetailChart("暂无日 K 数据", canvas); clearEntityDetailChart("暂无日 K 数据", canvas);
return; return;
+2 -1
View File
@@ -367,7 +367,8 @@ function selectStockPreviewChart(chart) {
} }
} else if ((payload.prices || []).length) { } else if ((payload.prices || []).length) {
setText("stockPreviewDate", payload.meta?.trade_date || "最新行情"); setText("stockPreviewDate", payload.meta?.trade_date || "最新行情");
setText("stockPreviewSource", `日 K 行情 · ${payload.prices.length} 个交易日`); const notice = String(payload.meta?.notice || "").trim();
setText("stockPreviewSource", dailyChartSourceLabel(payload.prices, notice));
drawDailyPreviewChart(payload.prices); drawDailyPreviewChart(payload.prices);
} else { } else {
setText("stockPreviewDate", payload.meta?.trade_date || "最新行情"); setText("stockPreviewDate", payload.meta?.trade_date || "最新行情");
+25 -12
View File
@@ -20,17 +20,9 @@ async function openStock(code, fallback = null) {
setText("detailStreak", row.status === "涨停" ? streakLabel(row.streak) : row.status || "--"); setText("detailStreak", row.status === "涨停" ? streakLabel(row.streak) : row.status || "--");
setText("detailReason", row.reason || "--"); setText("detailReason", row.reason || "--");
setText("detailSector", row.sector || "其他"); setText("detailSector", row.sector || "其他");
setText("detailFirst", row.first_time || "--"); setStockBoardFields(row);
setText("detailLast", row.last_time || "--");
setText("detailOpen", `${number(row.open_times)}`);
setText("detailTurnover", `${formatNumber(row.turnover_rate, 2)}%`);
setText("detailAmount", `${formatNumber(row.amount_billion, 2)} 亿`);
setText("detailSeal", `${formatNumber(row.seal_amount_million, 0)}`);
setText("chartSource", "正在加载行情"); setText("chartSource", "正在加载行情");
setText("flowNet", "--"); renderMoneyflow({});
setText("flowLarge", "--");
setText("flowMedium", "--");
setText("flowSmall", "--");
document.querySelector("#reasonInput").value = row.reason || ""; document.querySelector("#reasonInput").value = row.reason || "";
document.querySelector("#stockNoteContent").value = ""; document.querySelector("#stockNoteContent").value = "";
document.querySelector("#stockNotePlan").value = ""; document.querySelector("#stockNotePlan").value = "";
@@ -48,11 +40,13 @@ async function openStock(code, fallback = null) {
setText("detailName", stock.name || row.name); setText("detailName", stock.name || row.name);
setText("detailPrice", formatNumber(stock.price || row.price, 2)); setText("detailPrice", formatNumber(stock.price || row.price, 2));
setText("detailChange", `${signed(stock.change ?? row.change)}%`); setText("detailChange", `${signed(stock.change ?? row.change)}%`);
setStockBoardFields({ ...row, ...stock });
renderMoneyflow(payload.moneyflow || {}); renderMoneyflow(payload.moneyflow || {});
renderStockNotes(payload.notes || []); renderStockNotes(payload.notes || []);
updateWatchButton(); updateWatchButton();
if (state.stockDetailChartMode === "daily") { if (state.stockDetailChartMode === "daily") {
setText("chartSource", `日 K 行情 · ${payload.prices.length} 个交易日`); const notice = String(payload.meta?.notice || "").trim();
setText("chartSource", dailyChartSourceLabel(payload.prices, notice));
requestAnimationFrame(() => drawPriceChart(payload.prices || [])); requestAnimationFrame(() => drawPriceChart(payload.prices || []));
} }
} catch (error) { } catch (error) {
@@ -69,7 +63,11 @@ async function selectStockDetailChart(mode) {
syncDetailChartButtons("stock", selected); syncDetailChartButtons("stock", selected);
if (selected === "daily") { if (selected === "daily") {
const prices = state.stockDetail?.prices || []; const prices = state.stockDetail?.prices || [];
setText("chartSource", prices.length ? `日 K 行情 · ${prices.length} 个交易日` : "正在加载行情"); const notice = String(state.stockDetail?.meta?.notice || "").trim();
setText(
"chartSource",
prices.length ? dailyChartSourceLabel(prices, notice) : "正在加载行情",
);
if (prices.length) requestAnimationFrame(() => drawPriceChart(prices)); if (prices.length) requestAnimationFrame(() => drawPriceChart(prices));
else clearPriceChart("正在加载日 K 数据"); else clearPriceChart("正在加载日 K 数据");
return; return;
@@ -111,6 +109,21 @@ function renderStockDetailIntraday(payload) {
}); });
} }
function setStockBoardFields(row) {
const firstTime = String(row.first_time || "").trim();
const lastTime = String(row.last_time || "").trim();
setText("detailFirst", firstTime && firstTime !== "--" ? firstTime : "--");
setText("detailLast", lastTime && lastTime !== "--" ? lastTime : "--");
setText("detailOpen", row.open_times === null || row.open_times === undefined || row.open_times === "" ? "--" : `${number(row.open_times)}`);
setText("detailTurnover", presentMetric(row.turnover_rate) ? `${formatNumber(row.turnover_rate, 2)}%` : "--");
setText("detailAmount", presentMetric(row.amount_billion) ? `${formatNumber(row.amount_billion, 2)} 亿` : "--");
setText("detailSeal", presentMetric(row.seal_amount_million) ? `${formatNumber(row.seal_amount_million, 0)}` : "--");
}
function presentMetric(value) {
return meaningfulNumber(value) && Number(value) !== 0;
}
function openActiveStockInHeaven() { function openActiveStockInHeaven() {
const code = state.activeStock?.code; const code = state.activeStock?.code;
if (!/^\d{6}$/.test(String(code || ""))) return; if (!/^\d{6}$/.test(String(code || ""))) return;
+11 -1
View File
@@ -408,8 +408,18 @@ async function saveReasonOverride(event) {
} }
function renderMoneyflow(flow) { function renderMoneyflow(flow) {
for (const [id, value] of [["flowNet", flow.net_million], ["flowLarge", flow.large_million], ["flowMedium", flow.medium_million], ["flowSmall", flow.small_million]]) { const payload = flow || {};
const available = payload.available !== false && [
payload.net_million, payload.large_million, payload.medium_million, payload.small_million,
].some((value) => value !== null && value !== undefined && value !== "");
for (const [id, value] of [["flowNet", payload.net_million], ["flowLarge", payload.large_million], ["flowMedium", payload.medium_million], ["flowSmall", payload.small_million]]) {
const element = document.getElementById(id); const element = document.getElementById(id);
if (!element) continue;
if (!available || value === null || value === undefined || value === "") {
element.textContent = "--";
element.className = "";
continue;
}
element.textContent = formatMoneyMillion(value); element.textContent = formatMoneyMillion(value);
element.className = changeClass(value); element.className = changeClass(value);
} }
+23 -8
View File
@@ -41,11 +41,10 @@ async function openAdminSettings(refreshOnly = false) {
const ifind = data.ifind || {}; const ifind = data.ifind || {};
const llm = payload.llm || {}; const llm = payload.llm || {};
const membership = payload.membership || {}; const membership = payload.membership || {};
status.textContent = `Tushare ${data.configured ? "已配置" : "未配置"} · iFinD ${ifind.configured ? "已配置" : "未配置"} · ${number(data.snapshot_dates)} 个交易日`; status.textContent = `数据中枢 ${data.configured ? "已连接" : "未连接"} · iFinD ${ifind.configured ? "已配置" : "未配置"} · ${number(data.snapshot_dates)} 个交易日`;
status.classList.toggle("connected", Boolean(data.configured)); status.classList.toggle("connected", Boolean(data.configured));
setText("systemDataStatus", data.background_refresh_enabled ? "后台刷新已启用" : "后台刷新已暂停"); setText("systemDataStatus", data.background_refresh_enabled ? "后台刷新已启用" : "后台刷新已暂停");
document.querySelector("#systemTokenInput").value = ""; renderDatahubRouteStatus(data.datahub || {});
document.querySelector("#systemIfindTokenInput").value = "";
document.querySelector("#systemBackgroundRefresh").checked = Boolean(data.background_refresh_enabled); document.querySelector("#systemBackgroundRefresh").checked = Boolean(data.background_refresh_enabled);
document.querySelector("#memberDailyLimit").value = number(membership.member_daily_limit) || 50; document.querySelector("#memberDailyLimit").value = number(membership.member_daily_limit) || 50;
renderModelPool(llm.models || [], llm.primary_model_id || "", llm.fallback_model_id || ""); renderModelPool(llm.models || [], llm.primary_model_id || "", llm.fallback_model_id || "");
@@ -55,6 +54,26 @@ async function openAdminSettings(refreshOnly = false) {
} }
} }
function renderDatahubRouteStatus(hub) {
const box = document.querySelector("#datahubRouteStatus");
if (!box) return;
const label = box.querySelector("span");
const enabled = number(hub.enabled_reads);
const total = number(hub.total_reads) || enabled;
const fallbacks = hub.fallback_labels || [];
if (fallbacks.length) {
box.dataset.tone = "warning";
if (label) label.textContent = `数据中枢主线路 ${enabled}/${total} · 备用 ${fallbacks.length} 类:${fallbacks.join("、")}`;
return;
}
box.dataset.tone = hub.configured ? "success" : "idle";
if (label) {
label.textContent = hub.configured
? `数据中枢主线路 ${enabled}/${total},当前无备用`
: "数据中枢未配置,网站只保留已有真实快照";
}
}
function selectAdminPanel(panel) { function selectAdminPanel(panel) {
const selected = ["market", "models", "members"].includes(panel) ? panel : "market"; const selected = ["market", "models", "members"].includes(panel) ? panel : "market";
document.querySelector("#adminSectionSelect").value = selected; document.querySelector("#adminSectionSelect").value = selected;
@@ -184,13 +203,9 @@ async function saveMarketSettings(event) {
button.disabled = true; button.disabled = true;
try { try {
await apiRequest("/api/admin/settings", "POST", { await apiRequest("/api/admin/settings", "POST", {
tushare_token: document.querySelector("#systemTokenInput").value.trim(),
ifind_refresh_token: document.querySelector("#systemIfindTokenInput").value.trim(),
background_refresh_enabled: document.querySelector("#systemBackgroundRefresh").checked, background_refresh_enabled: document.querySelector("#systemBackgroundRefresh").checked,
}); });
document.querySelector("#systemTokenInput").value = ""; showToast("行情刷新设置已保存");
document.querySelector("#systemIfindTokenInput").value = "";
showToast("行情配置已保存");
await openAdminSettings(true); await openAdminSettings(true);
} catch (error) { } catch (error) {
showToast(error.message || "系统配置保存失败"); showToast(error.message || "系统配置保存失败");
+3
View File
@@ -62,6 +62,9 @@
try { try {
response = await fetch(url, requestOptions(method, body, options.signal)); response = await fetch(url, requestOptions(method, body, options.signal));
} catch (error) { } catch (error) {
if (error?.name === "AbortError") {
throw new ApiError("请求已取消或超时", 0, { aborted: true });
}
throw new ApiError(readableRequestError(error), 0, null); throw new ApiError(readableRequestError(error), 0, null);
} }
const payload = await parseJson(response); const payload = await parseJson(response);
+5
View File
@@ -67,6 +67,11 @@ async function startAdminRefresh() {
const actualCompact = actualDate.replaceAll("-", ""); const actualCompact = actualDate.replaceAll("-", "");
const updated = formatTimestamp(meta.updated_at); const updated = formatTimestamp(meta.updated_at);
const freshness = dashboardFreshnessMessage(meta); const freshness = dashboardFreshnessMessage(meta);
if (meta.realtime && actualCompact === requestedCompact && !meta.carried_forward) {
setAdminRefreshStatus("success", `刷新成功:已获取 ${actualDate} 的盘中行情,更新时间 ${updated}`, "circle-check");
showToast(`刷新成功:已获取 ${actualDate} 的盘中行情`);
return;
}
if (freshness || actualCompact !== requestedCompact || meta.carried_forward || meta.limit_data_source === "derived") { if (freshness || actualCompact !== requestedCompact || meta.carried_forward || meta.limit_data_source === "derived") {
setAdminRefreshStatus("warning", freshness || `部分正式数据尚未到齐,当前展示 ${actualDate || "最近可用数据"}`, "triangle-alert"); setAdminRefreshStatus("warning", freshness || `部分正式数据尚未到齐,当前展示 ${actualDate || "最近可用数据"}`, "triangle-alert");
setStatus(freshness || "部分正式数据尚未到齐,当前展示最近可用数据"); setStatus(freshness || "部分正式数据尚未到齐,当前展示最近可用数据");
+244 -15
View File
@@ -3,7 +3,8 @@ from __future__ import annotations
import copy import copy
import threading import threading
import unittest import unittest
from datetime import date, datetime, timedelta, timezone from datetime import date, datetime, timedelta, timezone, time as dt_time
from unittest.mock import patch
from pathlib import Path from pathlib import Path
from backend.features.market.service import MarketServiceMixin from backend.features.market.service import MarketServiceMixin
@@ -105,18 +106,84 @@ class FakeDerivedClient:
} }
SHANGHAI = timezone(timedelta(hours=8))
TRADE_DAY = date(2026, 9, 8)
def at_clock(hour: int, minute: int, day: date = TRADE_DAY) -> datetime:
return datetime(day.year, day.month, day.day, hour, minute, tzinfo=SHANGHAI)
class FakeMissingDailyClient: class FakeMissingDailyClient:
def __init__(self, open_today: bool = True):
self.open_today = open_today
def dashboard(self, trade_date: str): def dashboard(self, trade_date: str):
raise TushareError(f"No daily data returned for {trade_date}") raise TushareError(f"No daily data returned for {trade_date}")
def resolve_trade_context(self, requested: str):
if self.open_today:
return requested, "20260907"
return "20260907", "20260904"
class FakeRealtimeTodayClient:
def dashboard(self, trade_date: str):
return {
"meta": {
"trade_date": f"{trade_date[:4]}-{trade_date[4:6]}-{trade_date[6:8]}",
"requested_date": f"{trade_date[:4]}-{trade_date[4:6]}-{trade_date[6:8]}",
"realtime": True,
"mode": "realtime",
"market_status": "trading",
"notice": "盘中行情由 Tushare rt_k 实时计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。",
"updated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
},
"overview": {"limit_up_count": 15},
"limits": [{"code": "000001"}],
"broken": [],
"down_limits": [],
"yesterday_limits": [],
}
def resolve_trade_context(self, requested: str):
return requested, "20260907"
class FakeFreeRealtimeTodayClient:
def dashboard(self, trade_date: str):
return {
"meta": {
"trade_date": f"{trade_date[:4]}-{trade_date[4:6]}-{trade_date[6:8]}",
"requested_date": f"{trade_date[:4]}-{trade_date[4:6]}-{trade_date[6:8]}",
"realtime": True,
"mode": "realtime",
"quote_source": "eastmoney_clist",
"source": "eastmoney",
"market_status": "trading",
"notice": "盘中行情由东财免费实时快照计算;涨停原因、封板时间和开板次数以盘后榜单校正为准。",
"updated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"indices": [{"code": "000001", "price": 3800.1, "change": 0.5}],
},
"overview": {"limit_up_count": 18, "up_count": 2100, "amount_billion": 12345.6},
"limits": [{"code": "000001"}],
"broken": [],
"down_limits": [],
"yesterday_limits": [],
}
def resolve_trade_context(self, requested: str):
return requested, "20260907"
class SyncHarness(MarketServiceMixin): class SyncHarness(MarketServiceMixin):
def __init__(self, client, latest=None): def __init__(self, client, latest=None, clock=None):
self.configured = True self.configured = True
self.sync_lock = threading.Lock() self.sync_lock = threading.Lock()
self.database = FakeSyncDatabase(latest) self.database = FakeSyncDatabase(latest)
self._client = client self._client = client
self.current_user_id = 1 self.current_user_id = 1
self.clock = clock
def _tushare_client(self): def _tushare_client(self):
return self._client return self._client
@@ -142,23 +209,161 @@ class DashboardFreshnessTests(unittest.TestCase):
self.assertEqual(harness.database.finished[0][0][1], "success") self.assertEqual(harness.database.finished[0][0][1], "success")
self.assertEqual(verified_dashboard_result(payload), payload) self.assertEqual(verified_dashboard_result(payload), payload)
def test_missing_official_data_keeps_previous_day_with_preparing_notice(self): def test_intraday_refresh_keeps_today_and_does_not_fall_back_to_yesterday(self):
today = date.today() today = TRADE_DAY.strftime("%Y%m%d")
previous = (today - timedelta(days=1)).strftime("%Y-%m-%d")
latest = { latest = {
"meta": {"trade_date": previous, "source": "tushare"}, "meta": {"trade_date": "2026-09-07", "source": "tushare"},
"overview": {"limit_up_count": 20}, "overview": {"limit_up_count": 20},
} }
harness = SyncHarness(FakeMissingDailyClient(), latest) harness = SyncHarness(
payload = harness.sync_dashboard(today.strftime("%Y%m%d")) FakeRealtimeTodayClient(),
latest,
clock=lambda: at_clock(10, 5),
)
payload = harness.sync_dashboard(today)
meta = payload["meta"] meta = payload["meta"]
self.assertTrue(meta["carried_forward"]) self.assertFalse(meta.get("carried_forward"))
self.assertEqual(meta["data_status"], "preparing") self.assertTrue(meta["realtime"])
self.assertIn("今日数据正在准备,当前展示", meta["display_notice"]) self.assertEqual(meta["data_status"], "intraday")
self.assertIn("", meta["display_notice"]) self.assertEqual(str(meta["trade_date"]).replace("-", ""), today)
self.assertNotIn("No daily data", meta["display_notice"]) self.assertNotIn("今日数据正在准备", meta.get("display_notice") or "")
self.assertNotEqual(verified_dashboard_result(payload).get("status"), "failed") self.assertEqual(harness.database.saved[0][0], today)
def test_intraday_free_source_keeps_today_and_indices(self):
today = TRADE_DAY.strftime("%Y%m%d")
latest = {
"meta": {"trade_date": "2026-09-07", "source": "tushare"},
"overview": {"limit_up_count": 20},
}
harness = SyncHarness(
FakeFreeRealtimeTodayClient(),
latest,
clock=lambda: at_clock(10, 5),
)
payload = harness.sync_dashboard(today)
meta = payload["meta"]
self.assertFalse(meta.get("carried_forward"))
self.assertTrue(meta["realtime"])
self.assertEqual(meta["data_status"], "intraday")
self.assertEqual(str(meta["trade_date"]).replace("-", ""), today)
self.assertEqual(meta["quote_source"], "eastmoney_clist")
self.assertEqual(payload["overview"]["amount_billion"], 12345.6)
self.assertEqual(meta["indices"][0]["price"], 3800.1)
self.assertEqual(harness.database.saved[0][0], today)
def test_intraday_missing_quotes_do_not_carry_yesterday(self):
today = TRADE_DAY.strftime("%Y%m%d")
latest = {
"meta": {"trade_date": "2026-09-07", "source": "tushare"},
"overview": {"limit_up_count": 20},
}
harness = SyncHarness(
FakeMissingDailyClient(),
latest,
clock=lambda: at_clock(10, 5),
)
with self.assertRaises(ValueError) as ctx:
harness.sync_dashboard(today)
self.assertIn("当天盘中行情", str(ctx.exception))
self.assertFalse(harness.database.saved)
def test_intraday_keeps_existing_today_snapshot_when_refresh_fails(self):
today = TRADE_DAY.strftime("%Y%m%d")
existing = {
"meta": {
"trade_date": "2026-09-08",
"realtime": True,
"mode": "realtime",
"source": "tushare",
},
"overview": {"limit_up_count": 11},
"limits": [{"code": "600000"}],
"broken": [],
"down_limits": [],
"yesterday_limits": [],
}
harness = SyncHarness(
FakeMissingDailyClient(),
clock=lambda: at_clock(10, 5),
)
harness.database.get_snapshot = lambda *_args, **_kwargs: copy.deepcopy(existing)
payload = harness.sync_dashboard(today)
meta = payload["meta"]
self.assertEqual(str(meta["trade_date"]).replace("-", ""), today)
self.assertTrue(meta["realtime"])
self.assertEqual(meta["data_status"], "intraday")
self.assertFalse(meta.get("carried_forward"))
def test_lunch_and_after_hours_keep_today_until_official_arrives(self):
today = TRADE_DAY.strftime("%Y%m%d")
for clock in (lambda: at_clock(12, 0), lambda: at_clock(16, 10)):
harness = SyncHarness(
FakeRealtimeTodayClient(),
clock=clock,
)
payload = harness.sync_dashboard(today)
self.assertEqual(str(payload["meta"]["trade_date"]).replace("-", ""), today)
self.assertFalse(payload["meta"].get("carried_forward"))
def test_preopen_and_weekend_still_carry_last_session(self):
latest = {
"meta": {"trade_date": "2026-09-07", "source": "tushare"},
"overview": {"limit_up_count": 20},
}
preopen = SyncHarness(
FakeMissingDailyClient(),
latest,
clock=lambda: at_clock(8, 30),
)
preopen_payload = preopen.sync_dashboard(TRADE_DAY.strftime("%Y%m%d"))
self.assertTrue(preopen_payload["meta"]["carried_forward"])
self.assertEqual(preopen_payload["meta"]["data_status"], "preparing")
self.assertIn("今日数据正在准备,当前展示", preopen_payload["meta"]["display_notice"])
weekend = SyncHarness(
FakeMissingDailyClient(open_today=False),
latest,
clock=lambda: at_clock(10, 5, date(2026, 9, 5)),
)
weekend_payload = weekend.sync_dashboard("20260905")
self.assertTrue(weekend_payload["meta"]["carried_forward"])
def test_history_date_still_uses_official_or_preparing_notice(self):
latest = {
"meta": {"trade_date": "2026-09-01", "source": "tushare"},
"overview": {"limit_up_count": 8},
}
harness = SyncHarness(
FakeMissingDailyClient(),
latest,
clock=lambda: at_clock(10, 5),
)
payload = harness.sync_dashboard("20260902")
self.assertTrue(payload["meta"]["carried_forward"])
self.assertIn("所选日期数据尚未到齐", payload["meta"]["display_notice"])
def test_carried_today_snapshot_is_retried_immediately_in_session(self):
today = TRADE_DAY.strftime("%Y%m%d")
snapshot = {
"meta": {
"source": "tushare",
"trade_date": "2026-09-07",
"carried_forward": True,
"requested_date": "2026-09-08",
"updated_at": at_clock(10, 0).isoformat(),
},
"overview": {"limit_up_count": 1},
}
harness = SyncHarness(
FakeRealtimeTodayClient(),
clock=lambda: at_clock(10, 5),
)
harness.database.get_snapshot = lambda *_args, **_kwargs: copy.deepcopy(snapshot)
payload = harness.get_dashboard(today)
self.assertEqual(str(payload["meta"]["trade_date"]).replace("-", ""), today)
self.assertEqual(payload["meta"]["data_status"], "intraday")
self.assertTrue(harness.database.saved)
def test_weekend_carry_is_not_labeled_as_preparing(self): def test_weekend_carry_is_not_labeled_as_preparing(self):
snapshot = { snapshot = {
@@ -200,19 +405,43 @@ class DashboardFreshnessTests(unittest.TestCase):
{"meta": {"trade_date": iso, "limit_data_source": "derived"}}, {"meta": {"trade_date": iso, "limit_data_source": "derived"}},
) )
now = datetime.now().astimezone().time().replace(tzinfo=None) now = datetime.now().astimezone().time().replace(tzinfo=None)
if datetime.strptime("15:05", "%H:%M").time() <= now < datetime.strptime("22:00", "%H:%M").time(): if dt_time(15, 5) <= now < dt_time(22, 0):
self.assertFalse(due) self.assertFalse(due)
self.assertTrue(derived_due) self.assertTrue(derived_due)
else: else:
self.assertFalse(due) self.assertFalse(due)
self.assertFalse(derived_due) self.assertFalse(derived_due)
def test_official_catchup_is_due_for_intraday_snapshot_after_close(self):
today = TRADE_DAY.strftime("%Y%m%d")
snapshot = {
"meta": {
"trade_date": "2026-09-08",
"realtime": True,
"mode": "realtime",
}
}
with patch("backend.jobs.refresh.datetime") as mocked:
mocked.now.return_value = at_clock(16, 10)
mocked.strptime = datetime.strptime
self.assertTrue(official_catchup_due(today, snapshot))
official = {
"meta": {
"trade_date": "2026-09-08",
"limit_data_source": "official",
"realtime": False,
}
}
self.assertFalse(official_catchup_due(today, official))
class FrontendRefreshCopyTests(unittest.TestCase): class FrontendRefreshCopyTests(unittest.TestCase):
def test_dashboard_script_distinguishes_partial_from_failure(self): def test_dashboard_script_distinguishes_partial_from_failure(self):
script = (Path(__file__).resolve().parents[1] / "frontend" / "shared" / "dashboard.js").read_text(encoding="utf-8") script = (Path(__file__).resolve().parents[1] / "frontend" / "shared" / "dashboard.js").read_text(encoding="utf-8")
self.assertIn("今日数据正在准备,当前展示", script) self.assertIn("今日数据正在准备,当前展示", script)
self.assertIn("部分正式数据尚未到齐", script) self.assertIn("部分正式数据尚未到齐", script)
self.assertIn("盘中行情", script)
self.assertIn("meta.realtime && actualCompact === requestedCompact", script)
self.assertIn('job.status === "failed"', script) self.assertIn('job.status === "failed"', script)
failed_block = script.split("if (job.status === \"failed\")", 1)[1].split("const query", 1)[0] failed_block = script.split("if (job.status === \"failed\")", 1)[1].split("const query", 1)[0]
self.assertIn("后台刷新失败", failed_block) self.assertIn("后台刷新失败", failed_block)
+6 -5
View File
@@ -10,7 +10,7 @@ from database import ReviewDatabase
class BootstrapContainerTests(unittest.TestCase): class BootstrapContainerTests(unittest.TestCase):
def test_environment_credentials_preserve_legacy_model_fallbacks(self) -> None: def test_environment_credentials_exclude_provider_secrets_and_preserve_llm_fallbacks(self) -> None:
result = environment_credentials( result = environment_credentials(
{ {
"TUSHARE_TOKEN": " tushare ", "TUSHARE_TOKEN": " tushare ",
@@ -20,8 +20,8 @@ class BootstrapContainerTests(unittest.TestCase):
"LLM_MODEL": "legacy-model", "LLM_MODEL": "legacy-model",
} }
) )
self.assertEqual(result["tushare_token"], "tushare") self.assertNotIn("tushare_token", result)
self.assertEqual(result["ifind_refresh_token"], "refresh") self.assertNotIn("ifind_refresh_token", result)
self.assertEqual(result["platform_llm_primary_api_key"], "legacy-key") self.assertEqual(result["platform_llm_primary_api_key"], "legacy-key")
self.assertEqual(result["platform_llm_primary_base_url"], "https://legacy.example/v1") self.assertEqual(result["platform_llm_primary_base_url"], "https://legacy.example/v1")
self.assertEqual(result["platform_llm_primary_model"], "legacy-model") self.assertEqual(result["platform_llm_primary_model"], "legacy-model")
@@ -45,8 +45,9 @@ class BootstrapContainerTests(unittest.TestCase):
self.assertIs(container.strategy_tracking.repository.database, database) self.assertIs(container.strategy_tracking.repository.database, database)
self.assertIs(container.alert_service.repository.database, database) self.assertIs(container.alert_service.repository.database, database)
self.assertIs(container.trade_journal.repository.database, database) self.assertIs(container.trade_journal.repository.database, database)
self.assertIs(container.chart_data.ifind, container.ifind) self.assertIs(container.ifind, container.data_gateway.ifind)
self.assertTrue(container.ifind.configured) self.assertIs(container.chart_data.datahub, container.data_gateway.datahub)
self.assertIsNone(container.chart_data.ifind)
if __name__ == "__main__": if __name__ == "__main__":
+165 -1
View File
@@ -2,7 +2,7 @@ from __future__ import annotations
import unittest import unittest
from backend.features.market.charts import ChartDataError, EastmoneyChartClient from backend.features.market.charts import ChartDataError, EastmoneyChartClient, HIS_TRENDS_URL, MarketChartClient, TRENDS_URL
from server import DashboardService from server import DashboardService
@@ -72,6 +72,170 @@ class ChartDataProviderTests(unittest.TestCase):
self.client.stock_intraday("abc") self.client.stock_intraday("abc")
class LookbackChartClient(EastmoneyChartClient):
def __init__(self) -> None:
super().__init__(cache_ttl_seconds=20)
self.requests: list[tuple[str, dict[str, str]]] = []
def _request_json(self, url, params, referer):
self.requests.append((url, params))
if url == TRENDS_URL and params.get("ndays") == "1":
return {"data": {"code": "601318", "name": "中国平安", "preClose": 56.0, "trends": []}}
if url == TRENDS_URL and params.get("ndays") == "5":
return {"data": {"code": "601318", "name": "中国平安", "preClose": 56.0, "trends": []}}
if url == HIS_TRENDS_URL:
return {
"data": {
"code": "601318",
"name": "中国平安",
"preClose": 55.8,
"trends": [
"2026-09-07 09:30,55.80,55.90,56.00,55.70,100,5580.00,55.900",
"2026-09-07 15:00,56.10,56.20,56.30,56.00,200,11240.00,56.150",
"2026-09-08 09:30,0,0,0,0,0,0.00,0",
],
}
}
raise ChartDataError("unexpected url")
class ChartLookbackTests(unittest.TestCase):
def setUp(self) -> None:
EastmoneyChartClient._cache.clear()
self.client = LookbackChartClient()
def test_empty_today_falls_back_to_latest_available_session(self):
payload = self.client.stock_intraday("601318")
urls = [url for url, _ in self.client.requests]
self.assertEqual(urls[0], TRENDS_URL)
self.assertEqual(self.client.requests[0][1]["ndays"], "1")
self.assertEqual(urls[1], TRENDS_URL)
self.assertEqual(self.client.requests[1][1]["ndays"], "5")
self.assertEqual(urls[2], HIS_TRENDS_URL)
self.assertEqual(payload["trade_date"], "2026-09-07")
self.assertEqual([point["time"] for point in payload["points"]], ["09:30", "15:00"])
self.assertEqual(payload["points"][0]["close"], 55.9)
def test_delay_multiday_can_recover_without_his(self):
class DelayFive(EastmoneyChartClient):
def __init__(self):
super().__init__(cache_ttl_seconds=20)
self.requests = []
def _request_json(self, url, params, referer):
self.requests.append((url, params))
if params.get("ndays") == "1":
return {"data": {"code": "000001", "name": "平安银行", "preClose": 11.7, "trends": []}}
return {
"data": {
"code": "000001",
"name": "平安银行",
"preClose": 11.5,
"trends": [
"2026-09-07 09:30,11.50,11.60,11.70,11.40,100,1160.00,11.600",
"2026-09-07 15:00,11.70,11.80,11.90,11.60,200,2360.00,11.750",
],
}
}
EastmoneyChartClient._cache.clear()
client = DelayFive()
payload = client.stock_intraday("000001")
self.assertEqual(payload["trade_date"], "2026-09-07")
self.assertEqual(len(payload["points"]), 2)
self.assertEqual([url for url, _ in client.requests], [TRENDS_URL, TRENDS_URL])
def test_sh_sz_cyb_codes_use_correct_secid(self):
for code, secid in (("601318", "1.601318"), ("000001", "0.000001"), ("300750", "0.300750")):
EastmoneyChartClient._cache.clear()
client = LookbackChartClient()
client.stock_intraday(code)
self.assertEqual(client.requests[0][1]["secid"], secid)
class FakeHub:
def __init__(self, chart=None, error=None, daily=None):
self.chart = chart
self.error = error
self.daily = daily
self.calls: list[str] = []
self.legacy: list[str] = []
def try_intraday(self, code):
self.calls.append(code)
if self.error:
raise self.error
return self.chart
def try_daily_chart(self, code, end_date, limit=90, dataset="daily"):
self.calls.append(f"{dataset}:{code}")
if self.error:
raise self.error
return self.daily
def record_legacy(self, dataset, source="", error=""):
self.legacy.append(dataset)
class DatahubChartFallbackTests(unittest.TestCase):
def setUp(self) -> None:
EastmoneyChartClient._cache.clear()
def test_datahub_success_skips_old_channel(self):
hub = FakeHub(
{
"entity_type": "stock",
"identifier": "601318",
"name": "中国平安",
"code": "601318",
"trade_date": "2026-09-08",
"previous_close": 56.36,
"points": [{"date": "2026-09-08", "time": "09:30", "close": 56.5, "average": 56.4}],
"source": "datahub",
}
)
fallback = LookbackChartClient()
client = MarketChartClient(hub)
payload = client.stock_intraday("601318")
self.assertEqual(payload["source"], "datahub")
self.assertEqual(hub.calls, ["601318"])
self.assertEqual(fallback.requests, [])
def test_datahub_timeout_or_empty_does_not_use_old_channel(self):
fallback = LookbackChartClient()
for hub in (
FakeHub(chart=None),
FakeHub(error=RuntimeError("timeout")),
FakeHub(error=RuntimeError("datahub exploded")),
FakeHub(chart={"points": []}),
):
EastmoneyChartClient._cache.clear()
fallback.requests.clear()
client = MarketChartClient(hub)
with self.assertRaises(ChartDataError):
client.stock_intraday("000001")
self.assertEqual(fallback.requests, [])
def test_datahub_daily_skips_ifind(self):
hub = FakeHub(
daily=[
{
"trade_date": "2026-09-07",
"open": 10.0,
"high": 10.4,
"low": 9.9,
"close": 10.2,
"volume": 1000,
"amount_billion": 0.02,
}
]
)
client = MarketChartClient(hub)
rows = client.stock_daily("600000", "20260907")
self.assertEqual(rows[-1]["trade_date"], "2026-09-07")
self.assertIn("daily:600000", hub.calls)
class ChartServiceStub: class ChartServiceStub:
@staticmethod @staticmethod
def _payload(code: str, name: str): def _payload(code: str, name: str):
+38 -15
View File
@@ -12,6 +12,7 @@ from backend.data import (
QualityEvidence, QualityEvidence,
build_data_gateway, build_data_gateway,
) )
from backend.data.datahub.settings import DATASETS, DatahubSettings, DatasetFlags
from backend.data.quality import market_timezone from backend.data.quality import market_timezone
@@ -35,16 +36,30 @@ class DataGatewayTests(unittest.TestCase):
with self.assertRaises(DataPolicyError): with self.assertRaises(DataPolicyError):
policy.assert_allowed("market.level2", "unresolved", "display") policy.assert_allowed("market.level2", "unresolved", "display")
def test_gateway_uses_live_token_supplier_and_shared_ifind(self) -> None: def test_gateway_uses_hub_facade_and_proxies(self) -> None:
token = {"value": "first"} settings = DatahubSettings(
gateway = build_data_gateway( base_url="http://127.0.0.1:8766",
{"ifind_refresh_token": "refresh", "ifind_access_token": "access"}, token="hub-token",
lambda: token["value"], datasets={name: DatasetFlags(name, read=True) for name in DATASETS},
) )
self.assertEqual(gateway.tushare().token, "first") gateway = build_data_gateway(
token["value"] = "second" {},
self.assertEqual(gateway.tushare().token, "second") datahub_settings=settings,
self.assertIs(gateway.chart_data.ifind, gateway.ifind) )
client = gateway.tushare()
self.assertEqual(client.token, "datahub")
self.assertIsNone(client.realtime_aggregator)
self.assertFalse(hasattr(client, "_legacy"))
self.assertIs(gateway.ifind, gateway.ifind_provider.client)
self.assertIs(gateway.chart_data.datahub, gateway.datahub)
self.assertIsNone(gateway.chart_data.ifind)
from backend.data.datahub.bridge import DatahubAwareTushareClient
from backend.data.datahub.ifind_proxy import HubIfindProxy
from backend.data.datahub.realtime_proxy import HubRealtimeProxy
self.assertIsInstance(client, DatahubAwareTushareClient)
self.assertIsInstance(gateway.ifind, HubIfindProxy)
self.assertIsInstance(gateway.realtime_observer, HubRealtimeProxy)
def test_server_has_no_direct_runtime_tushare_construction(self) -> None: def test_server_has_no_direct_runtime_tushare_construction(self) -> None:
source = ( source = (
@@ -54,24 +69,29 @@ class DataGatewayTests(unittest.TestCase):
/ "market" / "market"
/ "service.py" / "service.py"
).read_text(encoding="utf-8") ).read_text(encoding="utf-8")
self.assertEqual(source.count("TushareClient(self.token)"), 1) self.assertNotIn("TushareClient(self.token)", source)
self.assertIn("return gateway.tushare()", source) self.assertIn("return gateway.tushare()", source)
def test_provider_construction_has_unique_declared_owners(self) -> None: def test_provider_construction_has_unique_declared_owners(self) -> None:
root = Path(__file__).resolve().parents[1] root = Path(__file__).resolve().parents[1]
owners = { owners = {
"EastmoneyChartClient": {"backend/data/gateway.py"},
"IfindHttpClient": {"backend/data/gateway.py"},
"IfindProvider": {"backend/data/gateway.py"}, "IfindProvider": {"backend/data/gateway.py"},
"MarketChartClient": {"backend/data/gateway.py"}, "MarketChartClient": {"backend/data/gateway.py"},
"TushareClient": {"backend/features/market/service.py"}, "TushareClient": set(),
"TushareProvider": {"backend/data/gateway.py"},
"WebRealtimeAggregator": {"backend/data/gateway.py"},
"DatahubClient": {"backend/data/gateway.py"}, "DatahubClient": {"backend/data/gateway.py"},
"DatahubAwareTushareClient": {"backend/data/gateway.py"}, "DatahubAwareTushareClient": {"backend/data/gateway.py"},
"DatahubBridge": {"backend/data/gateway.py"}, "DatahubBridge": {"backend/data/gateway.py"},
"HubIfindProxy": {"backend/data/gateway.py"},
"HubRealtimeProxy": {"backend/data/gateway.py"},
} }
found = {name: set() for name in owners} found = {name: set() for name in owners}
forbidden = {
"IfindHttpClient": set(),
"EastmoneyChartClient": set(),
"WebRealtimeAggregator": set(),
"TushareProvider": set(),
}
found_forbidden = {name: set() for name in forbidden}
for path in (root / "backend").rglob("*.py"): for path in (root / "backend").rglob("*.py"):
relative = path.relative_to(root).as_posix() relative = path.relative_to(root).as_posix()
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
@@ -81,7 +101,10 @@ class DataGatewayTests(unittest.TestCase):
name = getattr(node.func, "id", None) or getattr(node.func, "attr", None) name = getattr(node.func, "id", None) or getattr(node.func, "attr", None)
if name in found: if name in found:
found[name].add(relative) found[name].add(relative)
if name in found_forbidden:
found_forbidden[name].add(relative)
self.assertEqual(found, owners) self.assertEqual(found, owners)
self.assertEqual(found_forbidden, forbidden)
provider_source = (root / "backend/data/providers/tushare.py").read_text( provider_source = (root / "backend/data/providers/tushare.py").read_text(
encoding="utf-8" encoding="utf-8"
) )
+304 -61
View File
@@ -12,6 +12,8 @@ from backend.data.datahub.client import DatahubClient, DatahubResponse
from backend.data.datahub.compare import compare_rows from backend.data.datahub.compare import compare_rows
from backend.data.datahub.errors import DatahubError from backend.data.datahub.errors import DatahubError
from backend.data.datahub.native import to_canonical_row, to_native_row from backend.data.datahub.native import to_canonical_row, to_native_row
from backend.data.datahub.route_state import LEDGER
from backend.data.providers.tushare_transport import TushareError
from backend.data.datahub.settings import DATASETS, DatahubSettings, DatasetFlags from backend.data.datahub.settings import DATASETS, DatahubSettings, DatasetFlags
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
@@ -64,10 +66,19 @@ class FakeClient(DatahubClient):
meta={"tier": "official", "trade_date": "20240902", "stale": False, "staleness_seconds": 0}, meta={"tier": "official", "trade_date": "20240902", "stale": False, "staleness_seconds": 0},
) )
self.paths: list[str] = [] self.paths: list[str] = []
self.calls: list[tuple[str, dict[str, Any]]] = []
def get(self, path: str, params: dict[str, Any] | None = None) -> DatahubResponse: def get(self, path: str, params: dict[str, Any] | None = None) -> DatahubResponse:
return self._record(path, params)
def post(self, path: str, body: dict[str, Any] | None = None) -> DatahubResponse:
return self._record(path, body)
def _record(self, path: str, payload: dict[str, Any] | None) -> DatahubResponse:
self.paths.append(path) self.paths.append(path)
if TOKEN in json.dumps(params or {}) or TOKEN in path: self.calls.append((path, {key: value for key, value in (payload or {}).items()}))
packed = json.dumps(payload or {})
if TOKEN in packed or TOKEN in path:
raise AssertionError("token leaked into url") raise AssertionError("token leaked into url")
if self.error: if self.error:
raise self.error raise self.error
@@ -82,17 +93,21 @@ def flags(**enabled: tuple[bool, bool]) -> DatahubSettings:
class DatahubBridgeTests(unittest.TestCase): class DatahubBridgeTests(unittest.TestCase):
def test_default_config_keeps_legacy_and_does_not_call_datahub(self) -> None: def setUp(self) -> None:
LEDGER.clear()
def test_default_config_enables_official_reads(self) -> None:
settings = DatahubSettings.load(environ={}, credentials={}) settings = DatahubSettings.load(environ={}, credentials={})
self.assertFalse(settings.any_enabled()) self.assertTrue(settings.any_enabled())
self.assertTrue(all(not settings.flags(name).read and not settings.flags(name).shadow for name in DATASETS)) self.assertTrue(all(settings.flags(name).read and not settings.flags(name).shadow for name in DATASETS))
client = FakeClient(error=DatahubError("INTERNAL", "should not be called")) client = FakeClient()
legacy = FakeLegacy([LEGACY_DAILY]) legacy = FakeLegacy([LEGACY_DAILY])
wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(settings, client)) wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(settings, client))
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,close,vol,amount") rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,close,vol,amount")
self.assertEqual(rows[0]["amount"], 2000.0) self.assertEqual(rows[0]["amount"], 2000.0)
self.assertEqual(client.paths, []) self.assertEqual(client.paths, ["/v1/bars/daily"])
self.assertEqual(len(legacy.calls), 1) self.assertEqual(legacy.calls, [])
self.assertEqual(LEDGER.snapshot()[0]["route"], "datahub")
def test_each_dataset_has_independent_read_flag(self) -> None: def test_each_dataset_has_independent_read_flag(self) -> None:
settings = flags(daily=(True, False), auction=(False, False)) settings = flags(daily=(True, False), auction=(False, False))
@@ -102,6 +117,13 @@ class DatahubBridgeTests(unittest.TestCase):
source = (ROOT / "config" / "datahub.config.json").read_text(encoding="utf-8") source = (ROOT / "config" / "datahub.config.json").read_text(encoding="utf-8")
self.assertNotIn("master", source) self.assertNotIn("master", source)
self.assertNotIn("DATAHUB_READ_ALL", source) self.assertNotIn("DATAHUB_READ_ALL", source)
compose = (ROOT / "compose.yaml").read_text(encoding="utf-8")
for env_key in (
"CALENDAR", "STOCKS", "DAILY", "INDEX_DAILY", "VALUATION", "MONEYFLOW",
"AUCTION", "LIMIT_EVENTS", "POPULARITY", "DRAGON_TIGER", "SECTOR_DAILY",
"QUOTES", "INDEX_QUOTES", "INTRADAY", "STATUS",
):
self.assertIn(f'DATAHUB_READ_{env_key}: "1"', compose)
def test_read_flag_replaces_only_that_dataset_and_converts_units(self) -> None: def test_read_flag_replaces_only_that_dataset_and_converts_units(self) -> None:
shadows: list[dict[str, Any]] = [] shadows: list[dict[str, Any]] = []
@@ -117,16 +139,22 @@ class DatahubBridgeTests(unittest.TestCase):
self.assertEqual(legacy.calls, []) self.assertEqual(legacy.calls, [])
self.assertEqual(client.paths, ["/v1/bars/daily"]) self.assertEqual(client.paths, ["/v1/bars/daily"])
calendar_legacy = FakeLegacy([{"cal_date": "20240902", "is_open": 1}]) calendar_legacy = FakeLegacy([{"cal_date": "20240902", "is_open": 1}])
calendar_client = FakeClient(error=DatahubError("INTERNAL", "nope")) calendar_client = FakeClient(
response=DatahubResponse(
data=[{"cal_date": "20240902", "is_open": 1, "pretrade_date": "20240830"}],
meta={"source": "datahub", "stale": False, "staleness_seconds": 0},
)
)
calendar_wrapped = DatahubAwareTushareClient( calendar_wrapped = DatahubAwareTushareClient(
calendar_legacy, calendar_legacy,
DatahubBridge(flags(daily=(True, False)), calendar_client), DatahubBridge(flags(daily=(True, False)), calendar_client),
) )
calendar = calendar_wrapped.query("trade_cal", {"start_date": "20240902", "end_date": "20240902"}, "") calendar = calendar_wrapped.query("trade_cal", {"start_date": "20240902", "end_date": "20240902"}, "")
self.assertEqual(calendar[0]["is_open"], 1) self.assertEqual(calendar[0]["is_open"], 1)
self.assertEqual(calendar_client.paths, []) self.assertEqual(calendar_legacy.calls, [])
self.assertEqual(calendar_client.paths, ["/v1/query"])
def test_fallback_on_down_401_timeout_empty_unpublished_stale_and_incomplete(self) -> None: def test_hub_failure_does_not_call_website_legacy(self) -> None:
cases = [ cases = [
DatahubError("UNAVAILABLE", "down"), DatahubError("UNAVAILABLE", "down"),
DatahubError("UNAUTHORIZED", "401"), DatahubError("UNAUTHORIZED", "401"),
@@ -138,34 +166,21 @@ class DatahubBridgeTests(unittest.TestCase):
] ]
for error in cases: for error in cases:
with self.subTest(error=error.code): with self.subTest(error=error.code):
if error.code == "EMPTY": client = FakeClient(error=error)
client = FakeClient(response=DatahubResponse(data=[], meta={"stale": False, "staleness_seconds": 0}))
elif error.code == "STALE":
client = FakeClient(response=DatahubResponse(
data=[dict(HUB_DAILY)],
meta={"stale": True, "staleness_seconds": 999999},
))
elif error.code == "INCOMPLETE":
client = FakeClient(response=DatahubResponse(
data=[dict(HUB_DAILY)],
meta={
"stale": False,
"staleness_seconds": 0,
"incomplete": True,
"coverage": {"complete": False, "missing_count": 80},
},
))
else:
client = FakeClient(error=error)
legacy = FakeLegacy([LEGACY_DAILY]) legacy = FakeLegacy([LEGACY_DAILY])
wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(flags(daily=(True, False)), client)) wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(flags(daily=(True, False)), client))
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount") with self.assertRaises(TushareError):
self.assertEqual(rows[0]["amount"], 2000.0) wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount")
self.assertEqual(len(legacy.calls), 1) self.assertEqual(legacy.calls, [])
def test_shadow_compares_without_replacing_and_survives_hub_failure(self) -> None: def test_shadow_mode_no_longer_calls_website_tushare(self) -> None:
reports: list[dict[str, Any]] = [] reports: list[dict[str, Any]] = []
client = FakeClient() client = FakeClient(
response=DatahubResponse(
data=[dict(LEGACY_DAILY)],
meta={"source": "tushare", "stale": False, "staleness_seconds": 0, "row_shape": "tushare"},
)
)
legacy = FakeLegacy([LEGACY_DAILY]) legacy = FakeLegacy([LEGACY_DAILY])
wrapped = DatahubAwareTushareClient( wrapped = DatahubAwareTushareClient(
legacy, legacy,
@@ -173,21 +188,19 @@ class DatahubBridgeTests(unittest.TestCase):
) )
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount,vol") rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount,vol")
self.assertEqual(rows[0]["amount"], 2000.0) self.assertEqual(rows[0]["amount"], 2000.0)
self.assertEqual(len(legacy.calls), 1) self.assertEqual(legacy.calls, [])
self.assertEqual(reports[0]["equal"], True) self.assertEqual(client.paths, ["/v1/query"])
self.assertEqual(reports[0]["matched"], 1)
failed = FakeClient(error=DatahubError("UNAVAILABLE", TOKEN)) failed = FakeClient(error=DatahubError("UNAVAILABLE", TOKEN))
fail_reports: list[dict[str, Any]] = []
fail_legacy = FakeLegacy([LEGACY_DAILY]) fail_legacy = FakeLegacy([LEGACY_DAILY])
fail_wrapped = DatahubAwareTushareClient( fail_wrapped = DatahubAwareTushareClient(
fail_legacy, fail_legacy,
DatahubBridge(flags(daily=(False, True)), failed, shadow_sink=fail_reports.append), DatahubBridge(flags(daily=(False, True)), failed, shadow_sink=reports.append),
) )
again = fail_wrapped.query("daily", {"trade_date": "20240902"}, "amount") with self.assertRaises(TushareError):
self.assertEqual(again[0]["amount"], 2000.0) fail_wrapped.query("daily", {"trade_date": "20240902"}, "amount")
self.assertTrue(fail_reports[0]["hub_error"]) self.assertEqual(fail_legacy.calls, [])
self.assertNotIn(TOKEN, json.dumps(fail_reports[0])) self.assertNotIn(TOKEN, str(failed.calls))
def test_compare_classifies_unit_conversion_missing_row_and_value_diff(self) -> None: def test_compare_classifies_unit_conversion_missing_row_and_value_diff(self) -> None:
equal = compare_rows("daily", [LEGACY_DAILY], [HUB_DAILY], {"stale": False, "staleness_seconds": 0}) equal = compare_rows("daily", [LEGACY_DAILY], [HUB_DAILY], {"stale": False, "staleness_seconds": 0})
@@ -274,13 +287,12 @@ class DatahubBridgeTests(unittest.TestCase):
) )
wrapped = DatahubAwareTushareClient( wrapped = DatahubAwareTushareClient(
FakeLegacy([legacy_close_only]), FakeLegacy([legacy_close_only]),
DatahubBridge(flags(daily=(False, True)), client, shadow_sink=reports.append), DatahubBridge(flags(daily=(True, False)), client, shadow_sink=reports.append),
) )
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,trade_date,close,vol,amount") rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,trade_date,close,vol,amount")
self.assertEqual(rows[0]["close"], 10.20) self.assertEqual(rows[0]["close"], 10.20)
self.assertEqual(rows[0]["vol"], 1000.0) self.assertEqual(rows[0]["vol"], 1000.0)
self.assertTrue(reports[0]["equal"]) self.assertEqual(client.paths, ["/v1/bars/daily"])
self.assertEqual(reports[0]["matched"], 1)
def test_native_roundtrip_matches_known_scales(self) -> None: def test_native_roundtrip_matches_known_scales(self) -> None:
native = to_native_row("daily", HUB_DAILY) native = to_native_row("daily", HUB_DAILY)
@@ -290,8 +302,8 @@ class DatahubBridgeTests(unittest.TestCase):
self.assertEqual(canonical["vol"], 100000.0) self.assertEqual(canonical["vol"], 100000.0)
self.assertEqual(canonical["amount"], 2000000.0) self.assertEqual(canonical["amount"], 2000000.0)
def test_heaven_keeps_legacy_on_first_batch_even_when_read_flag_is_on(self) -> None: def test_heaven_can_use_hub_when_dataset_flag_is_on(self) -> None:
"""问天未永久冻结;首批只读接入仍走旧链路,后续迁移可以纳入""" """问天按数据依赖接入:已映射 API 跟随开关,不再整栈强制旧链路"""
self.assertTrue(looks_like_heaven("backend.features.heaven.market_context", "backend/features/heaven/market_context.py")) self.assertTrue(looks_like_heaven("backend.features.heaven.market_context", "backend/features/heaven/market_context.py"))
self.assertFalse(looks_like_heaven("backend.features.market.service", "backend/features/market/service.py")) self.assertFalse(looks_like_heaven("backend.features.market.service", "backend/features/market/service.py"))
client = FakeClient() client = FakeClient()
@@ -302,7 +314,8 @@ class DatahubBridgeTests(unittest.TestCase):
) )
rows = wrapped.query("daily", {"trade_date": "20240902"}, "amount") rows = wrapped.query("daily", {"trade_date": "20240902"}, "amount")
self.assertEqual(rows[0]["amount"], 2000.0) self.assertEqual(rows[0]["amount"], 2000.0)
self.assertEqual(client.paths, []) self.assertEqual(client.paths, ["/v1/bars/daily"])
self.assertEqual(legacy.calls, [])
def test_status_flag_does_not_run_when_off_and_falls_back_when_on(self) -> None: def test_status_flag_does_not_run_when_off_and_falls_back_when_on(self) -> None:
off = DatahubBridge(flags(), FakeClient(error=DatahubError("UNAVAILABLE", "down"))) off = DatahubBridge(flags(), FakeClient(error=DatahubError("UNAVAILABLE", "down")))
@@ -332,21 +345,251 @@ class DatahubBridgeTests(unittest.TestCase):
self.assertIn('"daily"', source) self.assertIn('"daily"', source)
self.assertIn("start_date", source) self.assertIn("start_date", source)
self.assertIn("end_date", source) self.assertIn("end_date", source)
client = FakeClient( client = FakeClient(error=DatahubError("INCOMPLETE", "truncated"))
response=DatahubResponse(
data=[dict(HUB_DAILY)],
meta={"stale": False, "staleness_seconds": 0, "incomplete": True, "coverage": {"complete": False, "missing_count": 89}},
)
)
legacy = FakeLegacy([LEGACY_DAILY]) legacy = FakeLegacy([LEGACY_DAILY])
wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(flags(daily=(True, False)), client)) wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(flags(daily=(True, False)), client))
rows = wrapped.query( with self.assertRaises(TushareError):
"daily", wrapped.query(
{"ts_code": "600000.SH", "start_date": "20240301", "end_date": "20240902"}, "daily",
"ts_code,amount", {"ts_code": "600000.SH", "start_date": "20240301", "end_date": "20240902"},
"ts_code,amount",
)
self.assertEqual(legacy.calls, [])
self.assertIn("/v1/query", client.paths)
def test_try_intraday_respects_switch_and_falls_back_on_bad_payload(self) -> None:
closed = DatahubBridge(flags(), FakeClient(error=DatahubError("INTERNAL", "should not run")))
self.assertIsNone(closed.try_intraday("601318"))
empty = DatahubBridge(
flags(intraday=(True, False)),
FakeClient(response=DatahubResponse(data={"points": []}, meta={"stale": False})),
) )
self.assertEqual(rows[0]["amount"], 2000.0) self.assertIsNone(empty.try_intraday("601318"))
self.assertEqual(len(legacy.calls), 1)
stale = DatahubBridge(
flags(intraday=(True, False)),
FakeClient(response=DatahubResponse(
data={
"entity_type": "stock",
"code": "601318",
"trade_date": "2026-09-07",
"previous_close": 55.8,
"points": [{"date": "2026-09-07", "time": "09:30", "close": 55.9, "avg_price": 55.85}],
},
meta={"stale": True},
)),
)
self.assertIsNone(stale.try_intraday("601318"))
ok = DatahubBridge(
flags(intraday=(True, False)),
FakeClient(response=DatahubResponse(
data={
"entity_type": "stock",
"identifier": "601318",
"name": "中国平安",
"code": "601318",
"trade_date": "2026-09-08",
"previous_close": 56.36,
"points": [
{"date": "2026-09-08", "time": "09:30", "close": 0},
{"date": "2026-09-08", "time": "09:31", "close": 56.5, "avg_price": 56.4},
],
},
meta={"stale": False},
)),
)
chart = ok.try_intraday("601318")
self.assertEqual(chart["source"], "datahub")
self.assertEqual(len(chart["points"]), 1)
self.assertEqual(chart["points"][0]["average"], 56.4)
self.assertEqual(ok.client.paths, ["/v1/intraday/points"])
self.assertEqual(ok.client.calls, [("/v1/intraday/points", {"code": "601318"})])
self.assertNotIn("date", ok.client.calls[0][1])
timeout = DatahubBridge(
flags(intraday=(True, False)),
FakeClient(error=DatahubError("TIMEOUT", "datahub request timed out")),
)
self.assertIsNone(timeout.try_intraday("601318"))
broken = DatahubBridge(
flags(intraday=(True, False)),
FakeClient(error=DatahubError("INTERNAL", "datahub exploded")),
)
self.assertIsNone(broken.try_intraday("601318"))
self.assertTrue(DatahubSettings.load(environ={}, credentials={}).flags("intraday").read)
def test_try_market_quotes_and_visible_fallback(self) -> None:
quotes = [
{
"ts_code": f"{600000 + index:06d}.SH",
"name": f"股票{index}",
"close": 10.2,
"pre_close": 10.0,
"open": 10.1,
"high": 10.3,
"low": 9.9,
"vol": 1000,
"amount": 2000000,
"quote_date": "20240902",
}
for index in range(220)
]
ok = DatahubBridge(
flags(quotes=(True, False)),
FakeClient(
response=DatahubResponse(
data=quotes,
meta={"stale": False, "staleness_seconds": 0, "source": "eastmoney:clist"},
)
),
)
rows = ok.try_market_quotes("20240902")
self.assertEqual(len(rows), 220)
self.assertEqual(rows[0]["pre_close"], 10.0)
self.assertEqual(ok.client.paths, ["/v1/quotes/latest"])
self.assertEqual(LEDGER.snapshot()[0]["route"], "datahub")
failed = DatahubBridge(
flags(quotes=(True, False)),
FakeClient(error=DatahubError("UNAVAILABLE", "down")),
)
self.assertIsNone(failed.try_market_quotes("20240902"))
snap = next(item for item in LEDGER.snapshot() if item["dataset"] == "quotes")
self.assertEqual(snap["route"], "datahub")
self.assertEqual(snap["source"], "unavailable")
gateway = build_data_gateway({}, datahub_settings=flags(quotes=(True, False)))
status = gateway.datahub_status()
self.assertEqual(status["enabled_reads"], 1)
self.assertEqual(status["total_reads"], len(DATASETS))
self.assertEqual(status["fallback_count"], 0)
def test_try_daily_chart_converts_hub_bars(self) -> None:
rows = [
{
"ts_code": "600000.SH",
"trade_date": "20240901",
"open": 10.0,
"high": 10.4,
"low": 9.9,
"close": 10.2,
"volume": 100000,
"amount": 2000000,
},
{
"ts_code": "600000.SH",
"trade_date": "20240902",
"open": 10.2,
"high": 10.5,
"low": 10.1,
"close": 10.4,
"volume": 120000,
"amount": 2400000,
},
]
hub = DatahubBridge(
flags(daily=(True, False)),
FakeClient(
response=DatahubResponse(
data=rows,
meta={"stale": False, "staleness_seconds": 0, "source": "tushare:daily"},
)
),
)
chart = hub.try_daily_chart("600000.SH", "20240902", 90, "daily")
self.assertEqual(chart[-1]["trade_date"], "2024-09-02")
self.assertEqual(chart[-1]["close"], 10.4)
self.assertAlmostEqual(chart[-1]["amount_billion"], 0.024)
def test_try_daily_chart_keeps_usable_bars_when_coverage_incomplete(self) -> None:
rows = [
{
"ts_code": "000001.SZ",
"trade_date": "20240901",
"open": 10.0,
"high": 10.4,
"low": 9.9,
"close": 10.2,
"volume": 100000,
"amount": 2000000,
},
{
"ts_code": "000001.SZ",
"trade_date": "20240902",
"open": 10.2,
"high": 10.5,
"low": 10.1,
"close": 10.4,
"volume": 120000,
"amount": 2400000,
},
]
hub = DatahubBridge(
flags(daily=(True, False)),
FakeClient(
response=DatahubResponse(
data=rows,
meta={
"stale": False,
"staleness_seconds": 0,
"incomplete": True,
"coverage": {"complete": False, "missing_count": 127},
"source": "tushare:daily",
},
)
),
)
chart = hub.try_daily_chart("000001.SZ", "20240902", 90, "daily")
self.assertIsNotNone(chart)
self.assertEqual(chart[-1]["trade_date"], "2024-09-02")
self.assertEqual(chart[-1]["close"], 10.4)
def test_gateway_tushare_facade_has_no_legacy_client(self) -> None:
quotes = [
{
"ts_code": f"{index:06d}.SZ",
"name": f"S{index}",
"pre_close": 10.0,
"open": 10.0,
"high": 10.5,
"low": 9.8,
"close": 10.2,
"vol": 100.0,
"amount": 1000.0,
"quote_date": "20240902",
}
for index in range(1, 221)
]
hub_client = FakeClient(
response=DatahubResponse(
data=quotes,
meta={"stale": False, "staleness_seconds": 0, "source": "eastmoney_clist"},
)
)
gateway = build_data_gateway(
{"tushare_token": "tok"},
datahub_settings=flags(quotes=(True, False), daily=(True, False)),
)
gateway.datahub.client = hub_client
wrapped = gateway.tushare()
self.assertFalse(hasattr(wrapped, "_legacy"))
self.assertIsNone(getattr(type(wrapped), "__getattr__", None))
self.assertTrue(callable(getattr(type(wrapped), "try_market_quotes", None)))
self.assertTrue(callable(getattr(type(wrapped), "try_index_quotes", None)))
self.assertTrue(callable(getattr(type(wrapped), "record_datahub_legacy", None)))
self.assertTrue(callable(getattr(type(wrapped), "dashboard", None)))
rows = wrapped.try_market_quotes("20240902")
self.assertGreaterEqual(len(rows or []), 200)
self.assertIn("/v1/quotes/latest", hub_client.paths)
hub_client.response = DatahubResponse(
data=[dict(HUB_DAILY)],
meta={"stale": False, "staleness_seconds": 0, "source": "tushare:daily"},
)
daily = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount")
self.assertEqual(daily[0]["amount"], 2000.0)
self.assertIn("/v1/bars/daily", hub_client.paths)
def test_features_do_not_import_datahub_client(self) -> None: def test_features_do_not_import_datahub_client(self) -> None:
violations = [] violations = []
+13
View File
@@ -459,6 +459,19 @@ class FrontendContractTests(unittest.TestCase):
self.assertIn('payload.question_preset = state.heartQuestionPreset;', self.script) self.assertIn('payload.question_preset = state.heartQuestionPreset;', self.script)
self.assertIn('payload.cast_at = state.heartCastAt;', self.script) self.assertIn('payload.cast_at = state.heartCastAt;', self.script)
def test_heaven_loading_timeout_clears_dimmed_state(self):
self.assertIn("controller.abort()", self.script)
self.assertIn('heavenView?.classList.remove("heaven-data-loading")', self.script)
self.assertIn("问天数据仍在准备,页面可继续输入和操作", self.script)
self.assertIn("const blocking = !state.heavenSetup;", self.script)
self.assertIn("payload?.aborted", self.script)
def test_stock_detail_does_not_display_missing_metrics_as_zero(self):
self.assertIn("function setStockBoardFields(row)", self.script)
self.assertIn("function presentMetric(value)", self.script)
self.assertIn("payload.available !== false", self.script)
self.assertIn('element.textContent = "--"', self.script)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+1 -1
View File
@@ -94,7 +94,7 @@ class GlobalSearchTests(unittest.TestCase):
self.assertIn('event.key.toLowerCase() !== "k"', script) self.assertIn('event.key.toLowerCase() !== "k"', script)
self.assertIn('openStock(item.id, { code: item.code', script) self.assertIn('openStock(item.id, { code: item.code', script)
self.assertNotIn('include_notes', script) self.assertNotIn('include_notes', script)
self.assertIn('const candles = (series || [])', script) self.assertIn('const candles = visibleDailyPrices((series || [])', script)
self.assertIn('renderStockNotes(payload.notes || [])', script) self.assertIn('renderStockNotes(payload.notes || [])', script)
+83 -1
View File
@@ -3,9 +3,10 @@ from __future__ import annotations
import http.client import http.client
import json import json
import unittest import unittest
from datetime import datetime
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
from backend.data.realtime import WebRealtimeAggregator from backend.data.realtime import RealtimeAggregateError, WebRealtimeAggregator
from backend.features.heaven.engine import _market_line_scores, build_manual_market_hexagram from backend.features.heaven.engine import _market_line_scores, build_manual_market_hexagram
from server import DashboardService from server import DashboardService
from backend.data.providers.tushare_client import ( from backend.data.providers.tushare_client import (
@@ -377,6 +378,87 @@ class RealtimeAggregatorTests(unittest.TestCase):
self.assertEqual(rows[0]["quote_time"][:10], "2026-07-20") self.assertEqual(rows[0]["quote_time"][:10], "2026-07-20")
self.assertAlmostEqual(rows[0]["amount_billion"], 12946.52) self.assertAlmostEqual(rows[0]["amount_billion"], 12946.52)
@patch.object(WebRealtimeAggregator, "_get_json")
def test_eastmoney_market_quotes_normalize_and_keep_expected_date(self, get_json: MagicMock):
epoch = datetime(2026, 7, 20, 10, 5).timestamp()
rows = []
for index in range(200):
sz = index < 100
rows.append(
{
"f12": f"{index:06d}" if sz else f"{600000 + index - 100:06d}",
"f13": 0 if sz else 1,
"f14": f"股票{index}",
"f2": 11.2,
"f3": 2.0,
"f5": 10,
"f6": 50000000,
"f15": 11.3,
"f16": 11.0,
"f17": 11.1,
"f18": 11.0,
"f124": epoch,
}
)
def fake_get_json(_url, params, referer=""):
page = int(params.get("pn") or 1)
start = (page - 1) * 100
return {"rc": 0, "data": {"total": 200, "diff": rows[start:start + 100]}}
get_json.side_effect = fake_get_json
aggregator = WebRealtimeAggregator()
aggregator._response_cache.clear()
quotes = aggregator.eastmoney_market_quotes("20260720")
self.assertEqual(len(quotes), 200)
self.assertEqual(quotes[0]["ts_code"], "000000.SZ")
self.assertTrue(quotes[100]["ts_code"].endswith(".SH"))
self.assertEqual(quotes[0]["vol"], 1000)
self.assertEqual(quotes[0]["quote_date"], "20260720")
@patch.object(WebRealtimeAggregator, "_get_text")
def test_tencent_stock_quote_keeps_expected_date(self, get_text: MagicMock):
fields = [""] * 38
fields[1] = "浦发银行"
fields[2] = "600000"
fields[3] = "11.20"
fields[4] = "11.00"
fields[5] = "11.10"
fields[6] = "1234"
fields[30] = "20260720103000"
fields[33] = "11.30"
fields[34] = "11.00"
fields[37] = "1380"
get_text.return_value = (f'v_sh600000="{"~".join(fields)}";', 0)
quote = WebRealtimeAggregator().tencent_stock_quote("600000", "20260720")
self.assertEqual(quote["ts_code"], "600000.SH")
self.assertEqual(quote["quote_date"], "20260720")
self.assertEqual(quote["vol"], 123400)
self.assertAlmostEqual(quote["amount"], 13_800_000)
@patch.object(WebRealtimeAggregator, "_get_json")
def test_eastmoney_stock_quote_rejects_stale_date(self, get_json: MagicMock):
epoch = datetime(2026, 7, 19, 15, 0).timestamp()
get_json.return_value = {
"rc": 0,
"data": {
"f43": 11.2,
"f44": 11.3,
"f45": 11.0,
"f46": 11.1,
"f47": 10,
"f48": 50000000,
"f57": "300750",
"f58": "宁德时代",
"f60": 11.0,
"f86": epoch,
},
}
with self.assertRaises(RealtimeAggregateError):
WebRealtimeAggregator().eastmoney_stock_quote("300750.SZ", "20260720")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+342
View File
@@ -0,0 +1,342 @@
from __future__ import annotations
import unittest
from unittest.mock import MagicMock, patch
from backend.data.providers.tushare_client import TushareClient, TushareError
from backend.data.providers.tushare_helpers import _moneyflow_payload
from backend.data.realtime import (
WebRealtimeAggregator,
_normalize_eastmoney_limit_row,
_normalize_eastmoney_stock_quote,
)
from backend.data.providers.tushare_daily import DailyMarketMixin
class MoneyflowPayloadTests(unittest.TestCase):
def test_missing_row_is_not_zero(self) -> None:
payload = _moneyflow_payload(None)
self.assertFalse(payload["available"])
self.assertIsNone(payload["net_million"])
self.assertIsNone(payload["large_million"])
def test_empty_row_is_not_zero(self) -> None:
payload = _moneyflow_payload({})
self.assertFalse(payload["available"])
self.assertIsNone(payload["net_million"])
def test_real_zero_net_is_kept_when_source_exists(self) -> None:
payload = _moneyflow_payload(
{
"net_mf_amount": 0,
"buy_lg_amount": 1,
"sell_lg_amount": 1,
"buy_elg_amount": 0,
"sell_elg_amount": 0,
"buy_md_amount": 0,
"sell_md_amount": 0,
"buy_sm_amount": 0,
"sell_sm_amount": 0,
}
)
self.assertTrue(payload["available"])
self.assertEqual(payload["net_million"], 0)
class LimitOverlayTests(unittest.TestCase):
def test_normalize_limit_keeps_missing_seal_as_none(self) -> None:
row = DailyMarketMixin._normalize_limit(
{
"ts_code": "000737.SZ",
"name": "北方铜业",
"close": 12.3,
"pct_chg": 10,
"amount": 1e8,
"amount_unit": "yuan",
},
"涨停",
)
self.assertIsNone(row["seal_amount_million"])
self.assertEqual(row["first_time"], "--")
def test_overlay_fills_board_times_from_official_list(self) -> None:
mixin = DailyMarketMixin()
mixin._load_limit_lists = lambda trade_date: [
{
"ts_code": "000737.SZ",
"first_time": "09:31:02",
"last_time": "10:18:11",
"fd_amount": 82000000,
"open_times": 1,
"turnover_ratio": 18.4,
}
]
mixin.realtime_aggregator = None
rows = mixin._overlay_board_fields(
[{"ts_code": "000737.SZ", "close": 12.3, "limit_type": "U"}],
"20260908",
)
self.assertEqual(rows[0]["first_time"], "09:31:02")
self.assertEqual(rows[0]["fd_amount"], 82000000)
self.assertEqual(rows[0]["turnover_ratio"], 18.4)
class ShenwanRealtimeSourceTests(unittest.TestCase):
def test_transport_refuses_rt_sw_k(self) -> None:
client = TushareClient(token="demo")
with self.assertRaisesRegex(TushareError, "rt_sw_k is disabled"):
client.query("rt_sw_k", {"ts_code": "801074.SI"})
def test_outer_realtime_uses_hub_sector_quote_not_rt_sw_k(self) -> None:
client = TushareClient(token="demo")
client.query = MagicMock(side_effect=AssertionError("should not call tushare"))
client.try_sector_quote = MagicMock(return_value={
"code": "801074.SI",
"name": "工业金属",
"close": 1234.5,
"pre_close": 1200,
"change": 2.88,
"pct_change": 2.88,
"quote_date": "20260908",
"quote_time": "2026-09-08T14:50:00+08:00",
"source": "eastmoney_sw",
})
row, source, error = client._sw_outer_realtime("801074.SI", "工业金属", "20260908")
self.assertEqual(source, "eastmoney_sw")
self.assertEqual(error, "")
self.assertEqual(row["change"], 2.88)
client.query.assert_not_called()
def test_outer_waiting_state_has_no_permission_error(self) -> None:
client = TushareClient(token="demo")
client.realtime_aggregator = None
row, source, error = client._sw_outer_realtime(
"801074.SI", "工业金属", "20260908", finalized=True
)
self.assertEqual(row, {})
self.assertIn("尚未入库", error)
self.assertNotIn("权限", error)
self.assertNotIn("rt_sw_k", error)
class EastmoneyHelperTests(unittest.TestCase):
def test_limit_pool_row_keeps_board_clock(self) -> None:
parsed = _normalize_eastmoney_limit_row(
{
"c": "000737",
"m": 0,
"n": "北方铜业",
"fbt": 93102,
"lbt": 101811,
"zbc": 1,
"lbc": 2,
"hs": 18.4,
"fund": 82000000,
},
"U",
)
self.assertEqual(parsed["ts_code"], "000737.SZ")
self.assertEqual(parsed["first_time"], "09:31:02")
self.assertEqual(parsed["last_time"], "10:18:11")
self.assertEqual(parsed["fd_amount"], 82000000)
def test_stock_quote_keeps_moneyflow_when_present(self) -> None:
quote = _normalize_eastmoney_stock_quote(
{
"f43": 12.3,
"f60": 11.18,
"f46": 11.2,
"f44": 12.3,
"f45": 11.1,
"f47": 1000,
"f48": 150000000,
"f58": "北方铜业",
"f86": 0,
"f168": 8.5,
"f62": 25000000,
"f78": 3000000,
"f84": -1000000,
},
"000737.SZ",
)
self.assertEqual(quote["net_mf_amount"], 2500)
payload = _moneyflow_payload(quote)
self.assertTrue(payload["available"])
self.assertEqual(payload["net_million"], 25)
@patch.object(WebRealtimeAggregator, "_get_json")
def test_shenwan_quote_uses_eastmoney_90_prefix(self, get_json: MagicMock) -> None:
get_json.return_value = {
"rc": 0,
"data": {
"diff": [
{
"f12": "801074",
"f14": "工业金属",
"f2": 1234.5,
"f3": 2.88,
"f18": 1200,
"f17": 1205,
"f15": 1240,
"f16": 1198,
"f6": 1,
"f124": 1757319000,
}
]
},
}
quote = WebRealtimeAggregator().eastmoney_shenwan_quote("801074.SI")
self.assertEqual(quote["source"], "eastmoney_sw")
self.assertAlmostEqual(quote["change"], 2.88)
params = get_json.call_args.args[1]
self.assertEqual(params["secids"], "90.801074")
class ChartWindowTests(unittest.TestCase):
def test_display_window_is_45_not_250(self) -> None:
from backend.features.market.charts import DAILY_CHART_LIMIT
self.assertEqual(DAILY_CHART_LIMIT, 45)
class MemberQuoteCoverageTests(unittest.TestCase):
def test_prefers_full_hub_market_over_truncated_named_quotes(self) -> None:
client = TushareClient(token="demo")
wanted = [f"{index:06d}.SZ" for index in range(205)]
market = [
{"ts_code": code, "close": 10.0, "pre_close": 9.0}
for code in wanted
]
client.try_market_quotes = MagicMock(return_value=market)
client.try_quotes = MagicMock(return_value=market[:60])
client.realtime_aggregator = MagicMock()
rows, source = client._load_member_realtime_quotes(wanted, "20260908")
self.assertEqual(len(rows), 205)
self.assertEqual(source, "datahub")
client.try_quotes.assert_not_called()
def test_hub_named_quotes_cover_members_when_market_missing(self) -> None:
client = TushareClient(token="demo")
wanted = ["000737.SZ", "000630.SZ"]
client.try_market_quotes = MagicMock(return_value=None)
client.try_quotes = MagicMock(return_value=[
{"ts_code": "000737.SZ", "close": 12.3, "pre_close": 11.2},
{"ts_code": "000630.SZ", "close": 4.5, "pre_close": 4.4},
])
client.realtime_aggregator = MagicMock()
rows, source = client._load_member_realtime_quotes(wanted, "20260908")
self.assertEqual(len(rows), 2)
self.assertEqual(source, "datahub")
client.try_quotes.assert_called()
client.realtime_aggregator.eastmoney_stock_quotes.assert_not_called()
client.realtime_aggregator.tencent_stock_quotes.assert_not_called()
def test_delayed_hub_quotes_are_kept_not_zeroed(self) -> None:
client = TushareClient(token="demo")
delayed = [
{
"ts_code": "000737.SZ",
"close": 12.3,
"pre_close": 11.2,
"delayed": True,
"delay_seconds": 90,
"delay_notice": "主备免费行情均暂不可用,显示 90 秒前的真实快照",
}
]
client.try_market_quotes = MagicMock(return_value=delayed)
client.try_quotes = MagicMock()
client.realtime_aggregator = MagicMock()
rows, source = client._load_member_realtime_quotes(["000737.SZ"], "20260908")
self.assertEqual(source, "datahub_delayed")
self.assertEqual(rows[0]["close"], 12.3)
client.try_quotes.assert_not_called()
def test_ignores_non_member_quotes_from_market_snapshot(self) -> None:
client = TushareClient(token="demo")
client.try_market_quotes = MagicMock(
return_value=[
{"ts_code": "000737.SZ", "close": 12.3, "pre_close": 11.2},
{"ts_code": "600000.SH", "close": 10.0, "pre_close": 9.9},
]
)
client.try_quotes = MagicMock(return_value=[])
client._free_realtime_quotes = MagicMock(return_value=([], "empty"))
rows, _source = client._load_member_realtime_quotes(
["000737.SZ", "000630.SZ"], "20260908"
)
self.assertEqual([row["ts_code"] for row in rows], ["000737.SZ"])
def test_local_sw_members_survive_tushare_outage(self) -> None:
import tempfile
from pathlib import Path
from backend.data.providers import tushare_industries as module
client = TushareClient(token="demo")
stored = [
{
"ts_code": "000737.SZ",
"name": "北方铜业",
"l2_code": "801074.SI",
"in_date": "20200101",
"out_date": "",
}
]
with tempfile.TemporaryDirectory() as tmp:
original = module._SW_MEMBER_DIR
module._SW_MEMBER_DIR = Path(tmp)
try:
client._sw_member_cache.clear()
client._write_local_sw_members("801074.SI", stored)
client.query = MagicMock(side_effect=TushareError("index_member_all down"))
members = client._sw_sector_members("801074.SI", "20260908")
finally:
module._SW_MEMBER_DIR = original
client._sw_member_cache.clear()
self.assertEqual([item["ts_code"] for item in members], ["000737.SZ"])
client.query.assert_not_called()
def test_closed_keeps_daily_inner_when_sw_daily_missing(self) -> None:
client = TushareClient(token="demo")
client.resolve_trade_context = lambda _date: ("20260908", "20260907")
client.sw_stock_industry = MagicMock(
return_value={"l2_code": "801074.SI", "l2_name": "工业金属"}
)
client._sw_sector_members = MagicMock(
return_value=[
{"ts_code": "000737.SZ", "name": "北方铜业"},
{"ts_code": "000630.SZ", "name": "铜陵有色"},
]
)
client._stock_listing_reference = MagicMock(return_value={})
client._load_daily = MagicMock(
return_value=[
{"ts_code": "000737.SZ", "name": "北方铜业", "pct_chg": 2, "amount": 1e8},
{"ts_code": "000630.SZ", "name": "铜陵有色", "pct_chg": 1, "amount": 1e8},
]
)
client._confirmed_suspended_members = MagicMock(return_value=[])
client.query = MagicMock(return_value=[])
client._sw_realtime_sector_snapshot = MagicMock(
side_effect=AssertionError("daily inner should be kept")
)
client.try_sector_quote = MagicMock(return_value={
"code": "801074.SI",
"name": "工业金属",
"change": 1.5,
"pct_change": 1.5,
"quote_date": "20260908",
"quote_time": "2026-09-08T15:00:00+08:00",
"source": "eastmoney_sw",
})
snapshot = client.sw_sector_snapshot(
"000737.SZ", "20260908", allow_realtime_close=True
)
self.assertEqual(snapshot["quote_count"], 2)
self.assertEqual(snapshot["member_count"], 2)
self.assertTrue(snapshot["inner_precise"])
self.assertTrue(snapshot["outer_precise"])
self.assertEqual(snapshot["inner_source"], "tushare_member_daily")
self.assertEqual(snapshot["change"], 1.5)
self.assertNotIn("权限", snapshot.get("outer_error") or "")
self.assertNotIn("rt_sw_k", snapshot.get("outer_error") or "")
+1 -1
View File
@@ -138,7 +138,7 @@ class HttpDispatchContractTests(unittest.TestCase):
self.assertTrue(claimed.isdisjoint(methods)) self.assertTrue(claimed.isdisjoint(methods))
claimed.update(methods) claimed.update(methods)
self.assertLessEqual(len(path.read_text(encoding="utf-8").splitlines()), line_limit) self.assertLessEqual(len(path.read_text(encoding="utf-8").splitlines()), line_limit)
self.assertEqual(len(claimed), 27) self.assertEqual(len(claimed), 28)
if __name__ == "__main__": if __name__ == "__main__":
+434
View File
@@ -0,0 +1,434 @@
from __future__ import annotations
import ast
import json
import re
import threading
import unittest
from pathlib import Path
from unittest.mock import patch
from backend.data import build_data_gateway
from backend.data.datahub.bridge import DatahubAwareTushareClient, DatahubBridge
from backend.data.datahub.client import DatahubClient
from backend.data.datahub.ifind_proxy import HubIfindProxy
from backend.data.datahub.realtime_proxy import HubRealtimeProxy
from backend.data.datahub.settings import DATASETS, DatahubSettings, DatasetFlags
from backend.data.providers.tushare_transport import TushareError
from backend.features.market.charts import MarketChartClient
from backend.features.system.service import SystemServiceMixin
from tests.test_datahub_bridge import FakeClient, FakeLegacy, flags
ROOT = Path(__file__).resolve().parents[1]
BLOCKED_HOSTS = (
"api.tushare.pro",
"push2.eastmoney.com",
"push2delay.eastmoney.com",
"push2his.eastmoney.com",
"push2ex.eastmoney.com",
"qt.gtimg.cn",
"hq.sinajs.cn",
"10jqka.com.cn",
"xuangubao.cn",
"quantapi.51ifind.com",
"51ifind.com",
)
LEFTOVER_WEBSITE_FILES = {
"backend/data/providers/ifind_client.py",
"backend/data/realtime.py",
"backend/features/market/charts.py",
"backend/data/providers/tushare_transport.py",
}
HUB_BASE = "http://127.0.0.1:8766"
def _enabled_settings() -> DatahubSettings:
return DatahubSettings(
base_url=HUB_BASE,
token="hub-token",
datasets={name: DatasetFlags(name, read=True) for name in DATASETS},
)
class _Resp:
def __init__(self, payload: dict) -> None:
self.status = 200
self._raw = json.dumps(payload).encode("utf-8")
def read(self):
return self._raw
def __enter__(self):
return self
def __exit__(self, *args):
return False
def hub_payload(request) -> dict:
url = str(getattr(request, "full_url", None) or request)
if any(host in url for host in BLOCKED_HOSTS):
raise AssertionError(f"website opened blocked host: {url}")
if HUB_BASE not in url:
raise AssertionError(f"unexpected url: {url}")
path = url.split(HUB_BASE, 1)[1].split("?", 1)[0]
if path == "/v1/bars/daily":
return {
"schema_version": 1,
"data": [
{
"ts_code": "600000.SH",
"trade_date": "20240902",
"open": 10.0,
"high": 10.4,
"low": 9.9,
"close": 10.2,
"volume": 100000,
"amount": 2000000,
}
],
"meta": {"stale": False, "staleness_seconds": 0, "source": "datahub"},
}
if path == "/v1/quotes/latest":
return {
"schema_version": 1,
"data": [
{
"ts_code": "600000.SH",
"code": "600000",
"name": "浦发银行",
"close": 10.2,
"price": 10.2,
"pre_close": 10.0,
"open": 10.1,
"high": 10.3,
"low": 9.9,
"vol": 1000,
"amount": 2000000,
"quote_date": "20240902",
"source": "datahub",
}
],
"meta": {"stale": False, "staleness_seconds": 0, "source": "eastmoney"},
}
if path == "/v1/indexes/quotes":
return {
"schema_version": 1,
"data": [
{
"ts_code": "000001.SH",
"code": "000001",
"name": "上证指数",
"price": 3000,
"pct_chg": 1.2,
"quote_time_epoch": 1725249600,
"source": "datahub",
},
{
"ts_code": "399001.SZ",
"code": "399001",
"name": "深证成指",
"price": 9000,
"pct_chg": 0.8,
"quote_time_epoch": 1725249600,
"source": "datahub",
},
{
"ts_code": "399006.SZ",
"code": "399006",
"name": "创业板指",
"price": 1800,
"pct_chg": 0.5,
"quote_time_epoch": 1725249600,
"source": "datahub",
},
],
"meta": {"stale": False, "staleness_seconds": 0, "source": "tencent"},
}
if path == "/v1/auction":
return {
"schema_version": 1,
"data": [
{
"ts_code": "600000.SH",
"trade_date": "20240902",
"close": 10.2,
"vol": 1000.0,
"amount": 2000.0,
}
],
"meta": {"stale": False, "staleness_seconds": 0, "source": "datahub"},
}
if path == "/v1/credentials/ifind":
return {
"schema_version": 1,
"data": {"configured": True, "access_ready": True, "access_expires_at": ""},
"meta": {"source": "ifind"},
}
if path == "/v1/intraday/points":
return {
"schema_version": 1,
"data": {
"entity_type": "stock",
"identifier": "600000",
"code": "600000",
"trade_date": "2024-09-02",
"previous_close": 10.0,
"points": [
{"date": "2024-09-02", "time": "09:30", "close": 10.2, "average": 10.1, "open": 10.1, "high": 10.2, "low": 10.0, "volume": 100, "amount": 1000}
],
},
"meta": {"stale": False, "source": "datahub"},
}
if path == "/v1/query":
body = json.loads(request.data.decode("utf-8") if request.data else "{}")
api_name = body.get("api_name")
if api_name == "ifind_status":
return {"schema_version": 1, "data": [{"configured": True, "access_ready": True, "access_expires_at": ""}], "meta": {"source": "ifind"}}
if api_name == "ifind_wencai":
return {
"schema_version": 1,
"data": [{"股票代码": "000001.SZ", "涨停原因": "重组", "首次涨停时间": "09:42:00"}],
"meta": {"source": "ifind"},
}
if api_name == "ifind_snapshots":
return {
"schema_version": 1,
"data": [
{
"time": "2024-09-02 09:21:00",
"thscode": "000001.SZ",
"latest": 10.5,
"preClose": 10,
"volume": 2000,
"amount": 21000,
}
],
"meta": {"source": "ifind"},
}
if api_name in {"daily", "rt_k", "stk_auction"}:
return {
"schema_version": 1,
"data": [{"ts_code": "600000.SH", "trade_date": "20240902", "close": 10.2, "amount": 2000.0}],
"meta": {"source": "datahub", "stale": False, "row_shape": "tushare"},
}
raise AssertionError(f"unexpected query api: {api_name}")
raise AssertionError(f"unexpected path: {path}")
def blocked_urlopen(request, timeout=None):
return _Resp(hub_payload(request))
class HubExclusiveWebsiteTests(unittest.TestCase):
def test_website_availability_depends_on_hub_not_provider_credentials(self) -> None:
service = SystemServiceMixin()
service._system_credentials = {}
service.data_gateway = build_data_gateway({}, datahub_settings=_enabled_settings())
self.assertTrue(service.configured)
self.assertEqual(service.token, "datahub")
def test_website_vault_scrubs_historical_provider_credentials(self) -> None:
class Vault:
@staticmethod
def decrypt_json(payload):
if isinstance(payload, str):
return json.loads(payload)
return dict(payload)
@staticmethod
def encrypt_json(payload):
return dict(payload)
class Database:
system = {
"tushare_token": "old-tushare",
"ifind_refresh_token": "old-refresh",
"ifind_access_token": "old-access",
}
personal = {"tushare_token": "old-personal", "llm_primary_model": "model"}
def get_system_setting(self, _name):
return dict(self.system)
@staticmethod
def first_user_id():
return 1
def get_user_credentials(self, _user_id):
return dict(self.personal)
@staticmethod
def list_user_credentials():
return [{"user_id": 1, "encrypted_payload": '{"tushare_token":"old-personal"}'}]
def save_system_setting(self, _name, payload):
self.system = dict(payload)
def save_user_credentials(self, _user_id, payload):
self.personal = dict(payload)
service = SystemServiceMixin()
service.database = Database()
service.vault = Vault()
service.system_lock = threading.Lock()
loaded = service._load_system_credentials({})
for key in ("tushare_token", "ifind_refresh_token", "ifind_access_token"):
self.assertNotIn(key, loaded)
self.assertNotIn(key, service.database.system)
self.assertNotIn(key, service.database.personal)
def test_query_never_calls_website_tushare_transport(self) -> None:
client = FakeClient()
legacy = FakeLegacy(TushareError("website tushare must stay dark"))
wrapped = DatahubAwareTushareClient(
legacy,
DatahubBridge(flags(daily=(True, False)), client),
)
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,amount")
self.assertEqual(rows[0]["amount"], 2000.0)
self.assertEqual(legacy.calls, [])
def test_blocked_external_hosts_still_read_hub(self) -> None:
settings = _enabled_settings()
hub_client = DatahubClient(settings, urlopen=blocked_urlopen)
legacy = FakeLegacy(TushareError("blocked"))
wrapped = DatahubAwareTushareClient(legacy, DatahubBridge(settings, hub_client))
with patch("urllib.request.urlopen", blocked_urlopen):
rows = wrapped.query("daily", {"trade_date": "20240902"}, "ts_code,close,amount")
self.assertEqual(rows[0]["close"], 10.2)
self.assertEqual(rows[0]["amount"], 2000.0)
self.assertEqual(legacy.calls, [])
def test_site_features_only_reach_hub_when_external_hosts_blocked(self) -> None:
settings = _enabled_settings()
hub_client = DatahubClient(settings, urlopen=blocked_urlopen)
bridge = DatahubBridge(settings, hub_client)
with patch("urllib.request.urlopen", blocked_urlopen):
quotes = bridge.try_quotes(["600000.SH"])
indices = bridge.try_index_quotes()
chart = MarketChartClient(bridge).stock_daily("600000", "20240902")
intraday = MarketChartClient(bridge).stock_intraday("600000")
ifind = HubIfindProxy(bridge)
wencai = ifind.wencai("2024年9月2日涨停股票")
snapshots = ifind.snapshots(["000001.SZ"], ["latest"], "2024-09-02 09:15:00", "2024-09-02 09:22:00")
realtime = HubRealtimeProxy(bridge)
index_rows = realtime.tencent_indices()
stock = realtime.tencent_stock_quote("600000.SH", "20240902")
health = realtime.health_snapshot()
self.assertEqual(quotes[0]["close"], 10.2)
self.assertEqual(len(indices), 3)
self.assertEqual(chart[-1]["close"], 10.2)
self.assertEqual(intraday["source"], "datahub")
self.assertEqual(wencai[0]["股票代码"], "000001.SZ")
self.assertEqual(snapshots[0]["latest"], 10.5)
self.assertEqual([row["code"] for row in index_rows], ["000001", "399001", "399006"])
self.assertEqual(stock["close"], 10.2)
self.assertTrue(health["ready"])
self.assertTrue(ifind.configured)
def test_production_gateway_does_not_construct_external_clients(self) -> None:
source = (ROOT / "backend" / "data" / "gateway.py").read_text(encoding="utf-8")
self.assertNotIn("IfindHttpClient", source)
self.assertNotIn("EastmoneyChartClient", source)
self.assertNotIn("WebRealtimeAggregator", source)
self.assertNotIn("TushareProvider", source)
self.assertIsNone(re.search(r"(?<![A-Za-z])TushareClient\(", source))
self.assertIn("HubIfindProxy", source)
self.assertIn("HubRealtimeProxy", source)
self.assertIn("DatahubAwareTushareClient", source)
facade = (ROOT / "backend" / "data" / "datahub" / "bridge.py").read_text(encoding="utf-8")
tree = ast.parse(facade)
cls = next(
node
for node in tree.body
if isinstance(node, ast.ClassDef) and node.name == "DatahubAwareTushareClient"
)
methods = {item.name for item in cls.body if isinstance(item, ast.FunctionDef)}
self.assertNotIn("__getattr__", methods)
self.assertIn("query", methods)
self.assertTrue(any(base.id == "DashboardMixin" for base in cls.bases if isinstance(base, ast.Name)))
def test_production_python_does_not_embed_blocked_hosts(self) -> None:
violations = []
for path in (ROOT / "backend").rglob("*.py"):
relative = path.relative_to(ROOT).as_posix()
if relative in LEFTOVER_WEBSITE_FILES:
continue
text = path.read_text(encoding="utf-8")
for host in BLOCKED_HOSTS:
if host in text:
violations.append(f"{relative} -> {host}")
self.assertEqual(violations, [])
def test_website_runtime_does_not_call_blocked_hosts_from_gateway(self) -> None:
gateway_src = (ROOT / "backend" / "data" / "gateway.py").read_text(encoding="utf-8")
self.assertNotIn("TushareProvider", gateway_src)
self.assertIsNone(re.search(r"(?<![A-Za-z])TushareClient\(", gateway_src))
self.assertIn("DatahubAwareTushareClient", gateway_src)
def test_bridge_query_has_no_legacy_call(self) -> None:
source = (ROOT / "backend" / "data" / "datahub" / "bridge.py").read_text(encoding="utf-8")
tree = ast.parse(source)
query_fn = next(
node
for node in tree.body
if isinstance(node, ast.ClassDef) and node.name == "DatahubBridge"
for item in node.body
if isinstance(item, ast.FunctionDef) and item.name == "query"
)
called = [
ast.unparse(item.func) if hasattr(ast, "unparse") else ""
for item in ast.walk(query_fn)
if isinstance(item, ast.Call)
]
self.assertTrue(any("query_api" in text for text in called))
self.assertFalse(any("legacy_query" in text for text in called))
def test_build_gateway_uses_hub_proxies_without_opening_external_hosts(self) -> None:
settings = _enabled_settings()
with patch("urllib.request.urlopen", blocked_urlopen):
gateway = build_data_gateway({"tushare_token": "tok"}, datahub_settings=settings)
hub_client = DatahubClient(settings, urlopen=blocked_urlopen)
gateway.datahub.client = hub_client
rows = gateway.ifind.wencai("涨停")
quotes = gateway.realtime_observer.tencent_indices()
chart = gateway.chart_data.stock_daily("600000", "20240902")
market = gateway.tushare()
market_quotes = market.try_quotes(["600000.SH"])
auction = market.query("stk_auction", {"trade_date": "20240902"}, "")
self.assertEqual(rows[0]["涨停原因"], "重组")
self.assertEqual(len(quotes), 3)
self.assertEqual(chart[-1]["close"], 10.2)
self.assertEqual(market_quotes[0]["close"], 10.2)
self.assertEqual(auction[0]["close"], 10.2)
self.assertIsNone(market.realtime_aggregator)
self.assertEqual(market.token, "datahub")
def test_website_does_not_accept_or_forward_provider_credentials(self) -> None:
proxy = HubIfindProxy(DatahubBridge(_enabled_settings(), FakeClient()))
self.assertFalse(hasattr(proxy, "set_credentials"))
client_source = (ROOT / "backend" / "data" / "datahub" / "client.py").read_text(encoding="utf-8")
application_source = (ROOT / "backend" / "application.py").read_text(encoding="utf-8")
self.assertNotIn("/v1/credentials", client_source)
self.assertNotIn("ifind_refresh_token", application_source)
def test_site_configuration_depends_on_hub_not_provider_credentials(self) -> None:
service = SystemServiceMixin()
service._system_credentials = {}
service.data_gateway = build_data_gateway({}, datahub_settings=_enabled_settings())
self.assertTrue(service.configured)
self.assertEqual(service.token, "datahub")
def test_compose_passes_ifind_env_to_hub(self) -> None:
overlay = (ROOT / "compose.datahub.yaml").read_text(encoding="utf-8")
standalone = (ROOT / "xiaobai-datahub" / "compose.yaml").read_text(encoding="utf-8")
for text in (overlay, standalone):
self.assertIn('IFIND_REFRESH_TOKEN: "${IFIND_REFRESH_TOKEN:-}"', text)
self.assertIn('IFIND_ACCESS_TOKEN: "${IFIND_ACCESS_TOKEN:-}"', text)
if __name__ == "__main__":
unittest.main()
+54 -3
View File
@@ -6,12 +6,26 @@ from datetime import date, datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
from backend.features.market.charts import EastmoneyChartClient, MarketChartClient from backend.features.market.charts import MarketChartClient
from database import ReviewDatabase from database import ReviewDatabase
from backend.features.market.insights import MarketInsightsService from backend.features.market.insights import MarketInsightsService
from server import DashboardService from server import DashboardService
class _FakeDailyHub:
def __init__(self, rows: list) -> None:
self.rows = rows
def try_daily_chart(self, code, end_date, limit, dataset="daily"):
return list(self.rows)
def try_quotes(self, codes):
return None
def try_index_quotes(self):
return None
class FakeIfind: class FakeIfind:
configured = True configured = True
@@ -128,13 +142,50 @@ class IfindFeatureTests(unittest.TestCase):
self.assertEqual(database.list_wencai_saved_queries(second["id"]), []) self.assertEqual(database.list_wencai_saved_queries(second["id"]), [])
def test_ifind_daily_chart_normalizes_change(self): def test_ifind_daily_chart_normalizes_change(self):
client = MarketChartClient(FakeIfind(), EastmoneyChartClient()) hub = _FakeDailyHub(
[
{
"trade_date": "2026-07-27",
"open": 10,
"high": 10.5,
"low": 9.8,
"close": 10.2,
"volume": 100,
"amount_billion": 0.01,
"change": 0,
},
{
"trade_date": "2026-07-28",
"open": 10.2,
"high": 10.8,
"low": 10.1,
"close": 10.5,
"volume": 120,
"amount_billion": 0.012,
"change": 2.9412,
},
]
)
client = MarketChartClient(hub)
rows = client.stock_daily("000001", "20260728") rows = client.stock_daily("000001", "20260728")
self.assertEqual(rows[-1]["trade_date"], "2026-07-28") self.assertEqual(rows[-1]["trade_date"], "2026-07-28")
self.assertAlmostEqual(rows[-1]["change"], 2.9412, places=4) self.assertAlmostEqual(rows[-1]["change"], 2.9412, places=4)
def test_ifind_daily_chart_keeps_last_traded_bar_before_market_open(self): def test_ifind_daily_chart_keeps_last_traded_bar_before_market_open(self):
client = MarketChartClient(FakeIfindStalePreopen(), EastmoneyChartClient()) hub = _FakeDailyHub(
[
{
"trade_date": "2026-07-28",
"open": 10.2,
"high": 10.8,
"low": 10.1,
"close": 10.5,
"volume": 120,
"amount_billion": 0.012,
}
]
)
client = MarketChartClient(hub)
with patch("backend.features.market.charts.datetime", FixedPreopenDatetime): with patch("backend.features.market.charts.datetime", FixedPreopenDatetime):
rows = client.stock_daily("000001", "20260729") rows = client.stock_daily("000001", "20260729")
+1 -1
View File
@@ -58,7 +58,7 @@ class MobileSystemPagesRegressionTests(unittest.TestCase):
"data-system-edit-model", "data-system-edit-model",
"data-system-open-member", "data-system-open-member",
"管理员专区", "管理员专区",
"保存密钥", "刷新状态",
"保存分工", "保存分工",
'location.assign("/login/")', 'location.assign("/login/")',
): ):
+323
View File
@@ -1,8 +1,16 @@
from __future__ import annotations from __future__ import annotations
import unittest import unittest
from datetime import datetime, timedelta, timezone
from backend.data.providers.tushare_client import TushareClient from backend.data.providers.tushare_client import TushareClient
from backend.data.providers.tushare_helpers import calendar_is_open
from backend.data.providers.tushare_transport import TushareError
from backend.data.realtime import (
RealtimeAggregateError,
_normalize_eastmoney_quote,
_parse_tencent_stock_quote,
)
class FakeRealtimeClient(TushareClient): class FakeRealtimeClient(TushareClient):
@@ -81,6 +89,72 @@ class FakeRealtimeClient(TushareClient):
raise AssertionError(f"Unexpected API call: {api_name} {params}") raise AssertionError(f"Unexpected API call: {api_name} {params}")
FREE_QUOTES = [
{
"ts_code": "000001.SZ", "name": "", "pre_close": 10.0,
"open": 10.1, "high": 11.0, "low": 10.0, "close": 11.0,
"vol": 1000, "amount": 100000000, "num": 10,
"quote_date": "20260720",
},
{
"ts_code": "000002.SZ", "name": "", "pre_close": 20.0,
"open": 19.5, "high": 20.0, "low": 18.0, "close": 18.0,
"vol": 2000, "amount": 200000000, "num": 20,
"quote_date": "20260720",
},
{
"ts_code": "000003.SZ", "name": "", "pre_close": 30.0,
"open": 31.0, "high": 33.0, "low": 30.0, "close": 32.0,
"vol": 3000, "amount": 300000000, "num": 30,
"quote_date": "20260720",
},
]
class FakeFreeAggregator:
def __init__(self, quotes=None, fail=False):
self.quotes = list(quotes if quotes is not None else FREE_QUOTES)
self.fail = fail
self.calls = 0
def eastmoney_market_quotes(self, expected_date=""):
self.calls += 1
if self.fail:
raise RealtimeAggregateError("eastmoney down")
if expected_date and self.quotes:
dated = [
row for row in self.quotes
if str(row.get("quote_date") or "") == str(expected_date).replace("-", "")
]
if dated:
return dated
return list(self.quotes)
def tencent_market_quotes(self, codes, expected_date=""):
return self.eastmoney_market_quotes(expected_date)
def tencent_stock_quotes(self, codes, expected_date="", minimum=None):
wanted = {str(code).strip() for code in (codes or []) if str(code).strip()}
rows = self.tencent_market_quotes(codes, expected_date)
if not wanted:
return rows
return [row for row in rows if str(row.get("ts_code") or "") in wanted]
def eastmoney_indices(self):
return [
{
"code": "000001",
"name": "上证指数",
"price": 3800.12,
"change": 0.85,
"previous_close": 3768.0,
"amount_billion": 4200.5,
"quote_time": "2026-07-20T10:05:00+08:00",
"source": "eastmoney_push2",
}
]
class RealtimeDashboardTests(unittest.TestCase): class RealtimeDashboardTests(unittest.TestCase):
def setUp(self): def setUp(self):
TushareClient._realtime_reference_cache.clear() TushareClient._realtime_reference_cache.clear()
@@ -130,6 +204,255 @@ class RealtimeDashboardTests(unittest.TestCase):
self.assertEqual(dashboard["meta"]["limit_data_source"], "derived") self.assertEqual(dashboard["meta"]["limit_data_source"], "derived")
self.assertIn("日线数据推算", dashboard["meta"]["notice"]) self.assertIn("日线数据推算", dashboard["meta"]["notice"])
def test_calendar_open_flag_accepts_string_and_bool(self):
self.assertTrue(calendar_is_open(1))
self.assertTrue(calendar_is_open("1"))
self.assertTrue(calendar_is_open(True))
self.assertFalse(calendar_is_open(0))
self.assertFalse(calendar_is_open("0"))
self.assertFalse(calendar_is_open(False))
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "trade_cal":
return [
{
"cal_date": params.get("start_date"),
"is_open": "1",
"pretrade_date": "20260907",
}
]
return original_query(api_name, params, fields)
self.client.query = query
trade_date, previous = self.client.resolve_trade_context("20260908")
self.assertEqual(trade_date, "20260908")
self.assertEqual(previous, "20260907")
def test_session_clock_uses_realtime_until_official_window(self):
today = "20260908"
self.client.clock = lambda: datetime(
2026, 9, 8, 10, 5, tzinfo=timezone(timedelta(hours=8))
)
self.assertTrue(self.client.should_use_realtime(today, today))
self.client.clock = lambda: datetime(
2026, 9, 8, 16, 10, tzinfo=timezone(timedelta(hours=8))
)
self.assertFalse(self.client.should_use_realtime(today, today))
def test_realtime_dashboard_survives_missing_limit_table(self):
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "stk_limit":
return []
return original_query(api_name, params, fields)
self.client.query = query
TushareClient._realtime_reference_cache.clear()
dashboard = self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertTrue(dashboard["meta"]["realtime"])
self.assertEqual(dashboard["meta"]["quote_count"], 3)
self.assertEqual(dashboard["overview"]["limit_up_count"], 0)
def test_hub_quotes_used_when_rt_k_denied(self):
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "rt_k":
raise TushareError("没有接口访问权限")
return original_query(api_name, params, fields)
self.client.query = query
self.client.try_market_quotes = lambda trade_date: list(FREE_QUOTES)
TushareClient._realtime_reference_cache.clear()
dashboard = self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertTrue(dashboard["meta"]["realtime"])
self.assertEqual(dashboard["meta"]["quote_source"], "datahub")
self.assertEqual(dashboard["meta"]["trade_date"], "2026-07-20")
self.assertEqual(dashboard["meta"]["quote_count"], 3)
self.assertEqual(dashboard["overview"]["limit_up_count"], 1)
self.assertEqual(dashboard["overview"]["limit_down_count"], 1)
self.assertEqual(dashboard["overview"]["amount_billion"], 6.0)
self.assertIn("数据中枢", dashboard["meta"]["notice"])
def test_hub_quotes_used_when_rt_k_empty(self):
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "rt_k":
return []
return original_query(api_name, params, fields)
self.client.query = query
self.client.try_market_quotes = lambda trade_date: list(FREE_QUOTES)
TushareClient._realtime_reference_cache.clear()
dashboard = self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertEqual(dashboard["meta"]["quote_source"], "datahub")
self.assertEqual(str(dashboard["meta"]["trade_date"]).replace("-", ""), "20260720")
def test_hub_failure_keeps_today_error(self):
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "rt_k":
raise TushareError("数据中枢行情暂不可用")
return original_query(api_name, params, fields)
self.client.query = query
TushareClient._realtime_reference_cache.clear()
with self.assertRaises(TushareError) as ctx:
self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertIn("当天盘中实时行情不可用", str(ctx.exception))
def test_hub_failover_is_invisible_to_website(self):
original_query = self.client.query
def query(api_name, params=None, fields=""):
if api_name == "rt_k":
raise TushareError("没有接口访问权限")
return original_query(api_name, params, fields)
self.client.query = query
self.client.try_market_quotes = lambda trade_date: list(FREE_QUOTES)
TushareClient._realtime_reference_cache.clear()
dashboard = self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertEqual(dashboard["meta"]["quote_source"], "datahub")
self.assertEqual(str(dashboard["meta"]["trade_date"]).replace("-", ""), "20260720")
self.assertIn("数据中枢", dashboard["meta"]["notice"])
self.assertEqual(dashboard["overview"]["amount_billion"], 6.0)
def test_normalize_eastmoney_quote_maps_units_and_exchange(self):
quote = _normalize_eastmoney_quote(
{
"f12": "600000",
"f13": 1,
"f14": "浦发银行",
"f2": 10.5,
"f5": 12.0,
"f6": 200000000,
"f15": 10.8,
"f16": 10.2,
"f17": 10.3,
"f18": 10.0,
"f124": 1752986700,
}
)
self.assertEqual(quote["ts_code"], "600000.SH")
self.assertEqual(quote["vol"], 1200)
self.assertEqual(quote["close"], 10.5)
self.assertEqual(quote["pre_close"], 10.0)
self.assertEqual(quote["source"], "eastmoney_clist")
def test_parse_tencent_stock_quote_keeps_today_and_units(self):
line = (
'v_sz000001="51~平安银行~000001~11.73~11.70~11.66~346232~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~0~'
'~20260720100500~0.03~0.26~11.79~11.65~11.73/346232/406045563~346232~40605~0.18~5.24~~11.79~11.65~1.20~'
'2276.29~2276.31~0.49~12.87~10.53~0.95~-3076~11.73~4.43~5.34~~~0.18~40604.5563~0.0000~0~";'
)
quote = _parse_tencent_stock_quote(line)
self.assertEqual(quote["ts_code"], "000001.SZ")
self.assertEqual(quote["quote_date"], "20260720")
self.assertEqual(quote["close"], 11.73)
self.assertEqual(quote["pre_close"], 11.70)
self.assertEqual(quote["vol"], 34623200)
self.assertEqual(quote["amount"], 406050000)
self.assertEqual(quote["source"], "tencent_qt")
def test_datahub_market_quotes_used_before_legacy(self):
calls = []
def try_market_quotes(trade_date):
calls.append(trade_date)
return list(FREE_QUOTES)
self.client.try_market_quotes = try_market_quotes
self.client.realtime_aggregator = FakeFreeAggregator(fail=True)
TushareClient._realtime_reference_cache.clear()
dashboard = self.client._realtime_dashboard("20260720", "20260720", "20260717")
self.assertEqual(calls, ["20260720"])
self.assertEqual(dashboard["meta"]["quote_source"], "datahub")
self.assertEqual(dashboard["meta"]["source"], "datahub")
self.assertEqual(dashboard["meta"]["quote_count"], 3)
self.assertIn("数据中枢", dashboard["meta"]["notice"])
def test_gateway_dashboard_uses_bound_market_quotes(self) -> None:
from backend.data import build_data_gateway
from backend.data.datahub.bridge import DatahubAwareTushareClient
from backend.data.datahub.client import DatahubResponse
from backend.data.datahub.settings import DATASETS, DatahubSettings, DatasetFlags
quotes = [
{
"ts_code": item["ts_code"],
"name": item["name"],
"pre_close": item["pre_close"],
"open": item["open"],
"high": item["high"],
"low": item["low"],
"close": item["close"],
"vol": item["vol"],
"amount": item["amount"],
"quote_date": "20260720",
}
for item in FREE_QUOTES
]
extras = [
{
"ts_code": f"{index:06d}.SZ",
"name": f"X{index}",
"pre_close": 10.0,
"open": 10.0,
"high": 10.2,
"low": 9.8,
"close": 10.1,
"vol": 100.0,
"amount": 1000.0,
"quote_date": "20260720",
}
for index in range(10, 230)
]
class QuoteHub:
def __init__(self):
self.calls = []
def query_api(self, api_name, params=None, fields=""):
rows = FakeRealtimeClient("tok").query(api_name, params or {}, fields)
return DatahubResponse(
data=rows,
meta={"source": "datahub", "stale": False, "staleness_seconds": 0, "row_shape": "tushare"},
)
def quotes_latest(self, **params):
return self.get("/v1/quotes/latest", params)
def get(self, path, params=None):
self.calls.append(path)
if path == "/v1/quotes/latest":
return DatahubResponse(
data=quotes + extras,
meta={"stale": False, "staleness_seconds": 0, "source": "eastmoney_clist"},
)
raise AssertionError(path)
datasets = {name: DatasetFlags(name) for name in DATASETS}
datasets["quotes"] = DatasetFlags("quotes", read=True, shadow=False)
settings = DatahubSettings(base_url="http://127.0.0.1:9", token="tok", datasets=datasets)
gateway = build_data_gateway({"tushare_token": "tok"}, datahub_settings=settings)
gateway.datahub.client = QuoteHub()
wrapped = gateway.tushare()
wrapped.clock = lambda: datetime(2026, 7, 20, 10, 30, tzinfo=timezone(timedelta(hours=8)))
wrapped.realtime_aggregator = FakeFreeAggregator(fail=True)
DatahubAwareTushareClient._realtime_reference_cache.clear()
dashboard = wrapped.dashboard("20260720")
self.assertEqual(dashboard["meta"]["quote_source"], "datahub")
self.assertIn("/v1/quotes/latest", gateway.datahub.client.calls)
self.assertTrue(callable(getattr(type(wrapped), "try_market_quotes", None)))
self.assertFalse(hasattr(wrapped, "_legacy"))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+318 -9
View File
@@ -5,6 +5,10 @@ import unittest
from datetime import datetime, timedelta from datetime import datetime, timedelta
from unittest.mock import patch from unittest.mock import patch
from backend.data.providers.tushare_client import TushareError
from backend.data.realtime import RealtimeAggregateError
from backend.features.market.charts import ChartDataError
from backend.features.market.service import TODAY_DAILY_UNAVAILABLE_NOTICE
from server import DashboardService from server import DashboardService
@@ -17,6 +21,10 @@ class DetailDatabaseStub:
def list_notes(user_id, code=""): def list_notes(user_id, code=""):
return [] return []
@staticmethod
def get_snapshot(trade_date):
return {}
class RealtimeClientStub: class RealtimeClientStub:
quote_calls = 0 quote_calls = 0
@@ -45,6 +53,18 @@ class RealtimeClientStub:
} }
class DataGatewayStub:
def __init__(self, client):
self.client = client
def tushare(self):
return self.client
@staticmethod
def datahub_status():
return {"configured": True}
class FixedMarketDatetime(datetime): class FixedMarketDatetime(datetime):
fixed_now = datetime(2026, 7, 31, 10, 30).astimezone() fixed_now = datetime(2026, 7, 31, 10, 30).astimezone()
@@ -61,6 +81,122 @@ class FixedPreopenDatetime(datetime):
return cls.fixed_now return cls.fixed_now
class FixedLunchDatetime(datetime):
fixed_now = datetime(2026, 7, 31, 11, 45).astimezone()
@classmethod
def now(cls, tz=None):
return cls.fixed_now
class FixedAfterCloseDatetime(datetime):
fixed_now = datetime(2026, 7, 31, 15, 30).astimezone()
@classmethod
def now(cls, tz=None):
return cls.fixed_now
class DeniedRealtimeClientStub:
quote_calls = 0
def __init__(self, token):
self.token = token
@staticmethod
def resolve_trade_context(requested_date):
return requested_date, requested_date
@classmethod
def realtime_stock_quote(cls, ts_code, reference_date=""):
cls.quote_calls += 1
raise TushareError("没有接口访问权限")
class FreeQuoteAggregator:
def __init__(self, quote=None, fail=False):
self.quote = quote
self.fail = fail
self.tencent_calls = 0
self.eastmoney_calls = 0
def tencent_stock_quote(self, code, expected_date=""):
self.tencent_calls += 1
if self.fail:
raise RealtimeAggregateError("tencent down")
if self.quote and self.quote.get("source") == "eastmoney_stock":
raise RealtimeAggregateError("tencent empty")
if self.quote:
return self.quote
raise RealtimeAggregateError("tencent empty")
def eastmoney_stock_quote(self, code, expected_date=""):
self.eastmoney_calls += 1
if self.fail:
raise RealtimeAggregateError("eastmoney down")
if self.quote and self.quote.get("source") == "eastmoney_stock":
return self.quote
raise RealtimeAggregateError("eastmoney empty")
class IntradayChartStub:
def __init__(self, points, previous_close=10.0, trade_date="2026-07-31"):
self.points = points
self.previous_close = previous_close
self.trade_date = trade_date
def stock_daily(self, code, end_date, limit=90):
raise ChartDataError("iFinD daily unavailable")
def stock_intraday(self, code):
return {
"trade_date": self.trade_date,
"previous_close": self.previous_close,
"points": self.points,
}
def _history_payload(code="002141"):
yesterday = (FixedMarketDatetime.fixed_now - timedelta(days=1)).strftime("%Y-%m-%d")
return {
"meta": {"trade_date": yesterday, "source": "tushare"},
"stock": {"code": code, "name": "旧名称", "price": 10, "change": 7.1},
"prices": [
{
"trade_date": yesterday,
"open": 9.5,
"high": 10.1,
"low": 9.4,
"close": 10,
"change": 7.1,
"volume": 100,
"amount_billion": 1.1,
}
],
"moneyflow": {},
}
def _free_quote(source="tencent_qt", **overrides):
quote = {
"ts_code": "002141.SZ",
"name": "贤程科技",
"pre_close": 10.0,
"open": 10.2,
"high": 10.8,
"low": 10.1,
"close": 10.6,
"vol": 250000,
"amount": 26_500_000,
"quote_date": "20260731",
"quote_time_epoch": int(datetime(2026, 7, 31, 10, 31).timestamp()),
"source": source,
"turnover_rate": 2.5,
}
quote.update(overrides)
return quote
class StockDetailRealtimeTests(unittest.TestCase): class StockDetailRealtimeTests(unittest.TestCase):
def setUp(self): def setUp(self):
self.service = DashboardService.__new__(DashboardService) self.service = DashboardService.__new__(DashboardService)
@@ -68,7 +204,13 @@ class StockDetailRealtimeTests(unittest.TestCase):
self.service.database = DetailDatabaseStub() self.service.database = DetailDatabaseStub()
self.service._request_context = threading.local() self.service._request_context = threading.local()
self.service._request_context.user_id = 1 self.service._request_context.user_id = 1
self.service.ifind = None
self.service.realtime_aggregator = None
self.service.chart_data = None
self.service._market_client_override = RealtimeClientStub("datahub")
self.service.data_gateway = DataGatewayStub(self.service._market_client_override)
RealtimeClientStub.quote_calls = 0 RealtimeClientStub.quote_calls = 0
DeniedRealtimeClientStub.quote_calls = 0
def test_today_detail_merges_rt_quote_without_mutating_daily_cache(self): def test_today_detail_merges_rt_quote_without_mutating_daily_cache(self):
today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d") today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d")
@@ -90,9 +232,7 @@ class StockDetailRealtimeTests(unittest.TestCase):
"moneyflow": {}, "moneyflow": {},
} }
with patch("backend.features.market.service.datetime", FixedMarketDatetime), patch( with patch("backend.features.market.service.datetime", FixedMarketDatetime):
"backend.features.market.service.TushareClient", RealtimeClientStub
):
result = self.service._prepare_stock_detail(cached, "002141", today) result = self.service._prepare_stock_detail(cached, "002141", today)
self.assertEqual(result["meta"]["trade_date"], FixedMarketDatetime.fixed_now.strftime("%Y-%m-%d")) self.assertEqual(result["meta"]["trade_date"], FixedMarketDatetime.fixed_now.strftime("%Y-%m-%d"))
@@ -112,9 +252,7 @@ class StockDetailRealtimeTests(unittest.TestCase):
"stock": {"code": "002141", "price": 10, "change": 1.2}, "stock": {"code": "002141", "price": 10, "change": 1.2},
"prices": [{"trade_date": historical, "close": 10, "change": 1.2}], "prices": [{"trade_date": historical, "close": 10, "change": 1.2}],
} }
with patch("backend.features.market.service.datetime", FixedMarketDatetime), patch( with patch("backend.features.market.service.datetime", FixedMarketDatetime):
"backend.features.market.service.TushareClient", RealtimeClientStub
):
result = self.service._prepare_stock_detail(payload, "002141", historical) result = self.service._prepare_stock_detail(payload, "002141", historical)
self.assertEqual(result["stock"]["change"], 1.2) self.assertEqual(result["stock"]["change"], 1.2)
@@ -151,9 +289,7 @@ class StockDetailRealtimeTests(unittest.TestCase):
}, },
], ],
} }
with patch("backend.features.market.service.datetime", FixedPreopenDatetime), patch( with patch("backend.features.market.service.datetime", FixedPreopenDatetime):
"backend.features.market.service.TushareClient", RealtimeClientStub
):
result = self.service._prepare_stock_detail(payload, "002141", today) result = self.service._prepare_stock_detail(payload, "002141", today)
self.assertEqual(result["meta"]["trade_date"], yesterday) self.assertEqual(result["meta"]["trade_date"], yesterday)
@@ -162,6 +298,179 @@ class StockDetailRealtimeTests(unittest.TestCase):
self.assertEqual(result["stock"]["change"], 1.2) self.assertEqual(result["stock"]["change"], 1.2)
self.assertEqual(RealtimeClientStub.quote_calls, 0) self.assertEqual(RealtimeClientStub.quote_calls, 0)
def test_today_detail_falls_back_to_tencent_quote_when_rt_k_denied(self):
today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d")
aggregator = FreeQuoteAggregator(_free_quote())
self.service.realtime_aggregator = aggregator
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedMarketDatetime):
result = self.service._prepare_stock_detail(_history_payload(), "002141", today)
bar = result["prices"][-1]
self.assertEqual(bar["trade_date"], "2026-07-31")
self.assertTrue(bar["realtime"])
self.assertEqual(bar["open"], 10.2)
self.assertEqual(bar["high"], 10.8)
self.assertEqual(bar["low"], 10.1)
self.assertEqual(bar["close"], 10.6)
self.assertAlmostEqual(bar["change"], 6.0, places=4)
self.assertEqual(bar["volume"], 2500)
self.assertAlmostEqual(bar["amount_billion"], 0.265)
self.assertEqual(len(result["prices"]), 2)
self.assertEqual(result["meta"]["notice"], "")
self.assertEqual(aggregator.tencent_calls, 1)
self.assertEqual(DeniedRealtimeClientStub.quote_calls, 1)
def test_today_detail_falls_back_to_eastmoney_then_intraday(self):
today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d")
aggregator = FreeQuoteAggregator(
_free_quote(
"eastmoney_stock",
ts_code="600000.SH",
name="浦发银行",
net_mf_amount=12,
),
)
self.service.realtime_aggregator = aggregator
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedMarketDatetime):
result = self.service._prepare_stock_detail(_history_payload("600000"), "600000", today)
self.assertEqual(result["prices"][-1]["trade_date"], "2026-07-31")
self.assertEqual(result["prices"][-1]["close"], 10.6)
self.assertEqual(aggregator.tencent_calls, 1)
self.assertEqual(aggregator.eastmoney_calls, 1)
aggregator = FreeQuoteAggregator(fail=True)
self.service.realtime_aggregator = aggregator
self.service.chart_data = IntradayChartStub(
[
{
"date": "2026-07-31",
"time": "09:30",
"open": 10.1,
"high": 10.2,
"low": 10.0,
"close": 10.15,
"volume": 120,
"amount": 121800,
},
{
"date": "2026-07-31",
"time": "10:05",
"open": 10.15,
"high": 10.5,
"low": 9.9,
"close": 10.4,
"volume": 80,
"amount": 83200,
},
]
)
with patch("backend.features.market.service.datetime", FixedMarketDatetime):
result = self.service._prepare_stock_detail(_history_payload("300750"), "300750", today)
bar = result["prices"][-1]
self.assertEqual(bar["trade_date"], "2026-07-31")
self.assertEqual(bar["open"], 10.1)
self.assertEqual(bar["high"], 10.5)
self.assertEqual(bar["low"], 9.9)
self.assertEqual(bar["close"], 10.4)
self.assertAlmostEqual(bar["change"], 4.0, places=4)
self.assertEqual(bar["volume"], 200)
self.assertTrue(bar["realtime"])
def test_today_detail_keeps_history_when_free_sources_fail(self):
today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d")
self.service.realtime_aggregator = FreeQuoteAggregator(fail=True)
self.service.chart_data = IntradayChartStub([], trade_date="2026-07-30")
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedMarketDatetime):
result = self.service._prepare_stock_detail(_history_payload(), "002141", today)
self.assertEqual(result["prices"][-1]["trade_date"], "2026-07-30")
self.assertFalse(result["meta"].get("realtime", False))
self.assertEqual(result["meta"]["notice"], TODAY_DAILY_UNAVAILABLE_NOTICE)
self.assertEqual(len(result["prices"]), 1)
def test_lunch_keeps_morning_realtime_bar(self):
today = FixedLunchDatetime.fixed_now.strftime("%Y%m%d")
self.service.realtime_aggregator = FreeQuoteAggregator(
_free_quote(quote_time_epoch=int(datetime(2026, 7, 31, 11, 30).timestamp()))
)
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedLunchDatetime):
result = self.service._prepare_stock_detail(_history_payload(), "002141", today)
self.assertEqual(result["prices"][-1]["trade_date"], "2026-07-31")
self.assertTrue(result["meta"]["realtime"])
def test_after_close_keeps_forming_bar_until_official_ready(self):
today = FixedAfterCloseDatetime.fixed_now.strftime("%Y%m%d")
self.service.realtime_aggregator = FreeQuoteAggregator(_free_quote())
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedAfterCloseDatetime):
forming = self.service._prepare_stock_detail(_history_payload(), "002141", today)
self.assertEqual(forming["prices"][-1]["trade_date"], "2026-07-31")
self.assertTrue(forming["prices"][-1]["realtime"])
official = _history_payload()
official["prices"].append(
{
"trade_date": "2026-07-31",
"open": 10.15,
"high": 10.9,
"low": 10.05,
"close": 10.7,
"change": 7.0,
"volume": 1800,
"amount_billion": 0.3,
}
)
RealtimeClientStub.quote_calls = 0
self.service._market_client_override = RealtimeClientStub("datahub")
with patch("backend.features.market.service.datetime", FixedAfterCloseDatetime):
replaced = self.service._prepare_stock_detail(official, "002141", today)
self.assertEqual(replaced["prices"][-1]["close"], 10.7)
self.assertFalse(replaced["prices"][-1].get("realtime", False))
self.assertEqual(len(replaced["prices"]), 2)
self.assertEqual(RealtimeClientStub.quote_calls, 0)
def test_same_date_bar_is_replaced_not_duplicated(self):
today = FixedMarketDatetime.fixed_now.strftime("%Y%m%d")
payload = _history_payload()
payload["prices"].append(
{
"trade_date": "2026-07-31",
"open": 10.0,
"high": 10.1,
"low": 9.9,
"close": 10.05,
"change": 0.5,
"volume": 10,
"amount_billion": 0.01,
"realtime": True,
}
)
self.service.realtime_aggregator = FreeQuoteAggregator(_free_quote())
self.service._market_client_override = DeniedRealtimeClientStub("datahub")
DeniedRealtimeClientStub.quote_calls = 0
with patch("backend.features.market.service.datetime", FixedMarketDatetime):
result = self.service._prepare_stock_detail(payload, "002141", today)
self.assertEqual(len(result["prices"]), 2)
self.assertEqual(result["prices"][-1]["close"], 10.6)
self.assertEqual(result["prices"][-1]["trade_date"], "2026-07-31")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+6 -7
View File
@@ -218,11 +218,10 @@ def build() -> dict[str, Any]:
), ),
"external_data_adapters": [ "external_data_adapters": [
{"provider": "tushare", "path": "backend/data/providers/tushare_client.py", "runtime_role": "stable client facade for primary deterministic market data"}, {"provider": "tushare", "path": "backend/data/providers/tushare_client.py", "runtime_role": "stable client facade for primary deterministic market data"},
{"provider": "datahub", "path": "backend/data/datahub/client.py", "runtime_role": "optional official EOD read path behind per-dataset flags"}, {"provider": "datahub", "path": "backend/data/datahub/client.py", "runtime_role": "website-only read path; official EOD, live quotes, and licensed iFinD"},
{"provider": "ifind", "path": "backend/data/providers/ifind_client.py", "runtime_role": "realtime, charts, snapshots, enrichment"}, {"provider": "ifind", "path": "xiaobai-datahub/datahub/adapters/ifind.py", "runtime_role": "licensed iFinD source inside the data hub"},
{"provider": "eastmoney", "path": "backend/features/market/charts.py", "runtime_role": "display chart fallback"}, {"provider": "eastmoney", "path": "xiaobai-datahub/datahub/adapters/eastmoney.py", "runtime_role": "free realtime quotes and shenwan inside the data hub"},
{"provider": "eastmoney", "path": "backend/data/realtime.py", "runtime_role": "isolated realtime observation"}, {"provider": "tencent", "path": "xiaobai-datahub/datahub/adapters/tencent.py", "runtime_role": "free index and stock quotes inside the data hub"},
{"provider": "tencent", "path": "backend/data/realtime.py", "runtime_role": "index observation fallback"},
], ],
"provider_domains": [ "provider_domains": [
{"provider": "tushare", "path": "backend/data/providers/tushare_transport.py", "responsibility": "HTTP transport and provider errors"}, {"provider": "tushare", "path": "backend/data/providers/tushare_transport.py", "responsibility": "HTTP transport and provider errors"},
@@ -240,9 +239,9 @@ def build() -> dict[str, Any]:
{"client": "DatahubClient", "owner": "backend/data/gateway.py"}, {"client": "DatahubClient", "owner": "backend/data/gateway.py"},
{"client": "DatahubBridge", "owner": "backend/data/gateway.py"}, {"client": "DatahubBridge", "owner": "backend/data/gateway.py"},
{"client": "DatahubAwareTushareClient", "owner": "backend/data/gateway.py"}, {"client": "DatahubAwareTushareClient", "owner": "backend/data/gateway.py"},
{"client": "IfindHttpClient", "owner": "backend/data/gateway.py"}, {"client": "HubIfindProxy", "owner": "backend/data/gateway.py"},
{"client": "HubRealtimeProxy", "owner": "backend/data/gateway.py"},
{"client": "MarketChartClient", "owner": "backend/data/gateway.py"}, {"client": "MarketChartClient", "owner": "backend/data/gateway.py"},
{"client": "WebRealtimeAggregator", "owner": "backend/data/gateway.py"},
], ],
"heaven_service_owners": { "heaven_service_owners": {
"facade": "backend/features/heaven/service.py", "facade": "backend/features/heaven/service.py",
+4
View File
@@ -10,4 +10,8 @@ DATAHUB_ADMIN_PASSWORD=
# Tushare Pro token. Stored encrypted after first launch; never returned by API or admin pages. # Tushare Pro token. Stored encrypted after first launch; never returned by API or admin pages.
TUSHARE_TOKEN= TUSHARE_TOKEN=
# Optional licensed iFinD tokens. Used only inside the hub; the website never dials 51ifind.com.
IFIND_REFRESH_TOKEN=
IFIND_ACCESS_TOKEN=
TZ=Asia/Shanghai TZ=Asia/Shanghai
+3 -2
View File
@@ -6,11 +6,12 @@
## 做什么 ## 做什么
- SQLite WAL `datahub.db`,容器名 `xiaobai-datahub`,端口 `8766` - SQLite WAL `datahub.db`,容器名 `xiaobai-datahub`,端口 `8766`
- Tushare 盘后正式数据:交易日历、股票主档、daily、daily_basic、adj_factor、index_daily、moneyflow、stk_auction - Tushare 盘后正式数据:交易日历、股票主档、daily、daily_basic、adj_factor、index_daily、moneyflow、stk_auction、limit_list_d、ths_hot/dc_hot、hm_detail、ths_daily/dc_index/sw_daily
- 盘中观察(provisional):东财/腾讯指数报价、个股最新价、全市场快照、分时点(`/v1/quotes/latest` 不传 codes 即全市场,`/v1/indexes/quotes` `/v1/intraday/points`);永不写入 eod_* 正式表
- 暂存 → 校验 → 整批原子发布 → 可回滚 - 暂存 → 校验 → 整批原子发布 → 可回滚
- `/v1` 稳定接口(`X-Datahub-Token` - `/v1` 稳定接口(`X-Datahub-Token`
- `/admin/` 最小管理后台(总览 / 数据源 / 调度 / 发布 / 数据集 / 审计) - `/admin/` 最小管理后台(总览 / 数据源 / 调度 / 发布 / 数据集 / 审计)
- 东财/腾讯/同花顺/选股宝/AKShare/iFinD 适配器位预留,本阶段不拉实时源 - 同花顺/选股宝/AKShare/iFinD 适配器位预留;东财/腾讯已接入盘中观察
## 单位口径(相对现站) ## 单位口径(相对现站)
+2
View File
@@ -14,6 +14,8 @@ services:
DATAHUB_TOKEN: "${DATAHUB_TOKEN:?DATAHUB_TOKEN must be set}" DATAHUB_TOKEN: "${DATAHUB_TOKEN:?DATAHUB_TOKEN must be set}"
DATAHUB_ADMIN_PASSWORD: "${DATAHUB_ADMIN_PASSWORD:?DATAHUB_ADMIN_PASSWORD must be set}" DATAHUB_ADMIN_PASSWORD: "${DATAHUB_ADMIN_PASSWORD:?DATAHUB_ADMIN_PASSWORD must be set}"
TUSHARE_TOKEN: "${TUSHARE_TOKEN:-}" TUSHARE_TOKEN: "${TUSHARE_TOKEN:-}"
IFIND_REFRESH_TOKEN: "${IFIND_REFRESH_TOKEN:-}"
IFIND_ACCESS_TOKEN: "${IFIND_ACCESS_TOKEN:-}"
DATAHUB_DB_PATH: /app/data/datahub.db DATAHUB_DB_PATH: /app/data/datahub.db
DATAHUB_BACKUP_DIR: /app/data/backups DATAHUB_BACKUP_DIR: /app/data/backups
TZ: Asia/Shanghai TZ: Asia/Shanghai
@@ -14,6 +14,7 @@
"list_limit_max": 5000, "list_limit_max": 5000,
"calendar_start": "20160101", "calendar_start": "20160101",
"index_history_trading_days": 260, "index_history_trading_days": 260,
"daily_history_trading_days": 250,
"eod_retry_start": "15:15", "eod_retry_start": "15:15",
"eod_retry_interval_minutes": 30, "eod_retry_interval_minutes": 30,
"eod_retry_cutoff": "23:30", "eod_retry_cutoff": "23:30",
+4 -4
View File
@@ -1,13 +1,13 @@
from datahub.adapters.akshare import ADAPTER as akshare from datahub.adapters.akshare import ADAPTER as akshare
from datahub.adapters.eastmoney import ADAPTER as eastmoney from datahub.adapters.eastmoney import EastmoneyAdapter
from datahub.adapters.ifind import ADAPTER as ifind from datahub.adapters.ifind import ADAPTER as ifind
from datahub.adapters.tencent import ADAPTER as tencent from datahub.adapters.tencent import TencentAdapter
from datahub.adapters.ths import ADAPTER as ths from datahub.adapters.ths import ADAPTER as ths
from datahub.adapters.xgb import ADAPTER as xgb from datahub.adapters.xgb import ADAPTER as xgb
RESERVED = { RESERVED = {
"eastmoney": eastmoney, "eastmoney": EastmoneyAdapter(),
"tencent": tencent, "tencent": TencentAdapter(),
"ths": ths, "ths": ths,
"xgb": xgb, "xgb": xgb,
"akshare": akshare, "akshare": akshare,
+500 -2
View File
@@ -1,3 +1,501 @@
from datahub.adapters.base import ReservedAdapter from __future__ import annotations
ADAPTER = ReservedAdapter("eastmoney") import json
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime
from typing import Any
from datahub.adapters.base import AdapterError, MarketAdapter
from datahub.numbers import finite_number, round4
EASTMONEY_INDEX_URL = "https://push2.eastmoney.com/api/qt/ulist.np/get"
EASTMONEY_CLIST_URL = "https://push2.eastmoney.com/api/qt/clist/get"
EASTMONEY_A_SHARE_BOARDS = (
"m:0+t:6",
"m:0+t:80",
"m:1+t:2",
"m:1+t:23",
"m:0+t:81",
)
EASTMONEY_QUOTE_FIELDS = "f12,f13,f14,f2,f3,f4,f5,f6,f15,f16,f17,f18,f8,f124"
EASTMONEY_MARKET_PAGE_SIZE = 100
TRENDS_URL = "https://push2delay.eastmoney.com/api/qt/stock/trends2/get"
HIS_TRENDS_URL = "https://push2his.eastmoney.com/api/qt/stock/trends2/get"
BROWSER_UA = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
)
INDEX_SECIDS = {
"000001.SH": "1.000001",
"399001.SZ": "0.399001",
"399006.SZ": "0.399006",
}
class EastmoneyAdapter(MarketAdapter):
name = "eastmoney"
def __init__(self, timeout: int = 8) -> None:
self.timeout = timeout
def probe(self) -> dict[str, Any]:
started = time.perf_counter()
try:
rows = self.fetch_indices()
state = "ok" if len(rows) == 3 else "empty"
except AdapterError as exc:
return {
"provider": self.name,
"configured": True,
"state": "error",
"message": str(exc),
"latency_ms": round((time.perf_counter() - started) * 1000),
}
return {
"provider": self.name,
"configured": True,
"state": state,
"latency_ms": round((time.perf_counter() - started) * 1000),
}
def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]:
if dataset in {"indexes_quotes", "index_quotes"}:
return self.fetch_indices()
if dataset in {"quotes", "quotes_latest"}:
codes = params.get("codes") or []
if isinstance(codes, str):
codes = [item.strip() for item in codes.split(",") if item.strip()]
if codes:
return self.fetch_quotes(list(codes))
return self.fetch_market_quotes()
if dataset in {"quotes_market", "market_quotes"}:
return self.fetch_market_quotes()
raise AdapterError(f"{self.name} unsupported dataset: {dataset}")
def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
return list(rows)
def fetch_indices(self) -> list[dict[str, Any]]:
payload = self._get_json(
EASTMONEY_INDEX_URL,
{
"secids": "1.000001,0.399001,0.399006",
"fltt": "2",
"invt": "2",
"fields": "f12,f14,f2,f3,f4,f15,f16,f17,f18,f6,f124",
},
referer="https://quote.eastmoney.com/",
)
rows = list((payload.get("data") or {}).get("diff") or [])
result = []
for row in rows:
code = str(row.get("f12") or "")
if code not in {"000001", "399001", "399006"}:
continue
epoch = int(finite_number(row.get("f124")) or 0)
ts_code = f"{code}.SH" if code.startswith("0") and code == "000001" else f"{code}.SZ"
if code == "000001":
ts_code = "000001.SH"
result.append(
{
"ts_code": ts_code,
"code": code,
"name": row.get("f14") or code,
"price": round4(finite_number(row.get("f2"))),
"pct_chg": round4(finite_number(row.get("f3"))),
"change_amount": round4(finite_number(row.get("f4"))),
"open": round4(finite_number(row.get("f17"))),
"high": round4(finite_number(row.get("f15"))),
"low": round4(finite_number(row.get("f16"))),
"previous_close": round4(finite_number(row.get("f18"))),
"amount": round4(finite_number(row.get("f6"))),
"quote_time_epoch": epoch,
"quote_time": (
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
if epoch
else ""
),
"source": "eastmoney_push2",
}
)
if len(result) != 3:
raise AdapterError(f"Eastmoney returned {len(result)}/3 indices")
return result
def fetch_quotes(self, codes: list[str]) -> list[dict[str, Any]]:
# Eastmoney ulist.np accepts ~60 secids per request; page remaining codes.
secids = []
for code in codes:
ts = str(code or "").upper()
symbol = ts.split(".")[0]
if ts.endswith(".SH") or symbol.startswith(("5", "6", "9")):
secids.append(f"1.{symbol}")
else:
secids.append(f"0.{symbol}")
if not secids:
return []
result: list[dict[str, Any]] = []
for index in range(0, len(secids), 60):
payload = self._get_json(
EASTMONEY_INDEX_URL,
{
"secids": ",".join(secids[index:index + 60]),
"fltt": "2",
"invt": "2",
"fields": "f12,f14,f2,f3,f4,f15,f16,f17,f18,f5,f6,f8,f124",
},
referer="https://quote.eastmoney.com/",
)
rows = list((payload.get("data") or {}).get("diff") or [])
for row in rows:
symbol = str(row.get("f12") or "")
if not symbol:
continue
ts_code = f"{symbol}.SH" if symbol.startswith(("5", "6", "9")) else f"{symbol}.SZ"
epoch = int(finite_number(row.get("f124")) or 0)
close = round4(finite_number(row.get("f2")))
previous = round4(finite_number(row.get("f18")))
quote_date = (
datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d")
if epoch
else ""
)
result.append(
{
"ts_code": ts_code,
"name": row.get("f14") or symbol,
"price": close,
"close": close,
"pct_chg": round4(finite_number(row.get("f3"))),
"change_amount": round4(finite_number(row.get("f4"))),
"open": round4(finite_number(row.get("f17"))),
"high": round4(finite_number(row.get("f15"))),
"low": round4(finite_number(row.get("f16"))),
"pre_close": previous,
"previous_close": previous,
"volume": round4(finite_number(row.get("f5"))),
"vol": round4(finite_number(row.get("f5")) * 100),
"amount": round4(finite_number(row.get("f6"))),
"turnover_rate": round4(finite_number(row.get("f8"))),
"quote_date": quote_date,
"quote_time_epoch": epoch,
"quote_time": (
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
if epoch
else ""
),
"source": "eastmoney_push2",
}
)
return result
def fetch_market_quotes(self) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
board_errors: list[str] = []
for board in EASTMONEY_A_SHARE_BOARDS:
try:
rows.extend(self._board_quotes(board))
except Exception as exc:
board_errors.append(f"{board}:{exc}")
quotes: list[dict[str, Any]] = []
seen: set[str] = set()
for row in rows:
quote = _normalize_market_quote(row)
ts_code = str((quote or {}).get("ts_code") or "")
if not quote or ts_code in seen:
continue
seen.add(ts_code)
quotes.append(quote)
if len(quotes) < 200:
detail = f"{'; '.join(board_errors)}" if board_errors else ""
raise AdapterError(f"Eastmoney market snapshot too small: {len(quotes)}{detail}")
return quotes
def _board_quotes(self, board: str) -> list[dict[str, Any]]:
first = self._market_page(board, 1)
data = first.get("data") or {}
rows = list(data.get("diff") or [])
total = int(finite_number(data.get("total")) or 0)
page_count = 1
if total > 0:
page_count = max(1, (total + EASTMONEY_MARKET_PAGE_SIZE - 1) // EASTMONEY_MARKET_PAGE_SIZE)
for page in range(2, min(page_count, 40) + 1):
payload = self._market_page(board, page)
rows.extend(list((payload.get("data") or {}).get("diff") or []))
return rows
def _market_page(self, board: str, page: int) -> dict[str, Any]:
return self._get_json(
EASTMONEY_CLIST_URL,
{
"pn": str(page),
"pz": str(EASTMONEY_MARKET_PAGE_SIZE),
"po": "1",
"np": "1",
"fltt": "2",
"invt": "2",
"fid": "f12",
"fs": board,
"fields": EASTMONEY_QUOTE_FIELDS,
},
referer="https://quote.eastmoney.com/center/gridlist.html",
)
def fetch_intraday(self, ts_code: str, date: str = "") -> dict[str, Any]:
code = str(ts_code or "").upper()
if code in INDEX_SECIDS:
secid = INDEX_SECIDS[code]
entity = "index"
identifier = code
elif code.startswith("BK") or code.endswith((".TI", ".SI")):
symbol = code.split(".")[0]
secid = f"90.{symbol}"
entity = "board"
identifier = symbol
else:
symbol = code.split(".")[0]
market = "1" if symbol.startswith(("5", "6", "9")) else "0"
secid = f"{market}.{symbol}"
entity = "stock"
identifier = symbol
params = {
"secid": secid,
"fields1": "f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,f13",
"fields2": "f51,f52,f53,f54,f55,f56,f57,f58",
"iscr": "0",
}
data: dict[str, Any] = {}
points: list[dict[str, Any]] = []
last_error: Exception | None = None
for url, ndays in ((TRENDS_URL, "1"), (TRENDS_URL, "5"), (HIS_TRENDS_URL, "5")):
try:
payload = self._get_json(
url,
{**params, "ndays": ndays},
referer="https://quote.eastmoney.com/",
)
except AdapterError as exc:
last_error = exc
continue
data = payload.get("data") or {}
parsed = [point for raw in data.get("trends") or [] if (point := _parse_trend(raw))]
points = _preferred_session(parsed, date)
if points:
break
if not points:
raise AdapterError("No intraday chart data returned") from last_error
return {
"entity_type": entity,
"identifier": identifier,
"ts_code": code if "." in code else f"{identifier}.{'SH' if identifier.startswith(('5','6','9')) else 'SZ'}",
"name": str(data.get("name") or ""),
"code": str(data.get("code") or identifier),
"trade_date": points[-1]["date"],
"previous_close": round4(finite_number(data.get("preClose"))),
"points": points,
"source": "eastmoney_trends2",
}
def fetch_shenwan_quote(self, ts_code: str) -> dict[str, Any]:
code = str(ts_code or "").split(".")[0]
if not code:
raise AdapterError("Invalid Shenwan code")
payload = self._get_json(
EASTMONEY_INDEX_URL,
{
"secids": f"90.{code}",
"fltt": "2",
"invt": "2",
"fields": "f12,f14,f2,f3,f4,f15,f16,f17,f18,f6,f8,f104,f105,f128,f136,f140,f124",
},
referer="https://quote.eastmoney.com/",
)
rows = list((payload.get("data") or {}).get("diff") or [])
row = next((item for item in rows if item), None)
if not row:
raise AdapterError(f"Eastmoney Shenwan quote missing for {code}")
epoch = int(finite_number(row.get("f124")) or 0)
close = round4(finite_number(row.get("f2")))
previous = round4(finite_number(row.get("f18")))
if close <= 0 or previous <= 0:
raise AdapterError(f"Eastmoney Shenwan quote empty for {code}")
quote_time = (
datetime.fromtimestamp(epoch).astimezone().isoformat(timespec="seconds")
if epoch
else ""
)
return {
"ts_code": f"{code}.SI",
"code": f"{code}.SI",
"name": row.get("f14") or code,
"price": close,
"close": close,
"pre_close": previous,
"previous_close": previous,
"open": round4(finite_number(row.get("f17"))),
"high": round4(finite_number(row.get("f15"))),
"low": round4(finite_number(row.get("f16"))),
"change": round4(finite_number(row.get("f3"))),
"pct_change": round4(finite_number(row.get("f3"))),
"pct_chg": round4(finite_number(row.get("f3"))),
"amount": round4(finite_number(row.get("f6"))),
"leader": row.get("f128") or "--",
"leader_code": row.get("f140") or "",
"leading_pct": round4(finite_number(row.get("f136"))),
"up_count": int(finite_number(row.get("f104")) or 0),
"down_count": int(finite_number(row.get("f105")) or 0),
"quote_time": quote_time,
"trade_time": quote_time,
"quote_date": datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d") if epoch else "",
"quote_time_epoch": epoch,
"source": "eastmoney_sw",
}
def fetch_limit_pool(self, trade_date: str = "") -> list[dict[str, Any]]:
day = str(trade_date or "").replace("-", "")
rows: list[dict[str, Any]] = []
for url, limit_type in (
("https://push2ex.eastmoney.com/getTopicZTPool", "U"),
("https://push2ex.eastmoney.com/getTopicZBPool", "Z"),
):
params = {
"ut": "7eea3edcaed734bea9cbfc24409ed989",
"dpt": "wz.ztzt",
"PageIndex": "0",
"PageSize": "200",
"sort": "fbt:asc",
"stat": "1",
}
if day:
params["date"] = day
try:
payload = self._get_json(url, params, referer="https://quote.eastmoney.com/")
except AdapterError:
continue
pool = ((payload.get("data") or {}).get("pool") or []) if isinstance(payload.get("data"), dict) else []
for item in pool:
code = str(item.get("c") or item.get("code") or "")
if not code:
continue
market = str(item.get("m") or item.get("market") or "")
suffix = "SH" if market in {"1", "SH"} or code.startswith(("5", "6", "9")) else "SZ"
first = str(item.get("fbt") or item.get("first_time") or "")
last = str(item.get("lbt") or item.get("last_time") or "")
rows.append(
{
"ts_code": f"{code}.{suffix}",
"limit_type": limit_type,
"first_time": first,
"last_time": last,
"fd_amount": item.get("fund") or item.get("fd_amount"),
"open_times": item.get("zbc") or item.get("open_times"),
"limit_times": item.get("lbc") or item.get("limit_times"),
"turnover_ratio": item.get("hs") or item.get("turnover_ratio"),
"source": "eastmoney_zt_pool",
}
)
if not rows:
raise AdapterError("Eastmoney limit pool empty")
return rows
def _get_json(self, url: str, params: dict[str, str], referer: str) -> dict[str, Any]:
request_url = f"{url}?{urllib.parse.urlencode(params)}"
request = urllib.request.Request(
request_url,
headers={
"Accept": "application/json,text/plain,*/*",
"User-Agent": BROWSER_UA,
"Referer": referer,
},
method="GET",
)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
return json.loads(response.read().decode("utf-8"))
except Exception as exc:
raise AdapterError(f"eastmoney request failed: {exc}") from exc
def _preferred_session(points: list[dict[str, Any]], preferred_date: str = "") -> list[dict[str, Any]]:
if not points:
return []
want = ""
digits = str(preferred_date or "").replace("-", "")[:8]
if len(digits) == 8 and digits.isdigit():
want = f"{digits[:4]}-{digits[4:6]}-{digits[6:8]}"
if want:
matched = [point for point in points if str(point.get("date") or "") == want]
if matched:
return matched
latest = max(str(point.get("date") or "") for point in points)
if not latest:
return points
return [point for point in points if str(point.get("date") or "") == latest]
def _normalize_market_quote(row: dict[str, Any]) -> dict[str, Any] | None:
symbol = str(row.get("f12") or "").strip()
if not symbol.isdigit() or len(symbol) != 6:
return None
close = round4(finite_number(row.get("f2")))
previous_close = round4(finite_number(row.get("f18")))
if close <= 0 or previous_close <= 0:
return None
market = int(finite_number(row.get("f13")) or 0)
if market == 1 or symbol.startswith(("5", "6", "9")):
ts_code = f"{symbol}.SH"
elif symbol.startswith(("4", "8")):
ts_code = f"{symbol}.BJ"
else:
ts_code = f"{symbol}.SZ"
epoch = int(finite_number(row.get("f124")) or 0)
quote_date = ""
if epoch > 0:
quote_date = datetime.fromtimestamp(epoch).astimezone().strftime("%Y%m%d")
return {
"ts_code": ts_code,
"name": row.get("f14") or symbol,
"pre_close": previous_close,
"previous_close": previous_close,
"open": round4(finite_number(row.get("f17"))),
"high": round4(finite_number(row.get("f15"))),
"low": round4(finite_number(row.get("f16"))),
"close": close,
"price": close,
"pct_chg": round4(finite_number(row.get("f3"))),
"vol": round4(finite_number(row.get("f5")) * 100),
"volume": round4(finite_number(row.get("f5")) * 100),
"amount": round4(finite_number(row.get("f6"))),
"quote_date": quote_date,
"quote_time_epoch": epoch,
"source": "eastmoney_clist",
}
def _parse_trend(raw: Any) -> dict[str, Any] | None:
text = str(raw or "")
parts = text.split(",")
if len(parts) < 8:
return None
stamp = parts[0]
try:
when = datetime.strptime(stamp, "%Y-%m-%d %H:%M")
except ValueError:
return None
close = round4(finite_number(parts[2]))
if close <= 0:
return None
return {
"time": when.strftime("%H:%M"),
"date": when.strftime("%Y-%m-%d"),
"open": round4(finite_number(parts[1])),
"close": close,
"high": round4(finite_number(parts[3])),
"low": round4(finite_number(parts[4])),
"avg_price": round4(finite_number(parts[7] if len(parts) > 7 else parts[2])),
"volume": round4(finite_number(parts[5])),
"amount": round4(finite_number(parts[6])),
}
+431 -2
View File
@@ -1,3 +1,432 @@
from datahub.adapters.base import ReservedAdapter from __future__ import annotations
ADAPTER = ReservedAdapter("ifind") import copy
import json
import threading
import time
import urllib.error
import urllib.request
from datetime import datetime, timedelta
from typing import Any, Callable
from datahub.adapters.base import AdapterError, MarketAdapter
UrlOpen = Callable[..., Any]
class IfindAdapter(MarketAdapter):
"""Licensed iFinD source used only inside the data hub."""
name = "ifind"
BASE_URL = "https://quantapi.51ifind.com/api/v1"
AUTH_ENDPOINT = "get_access_token"
AUTH_ERROR_CODES = {-1302, -1303, -1304, -4302, -4303}
def __init__(
self,
refresh_token: str = "",
access_token: str = "",
timeout: int = 15,
urlopen: UrlOpen = urllib.request.urlopen,
) -> None:
self.timeout = max(3, int(timeout))
self._urlopen = urlopen
self._refresh_token = str(refresh_token or "").strip()
self._access_token = str(access_token or "").strip()
self._access_expires_at: datetime | None = None
self._token_lock = threading.Lock()
self._cache_lock = threading.Lock()
self._cache: dict[str, dict[str, Any]] = {}
@property
def configured(self) -> bool:
return bool(self._refresh_token or self._access_token)
def set_credentials(self, refresh_token: str, access_token: str = "") -> None:
refresh_token = str(refresh_token or "").strip()
access_token = str(access_token or "").strip()
with self._token_lock:
refresh_changed = refresh_token != self._refresh_token
self._refresh_token = refresh_token
if access_token or refresh_changed:
self._access_token = access_token
self._access_expires_at = None
if refresh_changed:
with self._cache_lock:
self._cache.clear()
def status(self) -> dict[str, Any]:
return {
"configured": self.configured,
"access_ready": bool(self._access_token),
"access_expires_at": (
self._access_expires_at.isoformat(timespec="seconds")
if self._access_expires_at
else ""
),
}
def probe(self) -> dict[str, Any]:
started = time.perf_counter()
if not self.configured:
return {
"provider": self.name,
"configured": False,
"state": "unconfigured",
"message": "iFinD token 未配置",
"latency_ms": round((time.perf_counter() - started) * 1000),
}
try:
rows = self.real_time("000001.SH", ["latest"], cache_ttl=0)
state = "ok" if rows else "empty"
return {
"provider": self.name,
"configured": True,
"state": state,
"latency_ms": round((time.perf_counter() - started) * 1000),
}
except AdapterError as exc:
return {
"provider": self.name,
"configured": True,
"state": "error",
"message": str(exc),
"latency_ms": round((time.perf_counter() - started) * 1000),
}
def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]:
if dataset == "wencai":
return self.wencai(
str(params.get("query") or params.get("searchstring") or ""),
str(params.get("search_type") or params.get("searchtype") or "stock"),
int(params.get("cache_ttl") or 300),
)
if dataset == "snapshots":
return self.snapshots(
params.get("codes") or "",
_indicators(params.get("indicators")),
str(params.get("start_time") or ""),
str(params.get("end_time") or ""),
int(params.get("cache_ttl") or 8),
)
if dataset == "history":
return self.history(
params.get("codes") or "",
_indicators(params.get("indicators") or ["close", "volume", "amount"]),
str(params.get("start_date") or ""),
str(params.get("end_date") or ""),
int(params.get("cache_ttl") or 300),
)
if dataset == "realtime":
return self.real_time(
params.get("codes") or "",
_indicators(params.get("indicators") or ["latest"]),
int(params.get("cache_ttl") or 10),
)
if dataset == "intraday":
return self.intraday(
str(params.get("code") or params.get("codes") or ""),
str(params.get("start_time") or ""),
str(params.get("end_time") or ""),
int(params.get("cache_ttl") or 20),
)
raise AdapterError(f"{self.name} unsupported dataset: {dataset}")
def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
return list(rows)
def real_time(
self,
codes: str | list[str],
indicators: list[str],
cache_ttl: int = 10,
) -> list[dict[str, Any]]:
code_text = self._codes(codes)
payload = self._request(
"real_time_quotation",
{"codes": code_text, "indicators": ",".join(indicators)},
cache_key=f"rq:{code_text}:{','.join(indicators)}",
cache_ttl=cache_ttl,
)
return self._table_rows(payload)
def history(
self,
codes: str | list[str],
indicators: list[str],
start_date: str,
end_date: str,
cache_ttl: int = 300,
) -> list[dict[str, Any]]:
code_text = self._codes(codes)
payload = self._request(
"cmd_history_quotation",
{
"codes": code_text,
"indicators": ",".join(indicators),
"startdate": self._display_date(start_date),
"enddate": self._display_date(end_date),
"functionpara": {"CPS": "forward1", "Fill": "Omit"},
},
cache_key=f"hq:{code_text}:{start_date}:{end_date}:{','.join(indicators)}",
cache_ttl=cache_ttl,
)
return self._table_rows(payload)
def intraday(
self,
code: str,
start_time: str,
end_time: str,
cache_ttl: int = 20,
) -> list[dict[str, Any]]:
indicators = ["open", "high", "low", "close", "volume", "amount", "avgPrice"]
payload = self._request(
"high_frequency",
{
"codes": self._codes(code),
"indicators": ",".join(indicators),
"starttime": start_time,
"endtime": end_time,
"functionpara": {
"CPS": "forward1",
"Fill": "Previous",
"Timeformat": "LocalTime",
"Interval": "1",
"Limitstart": "09:30:00",
"Limitend": "15:00:00",
},
},
cache_key=f"hf:{code}:{start_time}:{end_time}",
cache_ttl=cache_ttl,
)
return self._table_rows(payload)
def snapshots(
self,
codes: str | list[str],
indicators: list[str],
start_time: str,
end_time: str,
cache_ttl: int = 8,
) -> list[dict[str, Any]]:
code_text = self._codes(codes)
payload = self._request(
"snap_shot",
{
"codes": code_text,
"indicators": ",".join(indicators),
"starttime": start_time,
"endtime": end_time,
},
cache_key=f"ss:{code_text}:{start_time}:{end_time}:{','.join(indicators)}",
cache_ttl=cache_ttl,
)
return self._table_rows(payload)
def wencai(self, query: str, search_type: str = "stock", cache_ttl: int = 300) -> list[dict[str, Any]]:
normalized = " ".join(str(query or "").split())
if not normalized:
raise AdapterError("问财查询不能为空。")
payload = self._request(
"smart_stock_picking",
{"searchstring": normalized, "searchtype": search_type},
cache_key=f"wc:{search_type}:{normalized}",
cache_ttl=cache_ttl,
)
return self._table_rows(payload)
def _request(
self,
endpoint: str,
body: dict[str, Any],
cache_key: str = "",
cache_ttl: int = 0,
) -> dict[str, Any]:
if not self.configured:
raise AdapterError("iFinD 尚未配置。")
if cache_key and cache_ttl > 0:
cached = self._cached(cache_key, cache_ttl)
if cached is not None:
return cached
payload = self._post(endpoint, body, self._ensure_access_token())
if self._is_auth_error(payload) and self._refresh_token:
self._invalidate_access_token()
payload = self._post(endpoint, body, self._ensure_access_token(force=True))
self._validate_payload(payload)
if cache_key and cache_ttl > 0:
with self._cache_lock:
self._cache[cache_key] = {
"created_at": time.time(),
"payload": copy.deepcopy(payload),
}
return payload
def _ensure_access_token(self, force: bool = False) -> str:
with self._token_lock:
now = datetime.now().astimezone().replace(tzinfo=None)
token_valid = bool(self._access_token) and (
self._access_expires_at is None
or self._access_expires_at > now + timedelta(minutes=2)
)
if token_valid and not force:
return self._access_token
if not self._refresh_token:
if self._access_token:
return self._access_token
raise AdapterError("iFinD Refresh Token 尚未配置。")
payload = self._post(self.AUTH_ENDPOINT, {}, "", self._refresh_token)
self._validate_payload(payload)
data = payload.get("data") or {}
token = str(data.get("access_token") or "").strip()
if not token:
raise AdapterError("iFinD 未返回 Access Token。")
expires_at = self._parse_datetime(data.get("expired_time"))
self._access_token = token
self._access_expires_at = expires_at
return token
def _post(
self,
endpoint: str,
body: dict[str, Any],
access_token: str,
refresh_token: str = "",
) -> dict[str, Any]:
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": "XiaobaiDatahub/1.0",
"ifindlang": "cn",
}
if access_token:
headers["access_token"] = access_token
if refresh_token:
headers["refresh_token"] = refresh_token
request = urllib.request.Request(
f"{self.BASE_URL}/{endpoint}",
data=json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode("utf-8"),
headers=headers,
method="POST",
)
try:
with self._urlopen(request, timeout=self.timeout) as response:
payload = json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
detail = ""
try:
detail_payload = json.loads(exc.read().decode("utf-8", errors="replace"))
detail = str(detail_payload.get("errmsg") or detail_payload.get("message") or "")
except (json.JSONDecodeError, OSError):
pass
raise AdapterError(f"iFinD HTTP {exc.code}{f'{detail[:160]}' if detail else ''}") from exc
except (urllib.error.URLError, TimeoutError, OSError, json.JSONDecodeError) as exc:
raise AdapterError("iFinD 数据请求失败。") from exc
if not isinstance(payload, dict):
raise AdapterError("iFinD 返回格式不正确。")
return payload
def _cached(self, key: str, ttl: int) -> dict[str, Any] | None:
with self._cache_lock:
cached = self._cache.get(key)
if not cached:
return None
if time.time() - float(cached.get("created_at") or 0) > ttl:
self._cache.pop(key, None)
return None
return copy.deepcopy(cached["payload"])
def _invalidate_access_token(self) -> None:
with self._token_lock:
self._access_token = ""
self._access_expires_at = None
@classmethod
def _validate_payload(cls, payload: dict[str, Any]) -> None:
try:
error_code = int(payload.get("errorcode") or 0)
except (TypeError, ValueError):
error_code = -1
if error_code != 0:
message = str(payload.get("errmsg") or "未知错误")
raise AdapterError(f"iFinD 返回错误:{message[:200]}")
@classmethod
def _is_auth_error(cls, payload: dict[str, Any]) -> bool:
try:
error_code = int(payload.get("errorcode") or 0)
except (TypeError, ValueError):
error_code = 0
message = str(payload.get("errmsg") or "").casefold()
return error_code in cls.AUTH_ERROR_CODES or "token" in message or "鉴权" in message
@staticmethod
def _table_rows(payload: dict[str, Any]) -> list[dict[str, Any]]:
tables = payload.get("tables") or []
if isinstance(tables, dict):
tables = [tables]
rows: list[dict[str, Any]] = []
for block in tables if isinstance(tables, list) else []:
if not isinstance(block, dict):
continue
table = block.get("table") or {}
if not isinstance(table, dict):
continue
times = block.get("time") or []
codes = block.get("thscode") or block.get("thscodes") or []
if isinstance(codes, str):
codes = [codes]
lengths = [len(value) for value in table.values() if isinstance(value, list)]
row_count = max(lengths or [len(times) if isinstance(times, list) else 0, 1 if table else 0])
for index in range(row_count):
row: dict[str, Any] = {}
if isinstance(times, list) and index < len(times):
row["time"] = times[index]
if codes:
row["thscode"] = codes[index] if index < len(codes) else codes[0]
for field, values in table.items():
if isinstance(values, list):
row[field] = values[index] if index < len(values) else None
elif index == 0:
row[field] = values
rows.append(row)
return rows
@staticmethod
def _codes(codes: str | list[str]) -> str:
if isinstance(codes, list):
values = [str(code or "").strip().upper() for code in codes]
else:
values = [part.strip().upper() for part in str(codes or "").split(",")]
values = [value for value in values if value]
if not values:
raise AdapterError("iFinD 证券代码不能为空。")
if len(values) > 100:
raise AdapterError("iFinD 单次证券代码过多。")
return ",".join(values)
@staticmethod
def _display_date(value: str) -> str:
compact = str(value or "").replace("-", "")
if len(compact) != 8 or not compact.isdigit():
raise AdapterError("iFinD 日期格式不正确。")
return f"{compact[:4]}-{compact[4:6]}-{compact[6:]}"
@staticmethod
def _parse_datetime(value: Any) -> datetime | None:
text = str(value or "").strip()
if not text:
return None
try:
return datetime.fromisoformat(text)
except ValueError:
return None
def _indicators(value: Any) -> list[str]:
if isinstance(value, list):
return [str(item).strip() for item in value if str(item).strip()]
return [part.strip() for part in str(value or "").split(",") if part.strip()]
ADAPTER = IfindAdapter()
+217 -2
View File
@@ -1,3 +1,218 @@
from datahub.adapters.base import ReservedAdapter from __future__ import annotations
ADAPTER = ReservedAdapter("tencent") import time
import urllib.request
from datetime import datetime
from typing import Any
from datahub.adapters.base import AdapterError, MarketAdapter
from datahub.numbers import finite_number, round4
TENCENT_INDEX_URL = "https://qt.gtimg.cn/q=sh000001,sz399001,sz399006"
TENCENT_QUOTE_URL = "https://qt.gtimg.cn/q="
TENCENT_QUOTE_BATCH = 80
BROWSER_UA = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
)
class TencentAdapter(MarketAdapter):
name = "tencent"
def __init__(self, timeout: int = 8) -> None:
self.timeout = timeout
def probe(self) -> dict[str, Any]:
started = time.perf_counter()
try:
rows = self.fetch_indices()
state = "ok" if len(rows) == 3 else "empty"
except AdapterError as exc:
return {
"provider": self.name,
"configured": True,
"state": "error",
"message": str(exc),
"latency_ms": round((time.perf_counter() - started) * 1000),
}
return {
"provider": self.name,
"configured": True,
"state": state,
"latency_ms": round((time.perf_counter() - started) * 1000),
}
def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]:
if dataset in {"indexes_quotes", "index_quotes"}:
return self.fetch_indices()
if dataset in {"quotes", "quotes_latest"}:
codes = params.get("codes") or []
if isinstance(codes, str):
codes = [item.strip() for item in codes.split(",") if item.strip()]
return self.fetch_quotes(list(codes))
if dataset in {"quotes_market", "market_quotes"}:
codes = params.get("codes") or []
if isinstance(codes, str):
codes = [item.strip() for item in codes.split(",") if item.strip()]
return self.fetch_quotes(list(codes))
raise AdapterError(f"{self.name} unsupported dataset: {dataset}")
def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
return list(rows)
def fetch_indices(self) -> list[dict[str, Any]]:
request = urllib.request.Request(
TENCENT_INDEX_URL,
headers={"User-Agent": BROWSER_UA, "Referer": "https://gu.qq.com/"},
method="GET",
)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
raw = response.read().decode("gb18030", errors="ignore")
except Exception as exc:
raise AdapterError(f"tencent request failed: {exc}") from exc
result = []
for line in raw.splitlines():
if '="' not in line:
continue
fields = line.split('="', 1)[1].rsplit('";', 1)[0].split("~")
if len(fields) < 38:
continue
code = fields[2]
if code not in {"000001", "399001", "399006"}:
continue
try:
quote_time = datetime.strptime(fields[30], "%Y%m%d%H%M%S").astimezone()
except ValueError as exc:
raise AdapterError(f"Tencent invalid quote time for {code}") from exc
ts_code = "000001.SH" if code == "000001" else f"{code}.SZ"
result.append(
{
"ts_code": ts_code,
"code": code,
"name": fields[1] or code,
"price": round4(finite_number(fields[3])),
"pct_chg": round4(finite_number(fields[32])),
"change_amount": round4(finite_number(fields[31])),
"open": round4(finite_number(fields[5])),
"high": round4(finite_number(fields[33])),
"low": round4(finite_number(fields[34])),
"previous_close": round4(finite_number(fields[4])),
"amount": round4(finite_number(fields[37]) * 10000),
"quote_time_epoch": int(quote_time.timestamp()),
"quote_time": quote_time.isoformat(timespec="seconds"),
"source": "tencent_qt",
}
)
if len(result) != 3:
raise AdapterError(f"Tencent returned {len(result)}/3 indices")
return result
def fetch_quotes(self, codes: list[str]) -> list[dict[str, Any]]:
symbols: list[str] = []
seen: set[str] = set()
for raw in codes:
symbol = _tencent_symbol(str(raw or ""))
if not symbol or symbol in seen:
continue
seen.add(symbol)
symbols.append(symbol)
if not symbols:
return []
result: list[dict[str, Any]] = []
errors: list[str] = []
for index in range(0, len(symbols), TENCENT_QUOTE_BATCH):
batch = symbols[index:index + TENCENT_QUOTE_BATCH]
try:
raw = self._get_text(f"{TENCENT_QUOTE_URL}{','.join(batch)}")
except AdapterError as exc:
errors.append(str(exc))
continue
for line in raw.splitlines():
quote = _parse_tencent_stock_quote(line)
if quote:
result.append(quote)
if not result:
detail = f"{'; '.join(errors[:3])}" if errors else ""
raise AdapterError(f"Tencent quotes empty{detail}")
return result
def _get_text(self, url: str) -> str:
request = urllib.request.Request(
url,
headers={"User-Agent": BROWSER_UA, "Referer": "https://gu.qq.com/"},
method="GET",
)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
return response.read().decode("gb18030", errors="ignore")
except Exception as exc:
raise AdapterError(f"tencent request failed: {exc}") from exc
def _tencent_symbol(code: str) -> str:
raw = str(code or "").strip().upper()
if not raw:
return ""
symbol = raw.split(".")[0]
if not symbol.isdigit() or len(symbol) != 6:
return ""
if raw.endswith(".SH") or symbol.startswith(("5", "6", "9")):
return f"sh{symbol}"
if raw.endswith(".BJ") or symbol.startswith(("4", "8")):
return f"bj{symbol}"
return f"sz{symbol}"
def _parse_tencent_stock_quote(line: str) -> dict[str, Any] | None:
if '="' not in line:
return None
prefix, payload = line.split('="', 1)
fields = payload.rsplit('";', 1)[0].split("~")
if len(fields) < 38:
return None
symbol = str(fields[2] or "")
if not symbol.isdigit() or len(symbol) != 6:
return None
close = round4(finite_number(fields[3]))
previous = round4(finite_number(fields[4]))
if not close or not previous or close <= 0 or previous <= 0:
return None
marker = prefix.lower()
if "sh" in marker:
ts_code = f"{symbol}.SH"
elif "bj" in marker:
ts_code = f"{symbol}.BJ"
else:
ts_code = f"{symbol}.SZ"
quote_stamp = ""
quote_date = ""
epoch = 0
try:
parsed = datetime.strptime(fields[30], "%Y%m%d%H%M%S")
quote_date = parsed.strftime("%Y%m%d")
epoch = int(parsed.timestamp())
quote_stamp = parsed.astimezone().isoformat(timespec="seconds")
except ValueError:
pass
return {
"ts_code": ts_code,
"name": fields[1] or symbol,
"price": close,
"close": close,
"pct_chg": round4(finite_number(fields[32])),
"change_amount": round4(finite_number(fields[31])),
"open": round4(finite_number(fields[5])),
"high": round4(finite_number(fields[33])),
"low": round4(finite_number(fields[34])),
"pre_close": previous,
"previous_close": previous,
"volume": round4(finite_number(fields[6]) * 100),
"vol": round4(finite_number(fields[6]) * 100),
"amount": round4(finite_number(fields[37]) * 10000),
"quote_date": quote_date,
"quote_time_epoch": epoch,
"quote_time": quote_stamp,
"source": "tencent_qt",
}
+119 -11
View File
@@ -11,8 +11,12 @@ from datahub.normalize import (
normalize_auction, normalize_auction,
normalize_calendar, normalize_calendar,
normalize_daily, normalize_daily,
normalize_dragon_tiger,
normalize_index_daily, normalize_index_daily,
normalize_limit_event,
normalize_moneyflow, normalize_moneyflow,
normalize_popularity,
normalize_sector_daily,
normalize_stock, normalize_stock,
normalize_valuation, normalize_valuation,
) )
@@ -31,6 +35,29 @@ TUSHARE_FIELDS = {
"buy_lg_amount,sell_lg_amount,buy_elg_amount,sell_elg_amount,net_mf_amount" "buy_lg_amount,sell_lg_amount,buy_elg_amount,sell_elg_amount,net_mf_amount"
), ),
"stk_auction": "ts_code,trade_date,vol,price,amount,pre_close,turnover_rate,volume_ratio,float_share", "stk_auction": "ts_code,trade_date,vol,price,amount,pre_close,turnover_rate,volume_ratio,float_share",
"limit_list_d": (
"trade_date,ts_code,industry,name,close,pct_chg,amount,limit_amount,"
"float_mv,total_mv,turnover_ratio,fd_amount,first_time,last_time,"
"open_times,up_stat,limit_times,limit_type"
),
"ths_hot": "ts_code,ts_name,hot,rank,pct_change,current_price,concept,data_type,trade_date",
"dc_hot": "ts_code,ts_name,rank,pct_change,current_price,hot,concept,data_type,trade_date",
"hm_detail": "trade_date,ts_code,ts_name,buy_amount,sell_amount,net_amount,hm_name,hm_orgs,tag",
"hm_list": "name,desc,orgs",
"top_list": "trade_date,ts_code,name,pct_change,reason",
"top_inst": "trade_date,ts_code,exalter,buy,buy_rate,sell,sell_rate,net_buy,side,reason",
"ths_index": "ts_code,name,count,exchange,list_date,type",
"ths_daily": "ts_code,trade_date,open,high,low,close,pre_close,pct_change,vol,turnover_rate",
"dc_index": "ts_code,trade_date,name,open,high,low,close,pre_close,pct_change,vol,amount,turnover_rate",
"sw_daily": "ts_code,trade_date,name,open,high,low,close,pct_change,vol,amount",
"index_member_all": (
"l1_code,l1_name,l2_code,l2_name,l3_code,l3_name,"
"ts_code,name,in_date,out_date,is_new"
),
"stk_limit": "ts_code,trade_date,up_limit,down_limit",
"suspend_d": "ts_code,suspend_date,resume_date,ann_date,suspend_reason,reason_type",
"ths_member": "ts_code,con_code,con_name,in_date,out_date,is_new",
"stk_mins": "ts_code,trade_time,open,close,high,low,vol,amount",
} }
DATASET_API = { DATASET_API = {
@@ -42,12 +69,15 @@ DATASET_API = {
"index_daily": "index_daily", "index_daily": "index_daily",
"moneyflow": "moneyflow", "moneyflow": "moneyflow",
"auction": "stk_auction", "auction": "stk_auction",
"limit_events": "limit_list_d",
"popularity": "ths_hot",
"dragon_tiger": "hm_detail",
"sector_daily": "ths_daily",
} }
# Website actual index usage: market cards / 90-day charts (SH/SZ/CYB) plus
# screener 沪深300 benchmark (lookback up to 260 trading days).
WEBSITE_INDEX_CODES = ("000001.SH", "399001.SZ", "399006.SZ", "000300.SH") WEBSITE_INDEX_CODES = ("000001.SH", "399001.SZ", "399006.SZ", "000300.SH")
DEFAULT_INDEX_CODES = WEBSITE_INDEX_CODES DEFAULT_INDEX_CODES = WEBSITE_INDEX_CODES
LIMIT_TYPES = ("U", "D", "Z")
class TushareAdapter(MarketAdapter): class TushareAdapter(MarketAdapter):
@@ -85,6 +115,14 @@ class TushareAdapter(MarketAdapter):
} }
def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]: def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]:
if dataset == "limit_events":
return self.fetch_limit_events(str(params.get("trade_date") or ""))
if dataset == "popularity":
return self.fetch_popularity(str(params.get("trade_date") or ""))
if dataset == "dragon_tiger":
return self.fetch_dragon_tiger(str(params.get("trade_date") or ""))
if dataset == "sector_daily":
return self.fetch_sector_daily(str(params.get("trade_date") or ""))
api_name = DATASET_API.get(dataset, dataset) api_name = DATASET_API.get(dataset, dataset)
fields = TUSHARE_FIELDS.get(api_name, "") fields = TUSHARE_FIELDS.get(api_name, "")
query_params = dict(params) query_params = dict(params)
@@ -93,10 +131,67 @@ class TushareAdapter(MarketAdapter):
if api_name == "trade_cal" and "exchange" not in query_params: if api_name == "trade_cal" and "exchange" not in query_params:
query_params["exchange"] = "SSE" query_params["exchange"] = "SSE"
if api_name == "index_daily" and "ts_code" not in query_params: if api_name == "index_daily" and "ts_code" not in query_params:
# Caller typically loops codes; a missing code would pull nothing useful.
query_params.setdefault("ts_code", DEFAULT_INDEX_CODES[0]) query_params.setdefault("ts_code", DEFAULT_INDEX_CODES[0])
return self._query(api_name, query_params, fields) return self._query(api_name, query_params, fields)
def fetch_limit_events(self, trade_date: str) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for limit_type in LIMIT_TYPES:
part = self._query(
"limit_list_d",
{"trade_date": trade_date, "limit_type": limit_type},
TUSHARE_FIELDS["limit_list_d"],
)
for row in part:
row = dict(row)
row.setdefault("limit_type", limit_type)
rows.append(row)
return rows
def fetch_popularity(self, trade_date: str) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for api_name, source in (("ths_hot", "ths"), ("dc_hot", "dc")):
for row in self._query(api_name, {"trade_date": trade_date}, TUSHARE_FIELDS[api_name]):
item = dict(row)
item["source"] = source
item.setdefault("trade_date", trade_date)
rows.append(item)
return rows
def fetch_dragon_tiger(self, trade_date: str) -> list[dict[str, Any]]:
details = self._query("hm_detail", {"trade_date": trade_date}, TUSHARE_FIELDS["hm_detail"])
top_rows = self._query("top_list", {"trade_date": trade_date}, TUSHARE_FIELDS["top_list"])
context = {
str(row.get("ts_code") or ""): row
for row in top_rows
if str(row.get("ts_code") or "")
}
rows: list[dict[str, Any]] = []
for row in details:
item = dict(row)
stock = context.get(str(item.get("ts_code") or ""), {})
if item.get("pct_change") is None and stock.get("pct_change") is not None:
item["pct_change"] = stock.get("pct_change")
if not item.get("reason") and stock.get("reason"):
item["reason"] = stock.get("reason")
if not item.get("ts_name") and stock.get("name"):
item["ts_name"] = stock.get("name")
rows.append(item)
return rows
def fetch_sector_daily(self, trade_date: str) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for api_name, family in (("ths_daily", "ths"), ("dc_index", "dc"), ("sw_daily", "sw")):
try:
part = self._query(api_name, {"trade_date": trade_date}, TUSHARE_FIELDS[api_name])
except AdapterError:
part = []
for row in part:
item = dict(row)
item["family"] = family
rows.append(item)
return rows
def fetch_index_daily(self, trade_date: str, codes: tuple[str, ...] = DEFAULT_INDEX_CODES) -> list[dict[str, Any]]: def fetch_index_daily(self, trade_date: str, codes: tuple[str, ...] = DEFAULT_INDEX_CODES) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = [] rows: list[dict[str, Any]] = []
for ts_code in codes: for ts_code in codes:
@@ -104,6 +199,17 @@ class TushareAdapter(MarketAdapter):
return rows return rows
def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]: def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
if dataset in {"limit_events", "limit_list_d"}:
return [normalize_limit_event(row) for row in rows]
if dataset == "popularity":
return [normalize_popularity(row, source=str(row.get("source") or "")) for row in rows]
if dataset == "dragon_tiger":
return [normalize_dragon_tiger(row) for row in rows]
if dataset == "sector_daily":
return [
normalize_sector_daily(row, family=str(row.get("family") or "ths"))
for row in rows
]
mapping = { mapping = {
"calendar": normalize_calendar, "calendar": normalize_calendar,
"trade_cal": normalize_calendar, "trade_cal": normalize_calendar,
@@ -148,12 +254,14 @@ class TushareAdapter(MarketAdapter):
try: try:
with urllib.request.urlopen(request, timeout=self.timeout) as response: with urllib.request.urlopen(request, timeout=self.timeout) as response:
result = json.loads(response.read().decode("utf-8")) result = json.loads(response.read().decode("utf-8"))
except json.JSONDecodeError: except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
raise AdapterError("Tushare returned invalid json") from None raise AdapterError(f"Tushare 请求失败: {exc}") from exc
except (urllib.error.URLError, TimeoutError) as exc: if result.get("code") not in (0, "0", None):
raise AdapterError(f"Tushare request failed: {exc}") from exc raise AdapterError(str(result.get("msg") or f"Tushare error {result.get('code')}"))
if result.get("code") != 0:
raise AdapterError(result.get("msg") or "Tushare returned an unknown error")
data = result.get("data") or {} data = result.get("data") or {}
columns = data.get("fields") or [] items = data.get("items") or []
return [dict(zip(columns, item)) for item in data.get("items") or []] fields_list = data.get("fields") or (fields.split(",") if fields else [])
return [dict(zip(fields_list, item)) for item in items]
def query_raw(self, api_name: str, params: dict[str, Any], fields: str = "") -> list[dict[str, Any]]:
return self._query(api_name, params, fields or TUSHARE_FIELDS.get(api_name, ""))
+25 -3
View File
@@ -13,11 +13,12 @@ from datahub.timeutil import isoformat, now_shanghai, session_phase, yyyymmdd
class AdminAPI: class AdminAPI:
def __init__(self, db: HubDB, pipeline: Pipeline, scheduler: Scheduler, auth: AuthService) -> None: def __init__(self, db: HubDB, pipeline: Pipeline, scheduler: Scheduler, auth: AuthService, ifind: Any = None) -> None:
self.db = db self.db = db
self.pipeline = pipeline self.pipeline = pipeline
self.scheduler = scheduler self.scheduler = scheduler
self.auth = auth self.auth = auth
self.ifind = ifind
def overview(self) -> dict[str, Any]: def overview(self) -> dict[str, Any]:
today = yyyymmdd(now_shanghai()) today = yyyymmdd(now_shanghai())
@@ -57,10 +58,26 @@ class AdminAPI:
} }
] ]
for name, adapter in RESERVED.items(): for name, adapter in RESERVED.items():
if name == "ifind":
live = self.ifind or adapter
cred = self.auth.credential_status("ifind_refresh_token") or {
"configured": bool(getattr(live, "configured", False)),
"last4": "",
"updated_at": "",
}
items.append(
{
"provider": name,
"role": "licensed",
"health": live.probe(),
"credential": cred,
}
)
continue
items.append( items.append(
{ {
"provider": name, "provider": name,
"role": "reserved", "role": "reserved" if name in {"ths", "xgb", "akshare"} else "free",
"health": adapter.probe(), "health": adapter.probe(),
"credential": {"configured": False, "last4": "", "updated_at": ""}, "credential": {"configured": False, "last4": "", "updated_at": ""},
} }
@@ -78,6 +95,11 @@ class AdminAPI:
def probe(self, provider: str) -> dict[str, Any]: def probe(self, provider: str) -> dict[str, Any]:
if provider == "tushare": if provider == "tushare":
return self.pipeline.adapter.probe() return self.pipeline.adapter.probe()
if provider == "ifind":
adapter = self.ifind or RESERVED.get("ifind")
if adapter is None:
raise ApiError("INVALID_ARGUMENT", "unknown provider: ifind")
return adapter.probe()
adapter = RESERVED.get(provider) adapter = RESERVED.get(provider)
if adapter is None: if adapter is None:
raise ApiError("INVALID_ARGUMENT", f"unknown provider: {provider}") raise ApiError("INVALID_ARGUMENT", f"unknown provider: {provider}")
@@ -94,7 +116,7 @@ class AdminAPI:
{"id": "eod_retry", "at": "15:15-23:30", "title": "盘后未出数自动重试(每 30 分钟,成功即停)"}, {"id": "eod_retry", "at": "15:15-23:30", "title": "盘后未出数自动重试(每 30 分钟,成功即停)"},
{"id": "eod_revise", "at": "20:00-23:20", "title": "估值发布后复核(轻量比对,有修订才整组原子追补)"}, {"id": "eod_revise", "at": "20:00-23:20", "title": "估值发布后复核(轻量比对,有修订才整组原子追补)"},
{"id": "stocks_refresh", "at": stocks_times, "title": "股票主档刷新与正式发布(新上市/更名,无变化跳过)"}, {"id": "stocks_refresh", "at": stocks_times, "title": "股票主档刷新与正式发布(新上市/更名,无变化跳过)"},
{"id": "history_backfill", "at": "manual", "title": "回补历史日历与指数日 K"}, {"id": "history_backfill", "at": "manual", "title": "回补历史日历、个股日 K 与指数日 K"},
{"id": "cleanup", "at": "00:30", "title": "清理 staging / 日志"}, {"id": "cleanup", "at": "00:30", "title": "清理 staging / 日志"},
{"id": "backup", "at": "00:40", "title": "SQLite 备份"}, {"id": "backup", "at": "00:40", "title": "SQLite 备份"},
], ],
+4 -2
View File
@@ -15,10 +15,11 @@ from datahub.timeutil import yyyymmdd
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="xiaobai-datahub CLI") parser = argparse.ArgumentParser(description="xiaobai-datahub CLI")
sub = parser.add_subparsers(dest="command", required=True) sub = parser.add_subparsers(dest="command", required=True)
history = sub.add_parser("history-backfill", help="回补 2016 年起交易日历和网站所用指数日 K") history = sub.add_parser("history-backfill", help="回补交易日历、个股日 K(默认 250 日)和网站所用指数日 K")
history.add_argument("--calendar-start", default=None, help="日历起点,默认配置 calendar_start") history.add_argument("--calendar-start", default=None, help="日历起点,默认配置 calendar_start")
history.add_argument("--index-days", type=int, default=None, help="指数回补交易日数量,默认 260") history.add_argument("--index-days", type=int, default=None, help="指数回补交易日数量,默认 260")
history.add_argument("--force", action="store_true", help="覆盖已发布的指数日期") history.add_argument("--daily-days", type=int, default=None, help="个股日 K 回补交易日数量,默认 250")
history.add_argument("--force", action="store_true", help="覆盖已发布的个股日 K / 指数日期")
refresh = sub.add_parser("eod-refresh", help="对指定交易日补跑盘后正式数据(跳过已完整发布的一致性边界,仍走质量门禁)") refresh = sub.add_parser("eod-refresh", help="对指定交易日补跑盘后正式数据(跳过已完整发布的一致性边界,仍走质量门禁)")
refresh.add_argument("--trade-date", default=None, help="交易日 YYYYMMDD,默认今天") refresh.add_argument("--trade-date", default=None, help="交易日 YYYYMMDD,默认今天")
refresh.add_argument( refresh.add_argument(
@@ -46,6 +47,7 @@ def main(argv: list[str] | None = None) -> int:
result = hub.pipeline.backfill_history( result = hub.pipeline.backfill_history(
calendar_start=args.calendar_start, calendar_start=args.calendar_start,
index_days=args.index_days, index_days=args.index_days,
daily_days=args.daily_days,
force=args.force, force=args.force,
) )
json.dump(result, sys.stdout, ensure_ascii=False, indent=2, default=str) json.dump(result, sys.stdout, ensure_ascii=False, indent=2, default=str)
+180
View File
@@ -0,0 +1,180 @@
"""Extended EOD datasets beyond the first-batch A/B release groups.
These publish independently (soft): a failure here must not block daily/valuation
release. Scheduler runs them after the core EOD window.
"""
from __future__ import annotations
from typing import Any
# Independent soft datasets (not part of A/B atomic groups).
EXTENDED_SOFT_DATASETS = {
"limit_events",
"popularity",
"dragon_tiger",
"sector_daily",
}
EXTENDED_SCHEMA = """
CREATE TABLE IF NOT EXISTS eod_limit_events (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, limit_type TEXT NOT NULL,
name TEXT, industry TEXT, close REAL, pct_chg REAL, amount REAL,
limit_amount REAL, float_mv REAL, total_mv REAL, turnover_ratio REAL,
fd_amount REAL, first_time TEXT, last_time TEXT,
open_times INTEGER, up_stat TEXT, limit_times INTEGER,
batch_id TEXT NOT NULL,
PRIMARY KEY (ts_code, trade_date, limit_type, batch_id)
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS staging_limit_events (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, limit_type TEXT NOT NULL, batch_id TEXT NOT NULL,
name TEXT, industry TEXT, close REAL, pct_chg REAL, amount REAL,
limit_amount REAL, float_mv REAL, total_mv REAL, turnover_ratio REAL,
fd_amount REAL, first_time TEXT, last_time TEXT,
open_times INTEGER, up_stat TEXT, limit_times INTEGER,
PRIMARY KEY (batch_id, ts_code, trade_date, limit_type)
);
CREATE TABLE IF NOT EXISTS eod_popularity (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, source TEXT NOT NULL,
ts_name TEXT, rank INTEGER, pct_change REAL, current_price REAL,
hot REAL, concept TEXT, data_type TEXT,
batch_id TEXT NOT NULL,
PRIMARY KEY (ts_code, trade_date, source, batch_id)
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS staging_popularity (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, source TEXT NOT NULL, batch_id TEXT NOT NULL,
ts_name TEXT, rank INTEGER, pct_change REAL, current_price REAL,
hot REAL, concept TEXT, data_type TEXT,
PRIMARY KEY (batch_id, ts_code, trade_date, source)
);
CREATE TABLE IF NOT EXISTS eod_dragon_tiger (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, hm_name TEXT NOT NULL,
ts_name TEXT, buy_amount REAL, sell_amount REAL, net_amount REAL,
hm_orgs TEXT, tag TEXT, pct_change REAL, reason TEXT,
batch_id TEXT NOT NULL,
PRIMARY KEY (ts_code, trade_date, hm_name, batch_id)
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS staging_dragon_tiger (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, hm_name TEXT NOT NULL, batch_id TEXT NOT NULL,
ts_name TEXT, buy_amount REAL, sell_amount REAL, net_amount REAL,
hm_orgs TEXT, tag TEXT, pct_change REAL, reason TEXT,
PRIMARY KEY (batch_id, ts_code, trade_date, hm_name)
);
CREATE TABLE IF NOT EXISTS eod_sector_daily (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, family TEXT NOT NULL,
name TEXT, open REAL, high REAL, low REAL, close REAL, pre_close REAL,
pct_change REAL, vol REAL, turnover_rate REAL, amount REAL,
batch_id TEXT NOT NULL,
PRIMARY KEY (ts_code, trade_date, family, batch_id)
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS staging_sector_daily (
ts_code TEXT NOT NULL, trade_date TEXT NOT NULL, family TEXT NOT NULL, batch_id TEXT NOT NULL,
name TEXT, open REAL, high REAL, low REAL, close REAL, pre_close REAL,
pct_change REAL, vol REAL, turnover_rate REAL, amount REAL,
PRIMARY KEY (batch_id, ts_code, trade_date, family)
);
CREATE TABLE IF NOT EXISTS sector_master (
ts_code TEXT PRIMARY KEY,
name TEXT,
family TEXT NOT NULL,
exchange TEXT,
list_date TEXT,
member_count INTEGER,
type TEXT,
updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_eod_limit_date ON eod_limit_events(trade_date, batch_id);
CREATE INDEX IF NOT EXISTS idx_eod_pop_date ON eod_popularity(trade_date, batch_id);
CREATE INDEX IF NOT EXISTS idx_eod_lhb_date ON eod_dragon_tiger(trade_date, batch_id);
CREATE INDEX IF NOT EXISTS idx_eod_sector_date ON eod_sector_daily(trade_date, family, batch_id);
"""
EXTENDED_DATASET_TABLES = {
"limit_events": ("eod_limit_events", "staging_limit_events"),
"popularity": ("eod_popularity", "staging_popularity"),
"dragon_tiger": ("eod_dragon_tiger", "staging_dragon_tiger"),
"sector_daily": ("eod_sector_daily", "staging_sector_daily"),
}
EXTENDED_STAGING_INSERT: dict[str, tuple[str, Any]] = {
"limit_events": (
"INSERT INTO staging_limit_events("
"ts_code,trade_date,limit_type,batch_id,name,industry,close,pct_chg,amount,"
"limit_amount,float_mv,total_mv,turnover_ratio,fd_amount,first_time,last_time,"
"open_times,up_stat,limit_times) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
lambda r, b: (
r["ts_code"], r["trade_date"], r["limit_type"], b,
r.get("name"), r.get("industry"), r.get("close"), r.get("pct_chg"), r.get("amount"),
r.get("limit_amount"), r.get("float_mv"), r.get("total_mv"), r.get("turnover_ratio"),
r.get("fd_amount"), r.get("first_time"), r.get("last_time"),
r.get("open_times"), r.get("up_stat"), r.get("limit_times"),
),
),
"popularity": (
"INSERT INTO staging_popularity("
"ts_code,trade_date,source,batch_id,ts_name,rank,pct_change,current_price,hot,concept,data_type) "
"VALUES (?,?,?,?,?,?,?,?,?,?,?)",
lambda r, b: (
r["ts_code"], r["trade_date"], r["source"], b,
r.get("ts_name"), r.get("rank"), r.get("pct_change"), r.get("current_price"),
r.get("hot"), r.get("concept"), r.get("data_type"),
),
),
"dragon_tiger": (
"INSERT INTO staging_dragon_tiger("
"ts_code,trade_date,hm_name,batch_id,ts_name,buy_amount,sell_amount,net_amount,"
"hm_orgs,tag,pct_change,reason) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)",
lambda r, b: (
r["ts_code"], r["trade_date"], r["hm_name"], b,
r.get("ts_name"), r.get("buy_amount"), r.get("sell_amount"), r.get("net_amount"),
r.get("hm_orgs"), r.get("tag"), r.get("pct_change"), r.get("reason"),
),
),
"sector_daily": (
"INSERT INTO staging_sector_daily("
"ts_code,trade_date,family,batch_id,name,open,high,low,close,pre_close,"
"pct_change,vol,turnover_rate,amount) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
lambda r, b: (
r["ts_code"], r["trade_date"], r["family"], b,
r.get("name"), r.get("open"), r.get("high"), r.get("low"), r.get("close"),
r.get("pre_close"), r.get("pct_change"), r.get("vol"), r.get("turnover_rate"),
r.get("amount"),
),
),
}
EXTENDED_EOD_COPY = {
"limit_events": (
"INSERT OR REPLACE INTO eod_limit_events "
"SELECT ts_code,trade_date,limit_type,name,industry,close,pct_chg,amount,"
"limit_amount,float_mv,total_mv,turnover_ratio,fd_amount,first_time,last_time,"
"open_times,up_stat,limit_times,batch_id "
"FROM staging_limit_events WHERE batch_id = ?"
),
"popularity": (
"INSERT OR REPLACE INTO eod_popularity "
"SELECT ts_code,trade_date,source,ts_name,rank,pct_change,current_price,hot,concept,data_type,batch_id "
"FROM staging_popularity WHERE batch_id = ?"
),
"dragon_tiger": (
"INSERT OR REPLACE INTO eod_dragon_tiger "
"SELECT ts_code,trade_date,hm_name,ts_name,buy_amount,sell_amount,net_amount,"
"hm_orgs,tag,pct_change,reason,batch_id "
"FROM staging_dragon_tiger WHERE batch_id = ?"
),
"sector_daily": (
"INSERT OR REPLACE INTO eod_sector_daily "
"SELECT ts_code,trade_date,family,name,open,high,low,close,pre_close,"
"pct_change,vol,turnover_rate,amount,batch_id "
"FROM staging_sector_daily WHERE batch_id = ?"
),
}
+5 -1
View File
@@ -7,9 +7,10 @@ from contextlib import contextmanager
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from datahub.datasets_ext import EXTENDED_DATASET_TABLES, EXTENDED_SCHEMA
from datahub.timeutil import isoformat from datahub.timeutil import isoformat
SCHEMA = """ _BASE_SCHEMA = """
CREATE TABLE IF NOT EXISTS schema_migrations ( CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY, version INTEGER PRIMARY KEY,
applied_at TEXT NOT NULL applied_at TEXT NOT NULL
@@ -295,6 +296,8 @@ CREATE INDEX IF NOT EXISTS idx_eod_bars_date ON eod_bars(trade_date, batch_id);
CREATE INDEX IF NOT EXISTS idx_calendar_open ON trade_calendar(is_open, cal_date); CREATE INDEX IF NOT EXISTS idx_calendar_open ON trade_calendar(is_open, cal_date);
""" """
SCHEMA = _BASE_SCHEMA + EXTENDED_SCHEMA
DATASET_TABLES = { DATASET_TABLES = {
"daily": ("eod_bars", "staging_bars"), "daily": ("eod_bars", "staging_bars"),
"valuation": ("eod_valuation", "staging_valuation"), "valuation": ("eod_valuation", "staging_valuation"),
@@ -302,6 +305,7 @@ DATASET_TABLES = {
"auction": ("eod_auction", "staging_auction"), "auction": ("eod_auction", "staging_auction"),
"index_daily": ("eod_index_bars", "staging_index_bars"), "index_daily": ("eod_index_bars", "staging_index_bars"),
"stocks": ("eod_stocks", "staging_stocks"), "stocks": ("eod_stocks", "staging_stocks"),
**EXTENDED_DATASET_TABLES,
} }
+14 -4
View File
@@ -42,7 +42,7 @@ class HubRequestHandler(BaseHTTPRequestHandler):
self._json({"status": "ok"}, HTTPStatus.OK) self._json({"status": "ok"}, HTTPStatus.OK)
return return
if path.startswith("/v1/"): if path.startswith("/v1/"):
self._v1(path, parsed.query) self._v1(path, parsed.query, method)
return return
if path.startswith("/admin/api/"): if path.startswith("/admin/api/"):
self._admin_api(method, path) self._admin_api(method, path)
@@ -66,11 +66,21 @@ class HubRequestHandler(BaseHTTPRequestHandler):
LOGGER.exception("internal error") LOGGER.exception("internal error")
self._json({"error": {"code": "INTERNAL", "message": "internal error"}}, HTTPStatus.INTERNAL_SERVER_ERROR) self._json({"error": {"code": "INTERNAL", "message": "internal error"}}, HTTPStatus.INTERNAL_SERVER_ERROR)
def _v1(self, path: str, query: str) -> None: def _v1(self, path: str, query: str, method: str = "GET") -> None:
token = self.headers.get("X-Datahub-Token", "") token = self.headers.get("X-Datahub-Token", "")
if not self.hub.auth.check_api_token(token): if not self.hub.auth.check_api_token(token):
self.hub.pipeline.audit("anonymous", "unauthorized", path, "") self.hub.pipeline.audit("anonymous", "unauthorized", path, "")
raise ApiError("UNAUTHORIZED", "missing or invalid X-Datahub-Token") raise ApiError("UNAUTHORIZED", "missing or invalid X-Datahub-Token")
if path == "/v1/query" and method == "POST":
body = self._read_json(max_bytes=1_000_000)
payload = self.hub.api.query_api(body)
self._json(payload, HTTPStatus.OK)
return
if path == "/v1/credentials/ifind" and method == "POST":
body = self._read_json()
payload = self.hub.put_ifind_credentials(body)
self._json(payload, HTTPStatus.OK)
return
payload = self.hub.api.handle(path, parse_query(query)) payload = self.hub.api.handle(path, parse_query(query))
self._json(payload, HTTPStatus.OK) self._json(payload, HTTPStatus.OK)
@@ -184,11 +194,11 @@ class HubRequestHandler(BaseHTTPRequestHandler):
self.end_headers() self.end_headers()
self.wfile.write(content) self.wfile.write(content)
def _read_json(self, allow_empty: bool = False) -> dict[str, Any]: def _read_json(self, allow_empty: bool = False, max_bytes: int = 65536) -> dict[str, Any]:
length = int(self.headers.get("Content-Length", "0") or 0) length = int(self.headers.get("Content-Length", "0") or 0)
if length == 0 and allow_empty: if length == 0 and allow_empty:
return {} return {}
if length <= 0 or length > 65536: if length <= 0 or length > max_bytes:
raise ValueError("请求内容为空或过大") raise ValueError("请求内容为空或过大")
raw = self.rfile.read(length) raw = self.rfile.read(length)
try: try:
+24 -2
View File
@@ -1,7 +1,9 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path from pathlib import Path
from typing import Any
from datahub.adapters.ifind import IfindAdapter
from datahub.adapters.tushare import TushareAdapter from datahub.adapters.tushare import TushareAdapter
from datahub.admin_api import AdminAPI from datahub.admin_api import AdminAPI
from datahub.auth import AuthService from datahub.auth import AuthService
@@ -28,7 +30,16 @@ class Hub:
if settings.tushare_token: if settings.tushare_token:
self.auth.store_credential("tushare_token", settings.tushare_token) self.auth.store_credential("tushare_token", settings.tushare_token)
token = settings.tushare_token token = settings.tushare_token
refresh = settings.ifind_refresh_token or self.auth.load_credential("ifind_refresh_token") or ""
access = settings.ifind_access_token or self.auth.load_credential("ifind_access_token") or ""
if settings.ifind_refresh_token:
self.auth.store_credential("ifind_refresh_token", settings.ifind_refresh_token)
refresh = settings.ifind_refresh_token
if settings.ifind_access_token:
self.auth.store_credential("ifind_access_token", settings.ifind_access_token)
access = settings.ifind_access_token
self.adapter = adapter or TushareAdapter(token) self.adapter = adapter or TushareAdapter(token)
self.ifind = IfindAdapter(refresh, access)
self.pipeline = Pipeline( self.pipeline = Pipeline(
self.db, self.db,
self.adapter, self.adapter,
@@ -38,10 +49,21 @@ class Hub:
) )
self.lkg = LastKnownGood(self.db) self.lkg = LastKnownGood(self.db)
self.scheduler = Scheduler(self.db, self.pipeline) self.scheduler = Scheduler(self.db, self.pipeline)
self.api = V1API(self.db, self.pipeline, settings) self.api = V1API(self.db, self.pipeline, settings, ifind=self.ifind)
self.admin = AdminAPI(self.db, self.pipeline, self.scheduler, self.auth) self.admin = AdminAPI(self.db, self.pipeline, self.scheduler, self.auth, ifind=self.ifind)
self.static_dir = Path(__file__).resolve().parents[1] / "admin" self.static_dir = Path(__file__).resolve().parents[1] / "admin"
def put_ifind_credentials(self, body: dict[str, Any] | None) -> dict[str, Any]:
from datahub.serving import envelope
payload = dict(body or {})
refresh = str(payload.get("ifind_refresh_token") or "").strip()
access = str(payload.get("ifind_access_token") or "").strip()
self.auth.store_credential("ifind_refresh_token", refresh)
self.auth.store_credential("ifind_access_token", access)
self.ifind.set_credentials(refresh, access)
return envelope(self.ifind.status(), {"source": "ifind"})
def start(self) -> None: def start(self) -> None:
if self.settings.scheduler_enabled: if self.settings.scheduler_enabled:
self.scheduler.start() self.scheduler.start()
+1
View File
@@ -12,6 +12,7 @@ from datahub.timeutil import isoformat
_SECRET_KEYS = ( _SECRET_KEYS = (
"token", "password", "secret", "key", "authorization", "credential", "token", "password", "secret", "key", "authorization", "credential",
"tushare_token", "datahub_token", "encryption_key", "cookie", "tushare_token", "datahub_token", "encryption_key", "cookie",
"ifind", "refresh_token", "access_token",
) )
_SECRET_JSON = re.compile( _SECRET_JSON = re.compile(
r'(?i)("(?:' + "|".join(re.escape(key) for key in _SECRET_KEYS) + r')"\s*:\s*")([^"\\]*(?:\\.[^"\\]*)*)(")' r'(?i)("(?:' + "|".join(re.escape(key) for key in _SECRET_KEYS) + r')"\s*:\s*")([^"\\]*(?:\\.[^"\\]*)*)(")'
+94
View File
@@ -156,6 +156,95 @@ def normalize_stock(row: dict[str, Any]) -> dict[str, Any]:
} }
def normalize_limit_event(row: dict[str, Any]) -> dict[str, Any]:
"""limit_list_d. float_mv/total_mv/limit_amount are 万元 → yuan; amount/fd_amount already yuan."""
return {
"ts_code": _code(row.get("ts_code")),
"trade_date": _date(row.get("trade_date")),
"limit_type": str(row.get("limit_type") or "").strip().upper() or "U",
"name": str(row.get("name") or "").strip() or None,
"industry": str(row.get("industry") or "").strip() or None,
"close": round4(finite_number(row.get("close"))),
"pct_chg": round4(finite_number(row.get("pct_chg"))),
"amount": round4(finite_number(row.get("amount"))),
"limit_amount": round4(_scale(row.get("limit_amount"), AMOUNT_WAN_YUAN)),
"float_mv": round4(_scale(row.get("float_mv"), AMOUNT_WAN_YUAN)),
"total_mv": round4(_scale(row.get("total_mv"), AMOUNT_WAN_YUAN)),
"turnover_ratio": round4(finite_number(row.get("turnover_ratio"))),
"fd_amount": round4(finite_number(row.get("fd_amount"))),
"first_time": str(row.get("first_time") or "").strip() or None,
"last_time": str(row.get("last_time") or "").strip() or None,
"open_times": _optional_int(row.get("open_times")),
"up_stat": str(row.get("up_stat") or "").strip() or None,
"limit_times": _optional_int(row.get("limit_times")),
}
def normalize_popularity(row: dict[str, Any], source: str = "") -> dict[str, Any]:
src = str(source or row.get("source") or "").strip().lower() or "ths"
return {
"ts_code": _code(row.get("ts_code")),
"trade_date": _date(row.get("trade_date")),
"source": src,
"ts_name": str(row.get("ts_name") or row.get("name") or "").strip() or None,
"rank": _optional_int(row.get("rank")),
"pct_change": round4(
finite_number(row.get("pct_change") if row.get("pct_change") is not None else row.get("pct_chg"))
),
"current_price": round4(finite_number(row.get("current_price") or row.get("price"))),
"hot": round4(finite_number(row.get("hot"))),
"concept": str(row.get("concept") or "").strip() or None,
"data_type": str(row.get("data_type") or "").strip() or None,
}
def normalize_dragon_tiger(row: dict[str, Any]) -> dict[str, Any]:
"""hm_detail amounts are 万元 → yuan."""
return {
"ts_code": _code(row.get("ts_code")),
"trade_date": _date(row.get("trade_date")),
"hm_name": str(row.get("hm_name") or "未命名游资").strip() or "未命名游资",
"ts_name": str(row.get("ts_name") or row.get("name") or "").strip() or None,
"buy_amount": round4(_scale(row.get("buy_amount"), AMOUNT_WAN_YUAN)),
"sell_amount": round4(_scale(row.get("sell_amount"), AMOUNT_WAN_YUAN)),
"net_amount": round4(_scale(row.get("net_amount"), AMOUNT_WAN_YUAN)),
"hm_orgs": str(row.get("hm_orgs") or "").strip() or None,
"tag": str(row.get("tag") or "").strip() or None,
"pct_change": round4(finite_number(row.get("pct_change"))),
"reason": str(row.get("reason") or "").strip() or None,
}
def normalize_sector_daily(row: dict[str, Any], family: str = "ths") -> dict[str, Any]:
fam = str(family or row.get("family") or "ths").strip().lower()
return {
"ts_code": _code(row.get("ts_code")),
"trade_date": _date(row.get("trade_date")),
"family": fam,
"name": str(row.get("name") or "").strip() or None,
"open": round4(finite_number(row.get("open"))),
"high": round4(finite_number(row.get("high"))),
"low": round4(finite_number(row.get("low"))),
"close": round4(finite_number(row.get("close"))),
"pre_close": round4(finite_number(row.get("pre_close"))),
"pct_change": round4(
finite_number(row.get("pct_change") if row.get("pct_change") is not None else row.get("pct_chg"))
),
"vol": round4(finite_number(row.get("vol"))),
"turnover_rate": round4(finite_number(row.get("turnover_rate"))),
"amount": round4(finite_number(row.get("amount"))),
}
def _optional_int(value: Any) -> int | None:
if value in (None, ""):
return None
try:
return int(float(value))
except (TypeError, ValueError):
return None
def apply_qfq(price: float | None, factor: float | None, latest_factor: float | None) -> float | None: def apply_qfq(price: float | None, factor: float | None, latest_factor: float | None) -> float | None:
if price is None: if price is None:
return None return None
@@ -184,6 +273,11 @@ NORMALIZERS = {
"calendar": normalize_calendar, "calendar": normalize_calendar,
"stock_basic": normalize_stock, "stock_basic": normalize_stock,
"stocks": normalize_stock, "stocks": normalize_stock,
"limit_events": normalize_limit_event,
"limit_list_d": normalize_limit_event,
"popularity": normalize_popularity,
"dragon_tiger": normalize_dragon_tiger,
"sector_daily": normalize_sector_daily,
} }
+190 -15
View File
@@ -9,6 +9,11 @@ from typing import Any
from datahub.adapters.base import AdapterError from datahub.adapters.base import AdapterError
from datahub.adapters.tushare import DEFAULT_INDEX_CODES, WEBSITE_INDEX_CODES, TushareAdapter from datahub.adapters.tushare import DEFAULT_INDEX_CODES, WEBSITE_INDEX_CODES, TushareAdapter
from datahub.datasets_ext import (
EXTENDED_EOD_COPY,
EXTENDED_SOFT_DATASETS,
EXTENDED_STAGING_INSERT,
)
from datahub.db import DATASET_TABLES, HubDB from datahub.db import DATASET_TABLES, HubDB
from datahub.governance.circuit import CircuitBreaker from datahub.governance.circuit import CircuitBreaker
from datahub.governance.ratelimit import TokenBucket from datahub.governance.ratelimit import TokenBucket
@@ -27,12 +32,16 @@ from datahub.timeutil import add_days, isoformat, now_shanghai, yyyymmdd
LOGGER = get_logger() LOGGER = get_logger()
HARD_DATASETS = {"daily", "valuation", "index_daily"} HARD_DATASETS = {"daily", "valuation", "index_daily"}
SOFT_DATASETS = {"moneyflow", "auction"} SOFT_DATASETS = {"moneyflow", "auction"} | EXTENDED_SOFT_DATASETS
OFFICIAL_DATASETS = HARD_DATASETS | SOFT_DATASETS OFFICIAL_DATASETS = HARD_DATASETS | {"moneyflow", "auction"} # A/B retry scope unchanged
STOCKS_DATASET = "stocks" STOCKS_DATASET = "stocks"
STOCK_SNAPSHOT_FIELDS = ("ts_code", "symbol", "name", "area", "industry", "market", "list_status", "list_date") STOCK_SNAPSHOT_FIELDS = ("ts_code", "symbol", "name", "area", "industry", "market", "list_status", "list_date")
EOD_A_DATASETS = ("daily", "valuation", "moneyflow", "auction") EOD_A_DATASETS = ("daily", "valuation", "moneyflow", "auction")
EOD_B_DATASETS = ("index_daily",) EOD_B_DATASETS = ("index_daily",)
EOD_C_DATASETS = ("limit_events",)
EOD_D_DATASETS = ("dragon_tiger",)
EOD_E_DATASETS = ("sector_daily",)
EOD_F_DATASETS = ("popularity",)
EMPTY_BATCH_ERROR = "empty official batch: 0 valid rows" EMPTY_BATCH_ERROR = "empty official batch: 0 valid rows"
STAGING_INSERT = { STAGING_INSERT = {
@@ -86,6 +95,7 @@ STAGING_INSERT = {
r.get("close"), r.get("pct_chg"), r.get("volume"), r.get("amount"), r.get("close"), r.get("pct_chg"), r.get("volume"), r.get("amount"),
), ),
), ),
**EXTENDED_STAGING_INSERT,
} }
EOD_COPY = { EOD_COPY = {
@@ -120,6 +130,7 @@ EOD_COPY = {
"SELECT ts_code,trade_date,open,high,low,close,pct_chg,volume,amount,batch_id " "SELECT ts_code,trade_date,open,high,low,close,pct_chg,volume,amount,batch_id "
"FROM staging_index_bars WHERE batch_id = ?" "FROM staging_index_bars WHERE batch_id = ?"
), ),
**EXTENDED_EOD_COPY,
} }
@@ -480,24 +491,96 @@ class Pipeline:
) )
return sorted(str(row["cal_date"]) for row in rows) return sorted(str(row["cal_date"]) for row in rows)
def backfill_daily_history(
self,
end_date: str | None = None,
trading_days: int | None = None,
force: bool = False,
) -> dict[str, Any]:
"""Incrementally publish official daily bars for the website K-line window.
Same-day EOD still uses the atomic A-group. History backfill publishes
daily (with adj_factor) first so K-line coverage cannot be blocked by
the looser moneyflow universe, then valuation and moneyflow/auction
best-effort. Complete daily dates are skipped unless ``force``.
"""
end = yyyymmdd(end_date or self.clock())
limit = int(trading_days or self.settings.daily_history_trading_days)
open_dates = self.open_trade_dates(end, limit)
if not open_dates:
return {
"start": None,
"end": end,
"requested_days": 0,
"published": [],
"skipped": [],
"failed": [{"error": "calendar has no open dates on or before end"}],
"ok": False,
}
start = open_dates[0]
published: list[dict[str, Any]] = []
skipped: list[str] = []
failed: list[dict[str, Any]] = []
for day in open_dates:
if not force and self.active_batch("daily", day):
skipped.append(day)
continue
try:
daily = self.run_dataset("daily", day)
datasets = {"daily": daily.get("state")}
try:
valuation = self.run_dataset("valuation", day)
datasets["valuation"] = valuation.get("state")
except Exception as exc:
datasets["valuation"] = f"failed:{exc}"[:180]
for name in ("moneyflow", "auction"):
try:
extra = self.run_dataset(name, day)
datasets[name] = extra.get("state")
except Exception as exc:
datasets[name] = f"failed:{exc}"[:180]
published.append({"trade_date": day, "datasets": datasets})
except Exception as exc:
failed.append({"trade_date": day, "error": str(exc)})
return {
"start": start,
"end": end,
"requested_days": len(open_dates),
"published": published,
"skipped": skipped,
"failed": failed,
"ok": not failed,
}
def backfill_history( def backfill_history(
self, self,
trade_date: str | None = None, trade_date: str | None = None,
calendar_start: str | None = None, calendar_start: str | None = None,
index_days: int | None = None, index_days: int | None = None,
daily_days: int | None = None,
codes: tuple[str, ...] | None = None, codes: tuple[str, ...] | None = None,
force: bool = False, force: bool = False,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Idempotent calendar + website-index history backfill.""" """Idempotent calendar + stock daily + website-index history backfill."""
day = yyyymmdd(trade_date or self.clock()) day = yyyymmdd(trade_date or self.clock())
calendar = self.ingest_reference(day, start=calendar_start) calendar = self.ingest_reference(day, start=calendar_start)
daily = self.backfill_daily_history(
end_date=day,
trading_days=daily_days,
force=force,
)
index = self.backfill_index_history( index = self.backfill_index_history(
end_date=day, end_date=day,
trading_days=index_days, trading_days=index_days,
codes=codes, codes=codes,
force=force, force=force,
) )
return {"calendar": calendar, "index_daily": index, "ok": bool(index.get("ok"))} return {
"calendar": calendar,
"daily": daily,
"index_daily": index,
"ok": bool(daily.get("ok")) and bool(index.get("ok")),
}
def backfill_index_history( def backfill_index_history(
self, self,
@@ -678,19 +761,85 @@ class Pipeline:
def run_eod_batch_b(self, trade_date: str, force: bool = False) -> dict[str, Any]: def run_eod_batch_b(self, trade_date: str, force: bool = False) -> dict[str, Any]:
return self.run_release_group(EOD_B_DATASETS, trade_date, force=force) return self.run_release_group(EOD_B_DATASETS, trade_date, force=force)
def run_extended_soft(self, datasets: tuple[str, ...], trade_date: str, force: bool = False) -> dict[str, Any]:
"""Publish extended soft datasets independently (not A/B atomic)."""
results: dict[str, Any] = {}
day = yyyymmdd(trade_date)
for dataset in datasets:
if not force and self.extended_publication_complete(dataset, day):
results[dataset] = {
"dataset": dataset,
"trade_date": day,
"state": "skipped",
"reason": "already_published",
}
continue
try:
rows = self._fetch_dataset(dataset, day)
if not rows and dataset in {"popularity", "dragon_tiger"}:
results[dataset] = {
"dataset": dataset,
"trade_date": day,
"state": "skipped",
"reason": "upstream_empty",
"rows": 0,
}
continue
results[dataset] = self.run_dataset(dataset, day, prepared_rows=rows)
except Exception as exc:
results[dataset] = {
"dataset": dataset,
"trade_date": day,
"state": "failed",
"error": str(exc),
}
LOGGER.exception("extended soft publish failed dataset=%s date=%s", dataset, day)
return results
def extended_publication_complete(self, dataset: str, trade_date: str) -> bool:
"""Do not mistake a partially published multi-source dataset for completion."""
day = yyyymmdd(trade_date)
batch_id = self.active_batch(dataset, day)
if not batch_id:
return False
if dataset != "sector_daily":
return True
rows = self.db.fetchall(
"SELECT DISTINCT family FROM eod_sector_daily "
"WHERE trade_date = ? AND batch_id = ?",
(day, batch_id),
)
families = {str(row.get("family") or "") for row in rows}
return {"ths", "dc", "sw"} <= families
def run_eod_batch_c(self, trade_date: str, force: bool = False) -> dict[str, Any]:
return self.run_extended_soft(EOD_C_DATASETS, trade_date, force=force)
def run_eod_batch_d(self, trade_date: str, force: bool = False) -> dict[str, Any]:
return self.run_extended_soft(EOD_D_DATASETS, trade_date, force=force)
def run_eod_batch_e(self, trade_date: str, force: bool = False) -> dict[str, Any]:
return self.run_extended_soft(EOD_E_DATASETS, trade_date, force=force)
def run_eod_batch_f(self, trade_date: str, force: bool = False) -> dict[str, Any]:
return self.run_extended_soft(EOD_F_DATASETS, trade_date, force=force)
def force_republish_boundary(self, dataset: str, trade_date: str) -> dict[str, Any]: def force_republish_boundary(self, dataset: str, trade_date: str) -> dict[str, Any]:
"""Force-republish the full A/B consistency boundary that owns ``dataset``. """Force-republish the full A/B consistency boundary that owns ``dataset``.
CLI ``eod-refresh --force`` and admin manual backfill must not publish a CLI ``eod-refresh --force`` and admin manual backfill must not publish a
single official member alone that would mix old and new batches inside single official member alone that would mix old and new batches inside
the same trade date. Naming any A-group member (or stocks) rebuilds the the same trade date. Naming any A-group member (or stocks) rebuilds the
whole A group; naming ``index_daily`` rebuilds B. whole A group; naming ``index_daily`` rebuilds B. Extended soft datasets
republish independently.
""" """
name = str(dataset or "").strip() name = str(dataset or "").strip()
if name in EOD_A_DATASETS or name == STOCKS_DATASET: if name in EOD_A_DATASETS or name == STOCKS_DATASET:
return self.run_eod_batch_a(trade_date, force=True) return self.run_eod_batch_a(trade_date, force=True)
if name in EOD_B_DATASETS: if name in EOD_B_DATASETS:
return self.run_eod_batch_b(trade_date, force=True) return self.run_eod_batch_b(trade_date, force=True)
if name in EXTENDED_SOFT_DATASETS:
return self.run_extended_soft((name,), trade_date, force=True)
raise ValueError(f"dataset is not part of an EOD release boundary: {dataset}") raise ValueError(f"dataset is not part of an EOD release boundary: {dataset}")
def published_official_rows(self, dataset: str, trade_date: str) -> list[dict[str, Any]]: def published_official_rows(self, dataset: str, trade_date: str) -> list[dict[str, Any]]:
@@ -742,10 +891,7 @@ class Pipeline:
"published_rows": len(published), "published_rows": len(published),
"upstream_rows": 0, "upstream_rows": 0,
} }
listed = self.db.fetchone( listed_n = self._listed_count(day)
"SELECT COUNT(*) AS n FROM stock_master WHERE list_status = 'L'",
)
listed_n = int((listed or {}).get("n") or 0)
floor = float(self.settings.quality.get("daily_row_ratio") or 0.98) floor = float(self.settings.quality.get("daily_row_ratio") or 0.98)
if listed_n and len(upstream) / listed_n < floor: if listed_n and len(upstream) / listed_n < floor:
return { return {
@@ -1253,16 +1399,44 @@ class Pipeline:
if isinstance(item, dict) and item.get("state") == "failed" if isinstance(item, dict) and item.get("state") == "failed"
] ]
def _listed_count(self, trade_date: str = "") -> int:
"""Count listed names that already existed on ``trade_date``.
Historical daily bars must not be judged against later IPOs, or a
correct past session fails the 0.98 row-ratio gate.
"""
day = yyyymmdd(trade_date) if trade_date else ""
if day:
listed = self.db.fetchone(
"""
SELECT COUNT(*) AS n FROM stock_master
WHERE list_status = 'L'
AND (list_date IS NULL OR TRIM(list_date) = '' OR list_date <= ?)
""",
(day,),
)
else:
listed = self.db.fetchone(
"SELECT COUNT(*) AS n FROM stock_master WHERE list_status = 'L'"
)
return int((listed or {}).get("n") or 0)
def validate(self, dataset: str, batch_id: str, trade_date: str, rows: list[dict[str, Any]]) -> dict[str, Any]: def validate(self, dataset: str, batch_id: str, trade_date: str, rows: list[dict[str, Any]]) -> dict[str, Any]:
quality = self.settings.quality quality = self.settings.quality
errors: list[str] = [] errors: list[str] = []
warnings: list[str] = [] warnings: list[str] = []
listed = self.db.fetchone( listed_n = self._listed_count(trade_date)
"SELECT COUNT(*) AS n FROM stock_master WHERE list_status = 'L'",
)
listed_n = int((listed or {}).get("n") or 0)
row_n = len(rows) row_n = len(rows)
keys = [(row.get("ts_code"), row.get("trade_date")) for row in rows] if dataset == "limit_events":
keys = [(row.get("ts_code"), row.get("trade_date"), row.get("limit_type")) for row in rows]
elif dataset == "popularity":
keys = [(row.get("ts_code"), row.get("trade_date"), row.get("source")) for row in rows]
elif dataset == "dragon_tiger":
keys = [(row.get("ts_code"), row.get("trade_date"), row.get("hm_name")) for row in rows]
elif dataset == "sector_daily":
keys = [(row.get("ts_code"), row.get("trade_date"), row.get("family")) for row in rows]
else:
keys = [(row.get("ts_code"), row.get("trade_date")) for row in rows]
dup = row_n - len(set(keys)) dup = row_n - len(set(keys))
if dup: if dup:
errors.append(f"duplicate keys: {dup}") errors.append(f"duplicate keys: {dup}")
@@ -1283,7 +1457,8 @@ class Pipeline:
errors.append(EMPTY_BATCH_ERROR) errors.append(EMPTY_BATCH_ERROR)
field_report = self._field_gate(dataset, trade_date, rows, errors) field_report = self._field_gate(dataset, trade_date, rows, errors)
if dataset in SOFT_DATASETS: if dataset in SOFT_DATASETS:
hard_fail = bool(dup or bad_date or empty) allow_empty = dataset in {"popularity", "dragon_tiger", "moneyflow", "auction"}
hard_fail = bool(dup or bad_date or (empty and not allow_empty))
else: else:
hard_fail = bool(errors) and (dataset in HARD_DATASETS or dataset == STOCKS_DATASET) hard_fail = bool(errors) and (dataset in HARD_DATASETS or dataset == STOCKS_DATASET)
report = { report = {
+417
View File
@@ -0,0 +1,417 @@
"""Provisional (盘中观察) serving: quotes, index quotes, intraday points.
Free sources only. Never writes official eod_* tables. Uses rt_cache + LKG.
"""
from __future__ import annotations
import hashlib
import json
import time
from datetime import datetime
from typing import Any
from datahub.adapters.base import AdapterError
from datahub.adapters.eastmoney import EastmoneyAdapter
from datahub.adapters.tencent import TencentAdapter
from datahub.codes import resolve_code
from datahub.db import HubDB
from datahub.governance.lkg import LastKnownGood
from datahub.timeutil import isoformat, now_shanghai, yyyymmdd
QUOTE_TTL = 60
INDEX_TTL = 60
INTRADAY_TTL = 20
QUOTE_BATCH = 60
class RealtimeApiError(RuntimeError):
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
def _envelope(data: Any, meta: dict[str, Any]) -> dict[str, Any]:
from datahub import SCHEMA_VERSION
return {"schema_version": SCHEMA_VERSION, "data": data, "meta": meta}
def fetch_index_quotes(db: HubDB) -> dict[str, Any]:
cache_key = "indexes:quotes"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
eastmoney = EastmoneyAdapter()
try:
rows = eastmoney.fetch_indices()
source = "eastmoney:ulist"
except Exception:
rows = TencentAdapter().fetch_indices()
source = "tencent:qt"
if len(rows) < 3:
raise RealtimeApiError("SOURCE_UNAVAILABLE", "index quotes incomplete")
payload = _envelope(
rows,
{
"tier": "provisional",
"trade_date": yyyymmdd(now_shanghai()),
"source": source,
"stale": False,
"staleness_seconds": 0,
"published_at": isoformat(now_shanghai()),
},
)
_write_cache(db, cache_key, payload, INDEX_TTL, source)
return payload
def fetch_market_quotes(db: HubDB) -> dict[str, Any]:
cache_key = "quotes:market"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
errors: list[str] = []
rows: list[dict[str, Any]] = []
source = ""
try:
rows = EastmoneyAdapter().fetch_market_quotes()
source = "eastmoney:clist"
except Exception as exc:
errors.append(f"eastmoney:{exc}")
try:
listed = _listed_ts_codes(db)
if not listed:
raise AdapterError("no local stock master for tencent market snapshot")
rows = TencentAdapter().fetch_quotes(listed)
if len(rows) < 200:
raise AdapterError(f"Tencent market snapshot too small: {len(rows)}")
source = "tencent:qt"
except Exception as backup_exc:
errors.append(f"tencent:{backup_exc}")
recovered = _load_quotes_lkg(db, cache_key)
if recovered is not None:
return recovered
raise RealtimeApiError(
"SOURCE_UNAVAILABLE",
"market quotes unavailable: " + "".join(errors),
) from backup_exc
payload = _quote_payload(rows, source, scope="market")
_write_cache(db, cache_key, payload, QUOTE_TTL, source)
return payload
def fetch_quotes(db: HubDB, codes: list[str]) -> dict[str, Any]:
if not codes:
return fetch_market_quotes(db)
resolved: list[str] = []
seen: set[str] = set()
for code in codes:
item = resolve_code(db, code) or _guess_ts_code(code)
if item and item not in seen:
seen.add(item)
resolved.append(item)
if not resolved:
raise RealtimeApiError("INVALID_ARGUMENT", "no resolvable codes")
digest = hashlib.sha1(",".join(sorted(resolved)).encode("utf-8")).hexdigest()
cache_key = f"quotes:{digest}:{len(resolved)}"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
minimum = max(1, int(len(resolved) * 0.5))
errors: list[str] = []
rows: list[dict[str, Any]] = []
source = ""
try:
rows = _eastmoney_named_quotes(resolved)
if len(rows) < minimum:
raise AdapterError(f"Eastmoney named quotes too small: {len(rows)}/{len(resolved)}")
source = "eastmoney:ulist"
except Exception as exc:
errors.append(f"eastmoney:{exc}")
try:
rows = TencentAdapter().fetch_quotes(resolved)
if len(rows) < minimum:
raise AdapterError(f"Tencent named quotes too small: {len(rows)}/{len(resolved)}")
source = "tencent:qt"
except Exception as backup_exc:
errors.append(f"tencent:{backup_exc}")
recovered = _load_quotes_lkg(db, cache_key)
if recovered is not None:
return recovered
raise RealtimeApiError(
"SOURCE_UNAVAILABLE",
"quotes unavailable: " + "".join(errors),
) from backup_exc
payload = _quote_payload(rows, source)
_write_cache(db, cache_key, payload, QUOTE_TTL, source)
return payload
def fetch_sector_quote(db: HubDB, code: str, expected_date: str = "") -> dict[str, Any]:
ts_code = str(code or "").strip().upper()
if ts_code.isdigit():
ts_code = f"{ts_code}.SI"
cache_key = f"sector:{ts_code}"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
errors: list[str] = []
try:
row = EastmoneyAdapter().fetch_shenwan_quote(ts_code)
source = str(row.get("source") or "eastmoney_sw")
except Exception as exc:
errors.append(f"eastmoney:{exc}")
recovered = _load_quotes_lkg(db, cache_key)
if recovered is not None:
return recovered
raise RealtimeApiError(
"SOURCE_UNAVAILABLE",
"sector quote unavailable: " + "".join(errors),
) from exc
want = str(expected_date or "").replace("-", "")[:8]
quote_date = str(row.get("quote_date") or "")
if want and quote_date and quote_date != want:
recovered = _load_quotes_lkg(db, cache_key)
if recovered is not None:
return recovered
raise RealtimeApiError("SOURCE_UNAVAILABLE", f"sector quote date {quote_date} != {want}")
payload = _envelope(
row,
{
"tier": "provisional",
"trade_date": quote_date or yyyymmdd(now_shanghai()),
"source": source,
"stale": False,
"staleness_seconds": 0,
"published_at": isoformat(now_shanghai()),
},
)
_write_cache(db, cache_key, payload, INDEX_TTL, source)
return payload
def fetch_limit_pool(db: HubDB, trade_date: str = "") -> dict[str, Any]:
day = yyyymmdd(trade_date or now_shanghai())
cache_key = f"limit-pool:{day}"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
try:
rows = EastmoneyAdapter().fetch_limit_pool(day)
source = "eastmoney:zt_pool"
except Exception as exc:
recovered = _load_quotes_lkg(db, cache_key)
if recovered is not None:
return recovered
raise RealtimeApiError("SOURCE_UNAVAILABLE", f"limit pool unavailable: {exc}") from exc
payload = _envelope(
rows,
{
"tier": "provisional",
"trade_date": day,
"source": source,
"stale": False,
"staleness_seconds": 0,
"published_at": isoformat(now_shanghai()),
},
)
_write_cache(db, cache_key, payload, QUOTE_TTL, source)
return payload
def _eastmoney_named_quotes(codes: list[str]) -> list[dict[str, Any]]:
adapter = EastmoneyAdapter()
rows: list[dict[str, Any]] = []
for index in range(0, len(codes), QUOTE_BATCH):
rows.extend(adapter.fetch_quotes(codes[index:index + QUOTE_BATCH]))
return rows
def _listed_ts_codes(db: HubDB) -> list[str]:
try:
rows = db.fetchall(
"SELECT ts_code FROM stock_master WHERE list_status = 'L' ORDER BY ts_code"
)
except Exception:
return []
return [str(row.get("ts_code") or "") for row in rows if row.get("ts_code")]
def _quote_payload(
rows: list[dict[str, Any]],
source: str,
scope: str = "",
) -> dict[str, Any]:
meta: dict[str, Any] = {
"tier": "provisional",
"trade_date": yyyymmdd(now_shanghai()),
"source": source,
"stale": False,
"staleness_seconds": 0,
"published_at": isoformat(now_shanghai()),
"failover": source.startswith("tencent"),
"delay_notice": "",
}
if scope:
meta["scope"] = scope
return _envelope(rows, meta)
def _load_quotes_lkg(db: HubDB, cache_key: str) -> dict[str, Any] | None:
store = LastKnownGood(db)
item = store.load(cache_key)
payload = item.get("payload") if item else None
if not isinstance(payload, dict):
return None
data = payload.get("data")
if not isinstance(data, list) or not data:
return None
stamped = dict(payload)
meta = dict(stamped.get("meta") or {})
stored = str((item or {}).get("stored_at") or "")
try:
age = max(0, int(time.time() - datetime.fromisoformat(stored).timestamp()))
except Exception:
age = QUOTE_TTL
meta["stale"] = True
meta["staleness_seconds"] = age
meta["delay_notice"] = f"主备免费行情均暂不可用,显示 {age} 秒前的真实快照"
meta["lkg_source"] = str((item or {}).get("source") or meta.get("source") or "")
stamped["meta"] = meta
return stamped
def fetch_intraday(db: HubDB, code: str, date: str = "") -> dict[str, Any]:
ts_code = resolve_code(db, code) or _guess_ts_code(code)
if not ts_code:
raise RealtimeApiError("INVALID_ARGUMENT", f"ambiguous code: {code}")
cache_key = f"intraday:{ts_code}:{date or 'today'}"
cached = _read_cache(db, cache_key)
if cached is not None:
return cached
adapter = EastmoneyAdapter()
try:
payload_data = adapter.fetch_intraday(ts_code, date)
source = "eastmoney:trends2"
except Exception as exc:
recovered = _load_intraday_lkg(db, ts_code, date)
if recovered is None:
raise RealtimeApiError("SOURCE_UNAVAILABLE", f"intraday unavailable: {exc}") from exc
return recovered
payload = _envelope(
payload_data,
{
"tier": "provisional",
"trade_date": yyyymmdd(payload_data.get("trade_date") or date or now_shanghai()),
"source": source,
"stale": False,
"staleness_seconds": 0,
"published_at": isoformat(now_shanghai()),
},
)
_write_cache(db, cache_key, payload, INTRADAY_TTL, source)
return payload
def _load_intraday_lkg(db: HubDB, ts_code: str, date: str = "") -> dict[str, Any] | None:
store = LastKnownGood(db)
keys = [f"intraday:{ts_code}:{date or 'today'}"]
if date:
keys.append(f"intraday:{ts_code}:today")
for key in keys:
item = store.load(key)
payload = _lkg_payload(item)
if payload is not None:
return payload
row = db.fetchone(
"SELECT * FROM last_known_good WHERE cache_key LIKE ? ORDER BY stored_at DESC LIMIT 1",
(f"intraday:{ts_code}:%",),
)
if not row:
return None
try:
raw = json.loads(row["payload"])
except json.JSONDecodeError:
return None
return _mark_stale(raw) if isinstance(raw, dict) else None
def _lkg_payload(item: dict[str, Any] | None) -> dict[str, Any] | None:
if not item:
return None
payload = item.get("payload")
return _mark_stale(payload) if isinstance(payload, dict) else None
def _mark_stale(payload: dict[str, Any]) -> dict[str, Any] | None:
data = payload.get("data")
if not isinstance(data, dict) or not data.get("points"):
return None
stamped = dict(payload)
meta = dict(stamped.get("meta") or {})
meta["stale"] = True
stamped["meta"] = meta
return stamped
def _guess_ts_code(code: str) -> str | None:
raw = str(code or "").strip().upper()
if "." in raw:
return raw
if len(raw) == 6 and raw.isdigit():
if raw.startswith(("5", "6", "9")):
return f"{raw}.SH"
return f"{raw}.SZ"
return None
def _read_cache(db: HubDB, cache_key: str) -> dict[str, Any] | None:
row = db.fetchone("SELECT * FROM rt_cache WHERE cache_key = ?", (cache_key,))
if not row:
return None
expires = str(row.get("expires_at") or "")
now = isoformat(now_shanghai())
if expires and expires < now:
return None
try:
payload = json.loads(row["payload"])
except json.JSONDecodeError:
return None
if isinstance(payload, dict) and isinstance(payload.get("meta"), dict):
stored = str(row.get("stored_at") or "")
try:
age = max(0, int(time.time() - datetime.fromisoformat(stored).timestamp()))
except Exception:
age = 0
payload["meta"]["staleness_seconds"] = age
payload["meta"]["stale"] = age > QUOTE_TTL
return payload
def _write_cache(db: HubDB, cache_key: str, payload: dict[str, Any], ttl: int, source: str) -> None:
from datetime import timedelta
now = now_shanghai()
stored = isoformat(now)
expires = isoformat(now + timedelta(seconds=ttl))
db.execute(
"""
INSERT INTO rt_cache(cache_key, payload, source, stored_at, expires_at)
VALUES (?,?,?,?,?)
ON CONFLICT(cache_key) DO UPDATE SET
payload=excluded.payload, source=excluded.source,
stored_at=excluded.stored_at, expires_at=excluded.expires_at
""",
(cache_key, json.dumps(payload, ensure_ascii=False), source, stored, expires),
)
db.execute(
"""
INSERT INTO last_known_good(cache_key, payload, source, stored_at)
VALUES (?,?,?,?)
ON CONFLICT(cache_key) DO UPDATE SET
payload=excluded.payload, source=excluded.source, stored_at=excluded.stored_at
""",
(cache_key, json.dumps(payload, ensure_ascii=False), source, stored),
)
+31 -2
View File
@@ -64,6 +64,10 @@ class Scheduler:
"precheck": self._precheck, "precheck": self._precheck,
"eod_a": self._eod_a, "eod_a": self._eod_a,
"eod_b": self._eod_b, "eod_b": self._eod_b,
"eod_c": self._eod_c,
"eod_d": self._eod_d,
"eod_e": self._eod_e,
"eod_f": self._eod_f,
"eod_retry": self._eod_retry, "eod_retry": self._eod_retry,
"eod_revise": self._eod_revise, "eod_revise": self._eod_revise,
"stocks_refresh": self._stocks_refresh, "stocks_refresh": self._stocks_refresh,
@@ -104,6 +108,10 @@ class Scheduler:
("precheck", time(8, 45)), ("precheck", time(8, 45)),
("eod_a", time(15, 5)), ("eod_a", time(15, 5)),
("eod_b", time(15, 10)), ("eod_b", time(15, 10)),
("eod_c", time(16, 40)),
("eod_d", time(16, 45)),
("eod_e", time(15, 20)),
("eod_f", time(22, 40)),
("cleanup", time(0, 30)), ("cleanup", time(0, 30)),
("backup", time(0, 40)), ("backup", time(0, 40)),
] ]
@@ -116,7 +124,10 @@ class Scheduler:
key = (job_id, day, at.strftime("%H%M")) key = (job_id, day, at.strftime("%H%M"))
if key in self._fired: if key in self._fired:
continue continue
if job_id in {"eod_a", "eod_b", "stocks_refresh"} and not open_day: if job_id not in self.jobs:
self._fired.add(key)
continue
if job_id in {"eod_a", "eod_b", "eod_c", "eod_d", "eod_e", "eod_f", "stocks_refresh"} and not open_day:
self._fired.add(key) self._fired.add(key)
continue continue
self._fired.add(key) self._fired.add(key)
@@ -127,7 +138,7 @@ class Scheduler:
try: try:
self.run_job(job_id, day) self.run_job(job_id, day)
except Exception: except Exception:
if job_id not in {"eod_a", "eod_b", "stocks_refresh"}: if job_id not in {"eod_a", "eod_b", "eod_c", "eod_d", "eod_e", "eod_f", "stocks_refresh"}:
raise raise
# Keep the tick alive; evening retries take over. # Keep the tick alive; evening retries take over.
LOGGER.exception("scheduled job %s failed for %s", job_id, day) LOGGER.exception("scheduled job %s failed for %s", job_id, day)
@@ -195,6 +206,12 @@ class Scheduler:
LOGGER.warning("eod retry failed for %s", day, exc_info=True) LOGGER.warning("eod retry failed for %s", day, exc_info=True)
ran.append("eod_retry") ran.append("eod_retry")
self._settle_eod(day) self._settle_eod(day)
if "eod_e" in self.jobs and not self.pipeline.extended_publication_complete("sector_daily", day):
try:
self.run_job("eod_e", day)
ran.append("eod_e")
except Exception:
LOGGER.exception("sector_daily retry failed for %s", day)
return ran return ran
def _settle_eod(self, day: str) -> None: def _settle_eod(self, day: str) -> None:
@@ -531,6 +548,18 @@ class Scheduler:
def _eod_b(self, trade_date: str) -> dict[str, Any]: def _eod_b(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_batch_b(trade_date) return self.pipeline.run_eod_batch_b(trade_date)
def _eod_c(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_batch_c(trade_date)
def _eod_d(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_batch_d(trade_date)
def _eod_e(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_batch_e(trade_date)
def _eod_f(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_batch_f(trade_date)
def _eod_retry(self, trade_date: str) -> dict[str, Any]: def _eod_retry(self, trade_date: str) -> dict[str, Any]:
return self.pipeline.run_eod_missing(trade_date) return self.pipeline.run_eod_missing(trade_date)
+138 -2
View File
@@ -50,10 +50,11 @@ def envelope(data: Any, meta: dict[str, Any]) -> dict[str, Any]:
class V1API: class V1API:
def __init__(self, db: HubDB, pipeline: Pipeline, settings: Settings) -> None: def __init__(self, db: HubDB, pipeline: Pipeline, settings: Settings, ifind: Any = None) -> None:
self.db = db self.db = db
self.pipeline = pipeline self.pipeline = pipeline
self.settings = settings self.settings = settings
self.ifind = ifind
def handle(self, path: str, query: dict[str, list[str]]) -> dict[str, Any]: def handle(self, path: str, query: dict[str, list[str]]) -> dict[str, Any]:
q = {key: values[-1] if values else "" for key, values in query.items()} q = {key: values[-1] if values else "" for key, values in query.items()}
@@ -73,12 +74,60 @@ class V1API:
return self.moneyflow(q) return self.moneyflow(q)
if path == "/v1/auction": if path == "/v1/auction":
return self.auction(q) return self.auction(q)
if path == "/v1/limit-events":
return self.limit_events(q)
if path == "/v1/popularity":
return self.popularity(q)
if path == "/v1/dragon-tiger":
return self.dragon_tiger(q)
if path == "/v1/sectors":
return self.sectors(q)
if path == "/v1/quotes/latest":
return self.quotes_latest(q)
if path == "/v1/indexes/quotes":
return self.index_quotes(q)
if path == "/v1/intraday/points":
return self.intraday_points(q)
if path == "/v1/sectors/quote":
return self.sector_quote(q)
if path == "/v1/limit-pool":
return self.limit_pool(q)
if path == "/v1/query":
return self.query_api(q)
if path == "/v1/datasets/status": if path == "/v1/datasets/status":
return self.dataset_status(q.get("date") or "") return self.dataset_status(q.get("date") or "")
if path == "/v1/batches": if path == "/v1/batches":
return self.batches(q.get("date") or "", q.get("dataset") or "") return self.batches(q.get("date") or "", q.get("dataset") or "")
raise ApiError("INVALID_ARGUMENT", f"unknown endpoint: {path}") raise ApiError("INVALID_ARGUMENT", f"unknown endpoint: {path}")
def query_api(self, body: dict[str, Any]) -> dict[str, Any]:
from datahub.steward import steward_query
payload = dict(body or {})
raw_params = payload.get("params")
if isinstance(raw_params, str):
payload["params"] = _parse_json(raw_params) or {}
return steward_query(self, payload)
def sector_quote(self, q: dict[str, str]) -> dict[str, Any]:
from datahub.realtime_serve import RealtimeApiError, fetch_sector_quote
code = str(q.get("code") or q.get("ts_code") or "").strip()
if not code:
raise ApiError("INVALID_ARGUMENT", "code is required")
try:
return fetch_sector_quote(self.db, code, str(q.get("date") or ""))
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
def limit_pool(self, q: dict[str, str]) -> dict[str, Any]:
from datahub.realtime_serve import RealtimeApiError, fetch_limit_pool
try:
return fetch_limit_pool(self.db, str(q.get("date") or q.get("trade_date") or ""))
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
def health(self) -> dict[str, Any]: def health(self) -> dict[str, Any]:
today = yyyymmdd(now_shanghai()) today = yyyymmdd(now_shanghai())
cal = self.db.fetchone( cal = self.db.fetchone(
@@ -87,12 +136,18 @@ class V1API:
) )
is_open = bool(cal and cal["is_open"] == 1) is_open = bool(cal and cal["is_open"] == 1)
sources = self.db.fetchall("SELECT * FROM src_health") sources = self.db.fetchall("SELECT * FROM src_health")
ifind = getattr(self, "ifind", None)
ifind_status = ifind.status() if ifind is not None else {"configured": False}
return envelope( return envelope(
{ {
"status": "ok", "status": "ok",
"session_phase": session_phase(now_shanghai(), is_open), "session_phase": session_phase(now_shanghai(), is_open),
"trade_date": today, "trade_date": today,
"is_open_day": is_open, "is_open_day": is_open,
"ifind": {
"configured": bool(ifind_status.get("configured")),
"access_ready": bool(ifind_status.get("access_ready")),
},
"sources": [ "sources": [
{ {
"provider": row["provider"], "provider": row["provider"],
@@ -197,9 +252,82 @@ class V1API:
def auction(self, q: dict[str, str]) -> dict[str, Any]: def auction(self, q: dict[str, str]) -> dict[str, Any]:
return self._published_rows(dataset="auction", table="eod_auction", q=q, source="tushare:stk_auction") return self._published_rows(dataset="auction", table="eod_auction", q=q, source="tushare:stk_auction")
def limit_events(self, q: dict[str, str]) -> dict[str, Any]:
return self._published_rows(
dataset="limit_events",
table="eod_limit_events",
q=q,
source="tushare:limit_list_d",
extra_filters={"limit_type": q.get("limit_type") or ""},
)
def popularity(self, q: dict[str, str]) -> dict[str, Any]:
return self._published_rows(
dataset="popularity",
table="eod_popularity",
q=q,
source="tushare:ths_hot+dc_hot",
extra_filters={"source": q.get("source") or ""},
)
def dragon_tiger(self, q: dict[str, str]) -> dict[str, Any]:
return self._published_rows(
dataset="dragon_tiger",
table="eod_dragon_tiger",
q=q,
source="tushare:hm_detail",
)
def sectors(self, q: dict[str, str]) -> dict[str, Any]:
return self._published_rows(
dataset="sector_daily",
table="eod_sector_daily",
q=q,
source="tushare:ths_daily+dc_index+sw_daily",
extra_filters={"family": q.get("family") or ""},
)
def quotes_latest(self, q: dict[str, str]) -> dict[str, Any]:
from datahub.realtime_serve import RealtimeApiError, fetch_market_quotes, fetch_quotes
codes = [item.strip() for item in str(q.get("codes") or "").split(",") if item.strip()]
try:
if codes:
return fetch_quotes(self.db, codes)
return fetch_market_quotes(self.db)
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
def index_quotes(self, q: dict[str, str]) -> dict[str, Any]:
from datahub.realtime_serve import RealtimeApiError, fetch_index_quotes
try:
return fetch_index_quotes(self.db)
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
def intraday_points(self, q: dict[str, str]) -> dict[str, Any]:
from datahub.realtime_serve import RealtimeApiError, fetch_intraday
code = str(q.get("code") or "").strip()
if not code:
raise ApiError("INVALID_ARGUMENT", "code is required")
raw_date = str(q.get("date") or "").strip()
try:
trade_date = yyyymmdd(raw_date or now_shanghai())
except ValueError as exc:
raise ApiError("INVALID_ARGUMENT", str(exc)) from exc
try:
return fetch_intraday(self.db, code, trade_date)
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
def dataset_status(self, date: str) -> dict[str, Any]: def dataset_status(self, date: str) -> dict[str, Any]:
trade_date = yyyymmdd(date or now_shanghai()) trade_date = yyyymmdd(date or now_shanghai())
datasets = ("daily", "valuation", "moneyflow", "auction", "index_daily", "stocks") datasets = (
"daily", "valuation", "moneyflow", "auction", "index_daily", "stocks",
"limit_events", "popularity", "dragon_tiger", "sector_daily",
)
items = [] items = []
for dataset in datasets: for dataset in datasets:
pub = self.db.fetchone( pub = self.db.fetchone(
@@ -244,6 +372,7 @@ class V1API:
source: str, source: str,
adjust: str = "none", adjust: str = "none",
default_code: str = "", default_code: str = "",
extra_filters: dict[str, str] | None = None,
) -> dict[str, Any]: ) -> dict[str, Any]:
trade_date = q.get("date") or q.get("trade_date") or "" trade_date = q.get("date") or q.get("trade_date") or ""
code = q.get("code") or default_code code = q.get("code") or default_code
@@ -264,6 +393,7 @@ class V1API:
if resolved is None: if resolved is None:
raise ApiError("INVALID_ARGUMENT", f"ambiguous code: {code}") raise ApiError("INVALID_ARGUMENT", f"ambiguous code: {code}")
ts_code = resolved ts_code = resolved
filters = {key: value for key, value in (extra_filters or {}).items() if value}
# For a range, use per-date published batch. Single-date is the common path. # For a range, use per-date published batch. Single-date is the common path.
if start == end: if start == end:
pub = self.db.fetchone( pub = self.db.fetchone(
@@ -282,6 +412,9 @@ class V1API:
if ts_code: if ts_code:
sql += " AND ts_code = ?" sql += " AND ts_code = ?"
params.append(ts_code) params.append(ts_code)
for key, value in filters.items():
sql += f" AND {key} = ?"
params.append(value)
sql += " ORDER BY ts_code LIMIT ? OFFSET ?" sql += " ORDER BY ts_code LIMIT ? OFFSET ?"
params.extend([limit, offset]) params.extend([limit, offset])
rows = [dict(row) for row in self.db.fetchall(sql, tuple(params))] rows = [dict(row) for row in self.db.fetchall(sql, tuple(params))]
@@ -317,6 +450,9 @@ class V1API:
if ts_code: if ts_code:
sql += " AND ts_code = ?" sql += " AND ts_code = ?"
params.append(ts_code) params.append(ts_code)
for key, value in filters.items():
sql += f" AND {key} = ?"
params.append(value)
sql += " ORDER BY ts_code" sql += " ORDER BY ts_code"
rows.extend(self.db.fetchall(sql, tuple(params))) rows.extend(self.db.fetchall(sql, tuple(params)))
sliced = rows[offset: offset + limit] sliced = rows[offset: offset + limit]
+8
View File
@@ -26,6 +26,8 @@ class Settings:
api_token: str = "" api_token: str = ""
admin_password: str = "" admin_password: str = ""
tushare_token: str = "" tushare_token: str = ""
ifind_refresh_token: str = ""
ifind_access_token: str = ""
db_path: Path = DEFAULT_DB_PATH db_path: Path = DEFAULT_DB_PATH
backup_dir: Path = DEFAULT_BACKUP_DIR backup_dir: Path = DEFAULT_BACKUP_DIR
quality: dict[str, Any] = field(default_factory=dict) quality: dict[str, Any] = field(default_factory=dict)
@@ -56,6 +58,10 @@ class Settings:
def index_history_trading_days(self) -> int: def index_history_trading_days(self) -> int:
return int(self.quality.get("index_history_trading_days") or 260) return int(self.quality.get("index_history_trading_days") or 260)
@property
def daily_history_trading_days(self) -> int:
return int(self.quality.get("daily_history_trading_days") or 250)
@property @property
def moneyflow_history_trading_days(self) -> int: def moneyflow_history_trading_days(self) -> int:
return int(self.quality.get("moneyflow_history_trading_days") or 60) return int(self.quality.get("moneyflow_history_trading_days") or 60)
@@ -109,6 +115,8 @@ def load_settings(
api_token=str(environ.get("DATAHUB_TOKEN") or "").strip(), api_token=str(environ.get("DATAHUB_TOKEN") or "").strip(),
admin_password=str(environ.get("DATAHUB_ADMIN_PASSWORD") or "").strip(), admin_password=str(environ.get("DATAHUB_ADMIN_PASSWORD") or "").strip(),
tushare_token=str(environ.get("TUSHARE_TOKEN") or "").strip(), tushare_token=str(environ.get("TUSHARE_TOKEN") or "").strip(),
ifind_refresh_token=str(environ.get("IFIND_REFRESH_TOKEN") or "").strip(),
ifind_access_token=str(environ.get("IFIND_ACCESS_TOKEN") or "").strip(),
db_path=db_path, db_path=db_path,
backup_dir=backup_dir, backup_dir=backup_dir,
quality=_load_quality(quality_path), quality=_load_quality(quality_path),
+420
View File
@@ -0,0 +1,420 @@
"""Website-facing data steward: pick source, fail over, cache, never fake zeros.
The main site asks for a business/Tushare-shaped API. This module decides whether
to serve a published EOD table, live free quotes, or an internal Tushare pull.
"""
from __future__ import annotations
import hashlib
import json
from typing import Any
from datahub.adapters.base import AdapterError
from datahub.adapters.tushare import TUSHARE_FIELDS
from datahub.numbers import finite_number
from datahub.realtime_serve import (
RealtimeApiError,
_read_cache,
_write_cache,
fetch_index_quotes,
fetch_market_quotes,
fetch_quotes,
)
from datahub.serving import ApiError, envelope
from datahub.timeutil import isoformat, now_shanghai, yyyymmdd
API_TO_DATASET = {
"trade_cal": "calendar",
"stock_basic": "stocks",
"daily": "daily",
"daily_basic": "valuation",
"index_daily": "index_daily",
"moneyflow": "moneyflow",
"stk_auction": "auction",
"limit_list_d": "limit_events",
"ths_hot": "popularity",
"dc_hot": "popularity",
"hm_detail": "dragon_tiger",
"ths_daily": "sector_daily",
"dc_index": "sector_daily",
"sw_daily": "sector_daily",
}
DATASET_FETCHER = {
"calendar": lambda api, q: api.calendar(q.get("from") or q.get("start_date") or "", q.get("to") or q.get("end_date") or ""),
"stocks": lambda api, q: api.stocks(q.get("updated_since") or "", q),
"daily": lambda api, q: api.daily_bars(_hub_query(q, adjust="none")),
"valuation": lambda api, q: api.valuation(_hub_query(q)),
"index_daily": lambda api, q: api.index_bars(_hub_query(q)),
"moneyflow": lambda api, q: api.moneyflow(_hub_query(q)),
"auction": lambda api, q: api.auction(_hub_query(q)),
"limit_events": lambda api, q: api.limit_events(_hub_query(q)),
"popularity": lambda api, q: api.popularity(_hub_query(q)),
"dragon_tiger": lambda api, q: api.dragon_tiger(_hub_query(q)),
"sector_daily": lambda api, q: api.sectors(_hub_query(q)),
}
SCALE_TO_TUSHARE = {
"daily": {"vol": 100.0, "amount": 1000.0},
"index_daily": {"vol": 100.0, "amount": 1000.0},
"valuation": {"total_mv": 10000.0, "circ_mv": 10000.0},
"moneyflow": {
"buy_sm_amount": 10000.0,
"sell_sm_amount": 10000.0,
"buy_md_amount": 10000.0,
"sell_md_amount": 10000.0,
"buy_lg_amount": 10000.0,
"sell_lg_amount": 10000.0,
"buy_elg_amount": 10000.0,
"sell_elg_amount": 10000.0,
"net_mf_amount": 10000.0,
},
"auction": {"vol": 100.0, "float_share": 10000.0},
"limit_events": {"limit_amount": 10000.0, "float_mv": 10000.0, "total_mv": 10000.0},
"dragon_tiger": {"buy_amount": 10000.0, "sell_amount": 10000.0, "net_amount": 10000.0},
}
LIVE_TTL = {
"index_member_all": 6 * 3600,
"stk_limit": 3600,
"suspend_d": 6 * 3600,
"adj_factor": 3600,
"hm_list": 24 * 3600,
"ths_index": 24 * 3600,
"ths_member": 6 * 3600,
"stk_mins": 20,
"top_list": 3600,
"top_inst": 3600,
}
BLOCKED_LIVE_APIS = {"rt_sw_k"}
IFIND_APIS = {
"ifind_wencai": "wencai",
"ifind_snapshots": "snapshots",
"ifind_history": "history",
"ifind_realtime": "realtime",
"ifind_intraday": "intraday",
"ifind_status": "status",
}
def steward_query(api, body: dict[str, Any]) -> dict[str, Any]:
api_name = str(body.get("api_name") or "").strip()
params = body.get("params") if isinstance(body.get("params"), dict) else {}
fields = str(body.get("fields") or "")
if not api_name:
raise ApiError("INVALID_ARGUMENT", "api_name is required")
if api_name in BLOCKED_LIVE_APIS:
raise ApiError("INVALID_ARGUMENT", "rt_sw_k is disabled; use published sw_daily or free Shenwan realtime")
if api_name in IFIND_APIS:
return _ifind_query(api, api_name, params, fields)
if api_name == "rt_k":
return _realtime_quotes(api, params, fields)
if api_name == "rt_idx_k":
return _realtime_index_quotes(api, params, fields)
dataset = API_TO_DATASET.get(api_name)
if dataset:
published = _try_published(api, api_name, dataset, params, fields)
if published is not None:
return published
rows = _live_tushare(api, api_name, params, fields)
return envelope(
_project(rows, fields),
{
"tier": "live",
"source": "tushare",
"stale": False,
"staleness_seconds": 0,
"row_shape": "tushare",
"published_at": isoformat(now_shanghai()),
},
)
def _ifind_query(api, api_name: str, params: dict[str, Any], fields: str) -> dict[str, Any]:
adapter = getattr(api, "ifind", None)
dataset = IFIND_APIS[api_name]
if adapter is None:
raise ApiError("SOURCE_UNAVAILABLE", "iFinD adapter is not attached")
if dataset == "status":
return envelope(
[dict(adapter.status())],
{
"tier": "live",
"source": "ifind",
"stale": False,
"staleness_seconds": 0,
"row_shape": "ifind",
"published_at": isoformat(now_shanghai()),
},
)
if not adapter.configured:
raise ApiError("SOURCE_UNAVAILABLE", "iFinD 尚未配置")
try:
rows = adapter.fetch(dataset, dict(params))
except AdapterError as exc:
raise ApiError("SOURCE_UNAVAILABLE", str(exc)) from exc
return envelope(
_project(rows, fields),
{
"tier": "live",
"source": "ifind",
"stale": False,
"staleness_seconds": 0,
"row_shape": "ifind",
"published_at": isoformat(now_shanghai()),
},
)
def _try_published(api, api_name: str, dataset: str, params: dict[str, Any], fields: str) -> dict[str, Any] | None:
fetcher = DATASET_FETCHER.get(dataset)
if fetcher is None:
return None
query = _hub_query(params)
if dataset == "popularity":
query["source"] = "ths" if api_name == "ths_hot" else "dc"
if dataset == "sector_daily":
query["family"] = {"ths_daily": "ths", "dc_index": "dc", "sw_daily": "sw"}.get(api_name, "")
if dataset == "limit_events":
limit_type = str(params.get("limit_type") or "").strip().upper()
if limit_type:
query["limit_type"] = limit_type
if dataset == "calendar" and not (query.get("from") and query.get("to")):
start = str(params.get("start_date") or params.get("from") or "")
end = str(params.get("end_date") or params.get("to") or start)
if not start or not end:
return None
query = {"from": start, "to": end}
try:
payload = fetcher(api, query)
except ApiError as exc:
if exc.code in {"DATASET_NOT_PUBLISHED", "STALE_DATA", "INVALID_ARGUMENT"}:
return None
raise
rows = list(payload.get("data") or [])
# A published multi-source sector batch can be temporarily incomplete when
# one upstream family is late. Let the hub try that family live instead of
# returning an authoritative-looking empty result to the website.
if dataset == "sector_daily" and query.get("family") and not rows:
return None
if dataset == "stocks":
rows = _filter_stocks(rows, params)
# The published master is intentionally the active list. Historical
# delisted/paused lookups still belong in the hub, so use its live
# Tushare adapter when those filters cannot be answered by the snapshot.
if not rows and any(params.get(key) for key in ("ts_code", "list_status", "name")):
return None
if dataset == "calendar":
rows = _filter_calendar(rows, params)
native = _to_tushare_native(dataset, rows)
meta = dict(payload.get("meta") or {})
meta["row_shape"] = "tushare"
meta["source"] = str(meta.get("source") or "datahub")
return envelope(_project(native, fields), meta)
def _realtime_quotes(api, params: dict[str, Any], fields: str) -> dict[str, Any]:
codes = [item.strip() for item in str(params.get("ts_code") or params.get("codes") or "").split(",") if item.strip()]
try:
payload = fetch_quotes(api.db, codes) if codes else fetch_market_quotes(api.db)
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
rows = [_quote_to_rt_k(item) for item in (payload.get("data") or []) if isinstance(item, dict)]
rows = [item for item in rows if item]
meta = dict(payload.get("meta") or {})
meta["row_shape"] = "tushare"
return envelope(_project(rows, fields), meta)
def _realtime_index_quotes(api, params: dict[str, Any], fields: str) -> dict[str, Any]:
try:
payload = fetch_index_quotes(api.db)
except RealtimeApiError as exc:
raise ApiError(exc.code, exc.message) from exc
wanted = {
item.strip()
for item in str(params.get("ts_code") or "").split(",")
if item.strip()
}
rows = []
for item in payload.get("data") or []:
if not isinstance(item, dict):
continue
converted = _quote_to_rt_k(item)
if not converted:
continue
if wanted and converted.get("ts_code") not in wanted and str(item.get("code") or "") not in {
code.split(".")[0] for code in wanted
}:
continue
rows.append(converted)
meta = dict(payload.get("meta") or {})
meta["row_shape"] = "tushare"
return envelope(_project(rows, fields), meta)
def _live_tushare(api, api_name: str, params: dict[str, Any], fields: str) -> list[dict[str, Any]]:
wanted_fields = fields or TUSHARE_FIELDS.get(api_name, "")
cache_key = _live_cache_key(api_name, params, wanted_fields)
ttl = LIVE_TTL.get(api_name, 1800)
cached = _read_cache(api.db, cache_key)
if cached is not None:
data = cached.get("data")
if isinstance(data, list):
return [dict(item) for item in data if isinstance(item, dict)]
pipeline = api.pipeline
if not pipeline.breaker.allow():
recovered = _live_lkg(api.db, cache_key)
if recovered is not None:
return recovered
raise ApiError("SOURCE_UNAVAILABLE", "Tushare circuit open")
pipeline.bucket.acquire()
try:
rows = pipeline.adapter.query_raw(api_name, dict(params), wanted_fields)
pipeline.breaker.record_success()
except Exception as exc:
pipeline.breaker.record_failure(str(exc))
recovered = _live_lkg(api.db, cache_key)
if recovered is not None:
return recovered
raise ApiError("SOURCE_UNAVAILABLE", f"Tushare {api_name} unavailable: {exc}") from exc
payload = envelope(
rows,
{
"tier": "live",
"source": "tushare",
"stale": False,
"staleness_seconds": 0,
"row_shape": "tushare",
"published_at": isoformat(now_shanghai()),
},
)
_write_cache(api.db, cache_key, payload, ttl, "tushare")
return rows
def _live_lkg(db, cache_key: str) -> list[dict[str, Any]] | None:
from datahub.governance.lkg import LastKnownGood
item = LastKnownGood(db).load(cache_key)
payload = item.get("payload") if item else None
if not isinstance(payload, dict):
return None
data = payload.get("data")
if not isinstance(data, list) or not data:
return None
return [dict(row) for row in data if isinstance(row, dict)]
def _live_cache_key(api_name: str, params: dict[str, Any], fields: str) -> str:
packed = json.dumps({"api": api_name, "params": params, "fields": fields}, sort_keys=True, ensure_ascii=False)
digest = hashlib.sha1(packed.encode("utf-8")).hexdigest()
return f"steward:{api_name}:{digest}"
def _hub_query(params: dict[str, Any], **extra: Any) -> dict[str, str]:
query = {key: str(value) for key, value in extra.items() if value not in (None, "")}
raw_date = params.get("trade_date") or params.get("date") or ""
date = yyyymmdd(raw_date) if raw_date else ""
raw_start = params.get("start_date") or params.get("from") or date
raw_end = params.get("end_date") or params.get("to") or date
start = yyyymmdd(raw_start) if raw_start else ""
end = yyyymmdd(raw_end) if raw_end else ""
code = str(params.get("ts_code") or params.get("code") or "").strip()
if code:
query["code"] = code
if date and not (params.get("start_date") or params.get("end_date")):
query["date"] = date
else:
if start:
query["from"] = start
if end:
query["to"] = end
return query
def _to_tushare_native(dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
scales = SCALE_TO_TUSHARE.get(dataset) or {}
converted: list[dict[str, Any]] = []
for row in rows:
item = dict(row)
if item.get("vol") in (None, ""):
item["vol"] = item.get("volume")
item.pop("volume", None)
for field, factor in scales.items():
if field in item and item[field] not in (None, ""):
number = finite_number(item.get(field))
item[field] = number / factor if factor else number
if dataset == "popularity" and item.get("ts_name") and not item.get("name"):
item["name"] = item.get("ts_name")
if dataset == "dragon_tiger" and item.get("ts_name") and not item.get("name"):
item["name"] = item.get("ts_name")
if dataset == "sector_daily" and item.get("pct_change") is not None and item.get("pct_chg") is None:
item["pct_chg"] = item.get("pct_change")
if dataset == "calendar":
item["is_open"] = 1 if item.get("is_open") in (True, 1, "1", "Y", "y") else 0
converted.append(item)
return converted
def _quote_to_rt_k(row: dict[str, Any]) -> dict[str, Any] | None:
ts_code = str(row.get("ts_code") or "").strip()
close = finite_number(row.get("close") if row.get("close") not in (None, "") else row.get("price"))
previous = finite_number(
row.get("pre_close") if row.get("pre_close") not in (None, "") else row.get("previous_close")
)
if not ts_code or close <= 0:
return None
item = {
"ts_code": ts_code,
"name": row.get("name") or "",
"open": row.get("open"),
"high": row.get("high"),
"low": row.get("low"),
"close": close,
"pre_close": previous,
"vol": row.get("vol") if row.get("vol") not in (None, "") else row.get("volume"),
"amount": row.get("amount"),
"pct_chg": row.get("pct_chg") if row.get("pct_chg") not in (None, "") else row.get("change"),
"trade_time": row.get("quote_time") or row.get("trade_time") or "",
"quote_date": row.get("quote_date") or "",
"source": row.get("source") or "",
"delayed": bool(row.get("delayed")),
"delay_seconds": row.get("delay_seconds") or 0,
"delay_notice": row.get("delay_notice") or "",
}
return item
def _filter_stocks(rows: list[dict[str, Any]], params: dict[str, Any]) -> list[dict[str, Any]]:
ts_code = str(params.get("ts_code") or "").strip().upper()
status = str(params.get("list_status") or "").strip()
name = str(params.get("name") or "").strip()
filtered = rows
if ts_code:
filtered = [row for row in filtered if str(row.get("ts_code") or "").upper() == ts_code]
if status:
filtered = [row for row in filtered if str(row.get("list_status") or status) == status]
if name:
filtered = [row for row in filtered if name.casefold() in str(row.get("name") or "").casefold()]
return filtered
def _filter_calendar(rows: list[dict[str, Any]], params: dict[str, Any]) -> list[dict[str, Any]]:
start = yyyymmdd(params.get("start_date") or params.get("from") or "")
end = yyyymmdd(params.get("end_date") or params.get("to") or start)
if start and end:
rows = [row for row in rows if start <= yyyymmdd(row.get("cal_date")) <= end]
if params.get("is_open") in (1, "1", True):
rows = [row for row in rows if int(row.get("is_open") or 0) == 1]
return rows
def _project(rows: list[dict[str, Any]], fields: str) -> list[dict[str, Any]]:
keys = [item.strip() for item in str(fields or "").split(",") if item.strip()]
if not keys:
return rows
return [{key: row.get(key) for key in keys} for row in rows]
+49 -3
View File
@@ -1,5 +1,6 @@
from __future__ import annotations from __future__ import annotations
import copy
import sys import sys
from pathlib import Path from pathlib import Path
@@ -45,13 +46,53 @@ RAW = {
{"ts_code": "600000.SH", "trade_date": "20240902", "vol": 100, "price": 10.15, "amount": 1500000, "pre_close": 10.00, "turnover_rate": 0.1, "volume_ratio": 1.2, "float_share": 2000}, {"ts_code": "600000.SH", "trade_date": "20240902", "vol": 100, "price": 10.15, "amount": 1500000, "pre_close": 10.00, "turnover_rate": 0.1, "volume_ratio": 1.2, "float_share": 2000},
{"ts_code": "000001.SZ", "trade_date": "20240902", "vol": 80, "price": 11.05, "amount": 1200000, "pre_close": 11.10, "turnover_rate": 0.2, "volume_ratio": 0.9, "float_share": 1800}, {"ts_code": "000001.SZ", "trade_date": "20240902", "vol": 80, "price": 11.05, "amount": 1200000, "pre_close": 11.10, "turnover_rate": 0.2, "volume_ratio": 0.9, "float_share": 1800},
], ],
"limit_list_d": [
{"trade_date": "20240902", "ts_code": "600000.SH", "industry": "银行", "name": "浦发银行", "close": 10.2, "pct_chg": 9.95, "amount": 1e8, "limit_amount": 5000, "float_mv": 800, "total_mv": 1000, "turnover_ratio": 5.0, "fd_amount": 2e7, "first_time": "09:30:01", "last_time": "14:55:00", "open_times": 0, "up_stat": "1/1", "limit_times": 1, "limit_type": "U"},
],
"ths_hot": [
{"ts_code": "600000.SH", "ts_name": "浦发银行", "hot": 90.0, "rank": 1, "pct_change": 1.2, "current_price": 10.2, "concept": "银行", "data_type": "热股", "trade_date": "20240902"},
],
"dc_hot": [
{"ts_code": "600000.SH", "ts_name": "浦发银行", "rank": 2, "pct_change": 1.2, "current_price": 10.2, "hot": 80.0, "concept": "银行", "data_type": "A股市场", "trade_date": "20240902"},
],
"hm_detail": [
{"trade_date": "20240902", "ts_code": "600000.SH", "ts_name": "浦发银行", "buy_amount": 1000, "sell_amount": 200, "net_amount": 800, "hm_name": "测试游资", "hm_orgs": "某某营业部", "tag": "超买"},
],
"top_list": [
{"trade_date": "20240902", "ts_code": "600000.SH", "name": "浦发银行", "pct_change": 9.95, "reason": "涨幅偏离值达7%"},
],
"ths_daily": [
{"ts_code": "885811.TI", "trade_date": "20240902", "open": 1000, "high": 1010, "low": 990, "close": 1005, "pre_close": 995, "pct_change": 1.0, "vol": 100, "turnover_rate": 1.2},
],
"dc_index": [
{"ts_code": "BK0475", "trade_date": "20240902", "name": "银行", "open": 100, "high": 101, "low": 99, "close": 100.5, "pre_close": 99.5, "pct_change": 1.0, "vol": 10, "amount": 1e8, "turnover_rate": 0.5},
],
"sw_daily": [
{"ts_code": "801780.SI", "trade_date": "20240902", "name": "银行", "open": 2000, "high": 2010, "low": 1990, "close": 2005, "pct_change": 0.8, "vol": 50, "amount": 2e8},
],
"stk_limit": [
{"ts_code": "600000.SH", "trade_date": "20240902", "up_limit": 11.22, "down_limit": 9.18},
{"ts_code": "000001.SZ", "trade_date": "20240902", "up_limit": 12.21, "down_limit": 9.99},
],
"index_member_all": [
{
"l2_code": "801780.SI",
"l2_name": "银行",
"ts_code": "600000.SH",
"name": "浦发银行",
"in_date": "20140101",
"out_date": "",
"is_new": "Y",
}
],
"hm_list": [{"name": "测试游资", "desc": "测试", "orgs": "某某营业部"}],
} }
def fake_transport(api_name: str, params: dict, fields: str): def fake_transport(api_name: str, params: dict, fields: str):
if api_name == "index_daily": if api_name == "index_daily":
code = params.get("ts_code") code = params.get("ts_code")
rows = [row for row in RAW["index_daily"] if row["ts_code"] == code] rows = [copy.deepcopy(row) for row in RAW["index_daily"] if row["ts_code"] == code]
trade_date = str(params.get("trade_date") or "") trade_date = str(params.get("trade_date") or "")
start = str(params.get("start_date") or "") start = str(params.get("start_date") or "")
end = str(params.get("end_date") or "") end = str(params.get("end_date") or "")
@@ -65,5 +106,10 @@ def fake_transport(api_name: str, params: dict, fields: str):
if api_name == "trade_cal": if api_name == "trade_cal":
start = str(params.get("start_date") or "") start = str(params.get("start_date") or "")
end = str(params.get("end_date") or "99999999") end = str(params.get("end_date") or "99999999")
return [row for row in RAW["trade_cal"] if start <= row["cal_date"] <= end] return [copy.deepcopy(row) for row in RAW["trade_cal"] if start <= row["cal_date"] <= end]
return list(RAW.get(api_name) or []) rows = copy.deepcopy(list(RAW.get(api_name) or []))
if api_name == "limit_list_d":
limit_type = str(params.get("limit_type") or "")
if limit_type:
rows = [row for row in rows if str(row.get("limit_type") or "") == limit_type]
return rows
+34
View File
@@ -61,6 +61,18 @@ class ApiContractTests(unittest.TestCase):
self.hub.stop() self.hub.stop()
self.tmp.cleanup() self.tmp.cleanup()
def _post(self, path: str, body: dict, token: str | None = None) -> tuple[int, dict]:
headers = {"Content-Type": "application/json"}
if token is not None:
headers["X-Datahub-Token"] = token
raw = json.dumps(body).encode("utf-8")
req = Request(self.base + path, data=raw, headers=headers, method="POST")
try:
with urlopen(req, timeout=5) as resp:
return resp.status, json.loads(resp.read().decode())
except HTTPError as exc:
return exc.code, json.loads(exc.read().decode())
def _get(self, path: str, token: str | None = None) -> tuple[int, dict]: def _get(self, path: str, token: str | None = None) -> tuple[int, dict]:
headers = {} headers = {}
if token is not None: if token is not None:
@@ -145,6 +157,28 @@ class ApiContractTests(unittest.TestCase):
self.assertNotIn("tushare-secret-token-xyz", blob) self.assertNotIn("tushare-secret-token-xyz", blob)
self.assertNotIn(self.token, blob) self.assertNotIn(self.token, blob)
def test_ifind_credentials_require_token_and_update_adapter(self) -> None:
status, body = self._post(
"/v1/credentials/ifind",
{"ifind_refresh_token": "refresh-secret", "ifind_access_token": "access-secret"},
token=None,
)
self.assertEqual(status, 401)
self.assertEqual(body["error"]["code"], "UNAUTHORIZED")
status, body = self._post(
"/v1/credentials/ifind",
{"ifind_refresh_token": "refresh-secret", "ifind_access_token": "access-secret"},
token=self.token,
)
self.assertEqual(status, 200, body)
self.assertTrue(body["data"]["configured"])
self.assertTrue(body["data"]["access_ready"])
self.assertTrue(self.hub.ifind.configured)
self.assertEqual(self.hub.auth.load_credential("ifind_refresh_token"), "refresh-secret")
blob = json.dumps(body)
self.assertNotIn("refresh-secret", blob)
self.assertNotIn("access-secret", blob)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
@@ -0,0 +1,81 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from datahub.adapters.tushare import TushareAdapter
from datahub.crypto import SecretVault
from datahub.hub import Hub
from datahub.settings import Settings
from tests.fixtures import TRADE_DATE, fake_transport
class ExtendedEodTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
key = SecretVault.generate_key()
settings = Settings(
host="127.0.0.1",
port=0,
encryption_key=key,
api_token="k" * 32,
admin_password="StartPass1",
tushare_token="tushare-secret",
db_path=Path(self.tmp.name) / "hub.db",
backup_dir=Path(self.tmp.name) / "backups",
scheduler_enabled=False,
quality={"daily_row_ratio": 0.5, "null_rate_max": 0.5, "list_limit_default": 5000, "list_limit_max": 5000},
)
adapter = TushareAdapter("tushare-secret", transport=fake_transport)
self.hub = Hub(settings, adapter=adapter)
self.hub.pipeline.ingest_reference(TRADE_DATE)
for dataset in ("daily", "valuation", "moneyflow", "auction", "index_daily"):
self.hub.pipeline.run_dataset(dataset, TRADE_DATE)
def tearDown(self) -> None:
self.hub.stop()
self.tmp.cleanup()
def test_extended_soft_datasets_publish_and_serve(self) -> None:
results = self.hub.pipeline.run_extended_soft(
("limit_events", "popularity", "dragon_tiger", "sector_daily"),
TRADE_DATE,
)
for name in ("limit_events", "popularity", "dragon_tiger", "sector_daily"):
self.assertEqual(results[name]["state"], "published", results[name])
api = self.hub.api
limits = api.handle("/v1/limit-events", {"date": [TRADE_DATE]})
self.assertGreaterEqual(len(limits["data"]), 1)
self.assertEqual(limits["meta"]["tier"], "official")
pop = api.handle("/v1/popularity", {"date": [TRADE_DATE], "source": ["ths"]})
self.assertEqual(pop["data"][0]["source"], "ths")
lhb = api.handle("/v1/dragon-tiger", {"date": [TRADE_DATE]})
self.assertEqual(lhb["data"][0]["hm_name"], "测试游资")
# hub stores 万元→元
self.assertEqual(lhb["data"][0]["buy_amount"], 10_000_000.0)
sectors = api.handle("/v1/sectors", {"date": [TRADE_DATE], "family": ["ths"]})
self.assertEqual(sectors["data"][0]["family"], "ths")
status = api.handle("/v1/datasets/status", {"date": [TRADE_DATE]})
names = {item["dataset"] for item in status["data"]}
self.assertTrue({"limit_events", "popularity", "dragon_tiger", "sector_daily"} <= names)
def test_partial_sector_families_are_retried_instead_of_skipped(self) -> None:
first = self.hub.pipeline.run_eod_batch_e(TRADE_DATE)
self.assertEqual(first["sector_daily"]["state"], "published")
batch_id = self.hub.pipeline.active_batch("sector_daily", TRADE_DATE)
with self.hub.db.write() as connection:
connection.execute(
"DELETE FROM eod_sector_daily WHERE trade_date = ? AND batch_id = ? AND family = 'sw'",
(TRADE_DATE, batch_id),
)
self.assertFalse(self.hub.pipeline.extended_publication_complete("sector_daily", TRADE_DATE))
retried = self.hub.pipeline.run_eod_batch_e(TRADE_DATE)
self.assertEqual(retried["sector_daily"]["state"], "published")
self.assertTrue(self.hub.pipeline.extended_publication_complete("sector_daily", TRADE_DATE))
if __name__ == "__main__":
unittest.main()
+56 -1
View File
@@ -5,7 +5,7 @@ from datetime import date, timedelta
from datahub.coverage import calendar_coverage, point_coverage, published_range_coverage from datahub.coverage import calendar_coverage, point_coverage, published_range_coverage
from datahub.serving import V1API from datahub.serving import V1API
from tests.fixtures import TRADE_DATE, fake_transport from tests.fixtures import RAW, TRADE_DATE, fake_transport
from tests.test_pipeline import make_pipeline from tests.test_pipeline import make_pipeline
@@ -48,6 +48,8 @@ def history_transport(open_dates: list[str], extra_closed: list[str] | None = No
} }
) )
dated_apis = ("daily", "daily_basic", "adj_factor", "moneyflow", "stk_auction")
def transport(api_name, params, fields): def transport(api_name, params, fields):
if api_name == "trade_cal": if api_name == "trade_cal":
start = str(params.get("start_date") or "") start = str(params.get("start_date") or "")
@@ -66,6 +68,11 @@ def history_transport(open_dates: list[str], extra_closed: list[str] | None = No
if end: if end:
rows = [row for row in rows if row["trade_date"] <= end] rows = [row for row in rows if row["trade_date"] <= end]
return rows return rows
if api_name in dated_apis:
day = str(params.get("trade_date") or "")
if day not in open_set:
return []
return [{**row, "trade_date": day} for row in RAW.get(api_name) or []]
return fake_transport(api_name, params, fields) return fake_transport(api_name, params, fields)
return transport return transport
@@ -221,6 +228,54 @@ class HistoryBackfillTests(unittest.TestCase):
self.assertEqual(result["rows"], 1) self.assertEqual(result["rows"], 1)
self.assertEqual(calls["n"], before) self.assertEqual(calls["n"], before)
def test_daily_history_is_idempotent_and_covers_requested_days(self) -> None:
open_dates = consecutive_open_days(TRADE_DATE, 5)
pipe, db = make_pipeline(
quality={
"index_history_trading_days": 5,
"daily_history_trading_days": 5,
"calendar_start": open_dates[0],
}
)
pipe.adapter._transport = history_transport(open_dates)
first = pipe.backfill_history(TRADE_DATE, index_days=5, daily_days=5)
self.assertTrue(first["ok"])
self.assertEqual(first["daily"]["requested_days"], 5)
self.assertEqual(len(first["daily"]["published"]), 5)
pubs = db.fetchall("SELECT trade_date FROM publications WHERE dataset='daily'")
self.assertEqual(sorted(row["trade_date"] for row in pubs), open_dates)
for day in open_dates:
rows = db.fetchall(
"""
SELECT COUNT(*) AS n FROM eod_bars
WHERE trade_date = ? AND batch_id = (
SELECT active_batch FROM publications WHERE dataset='daily' AND trade_date = ?
)
""",
(day, day),
)
self.assertEqual(rows[0]["n"], 2)
second = pipe.backfill_daily_history(end_date=TRADE_DATE, trading_days=5)
self.assertTrue(second["ok"])
self.assertEqual(second["published"], [])
self.assertEqual(second["skipped"], open_dates)
def test_daily_row_ratio_ignores_later_ipos(self) -> None:
pipe, db = make_pipeline()
pipe.adapter._transport = history_transport([TRADE_DATE])
pipe.ingest_reference(TRADE_DATE, start=TRADE_DATE)
with db.write() as connection:
connection.execute(
"INSERT INTO stock_master(ts_code, symbol, name, list_status, list_date, updated_at) "
"VALUES (?,?,?,?,?,?)",
("688001.SH", "688001", "未来上市", "L", "20250101", "2024-09-02T00:00:00+08:00"),
)
self.assertEqual(pipe._listed_count(TRADE_DATE), 2)
result = pipe.run_eod_batch_a(TRADE_DATE)
self.assertEqual(pipe.eod_failures(result), [])
self.assertEqual(result["daily"]["state"], "published")
def test_coverage_helpers_point_and_calendar(self) -> None: def test_coverage_helpers_point_and_calendar(self) -> None:
pipe, db = make_pipeline() pipe, db = make_pipeline()
pipe.ingest_reference(TRADE_DATE) pipe.ingest_reference(TRADE_DATE)
+103
View File
@@ -0,0 +1,103 @@
from __future__ import annotations
import json
import unittest
from datahub.adapters.ifind import IfindAdapter
from datahub.serving import ApiError
from datahub.steward import steward_query
class _Resp:
def __init__(self, payload: dict, status: int = 200) -> None:
self.status = status
self._raw = json.dumps(payload).encode("utf-8")
def read(self):
return self._raw
def __enter__(self):
return self
def __exit__(self, *args):
return False
class IfindAdapterTests(unittest.TestCase):
def test_unconfigured_probe_does_not_dial_vendor(self) -> None:
calls = []
def urlopen(request, timeout=None):
calls.append(str(getattr(request, "full_url", request)))
raise AssertionError("should not dial")
adapter = IfindAdapter(urlopen=urlopen)
probe = adapter.probe()
self.assertEqual(probe["state"], "unconfigured")
self.assertFalse(probe["configured"])
self.assertEqual(calls, [])
def test_wencai_and_snapshots_go_to_ifind_http(self) -> None:
calls = []
def urlopen(request, timeout=None):
url = str(getattr(request, "full_url", request))
calls.append(url)
if url.endswith("get_access_token"):
return _Resp({"errorcode": 0, "data": {"access_token": "acc", "expired_time": ""}})
body = json.loads(request.data.decode("utf-8"))
if url.endswith("smart_stock_picking"):
self.assertEqual(body["searchstring"], "涨停")
return _Resp(
{
"errorcode": 0,
"tables": [
{
"thscode": ["000001.SZ"],
"table": {"涨停原因": ["重组"]},
}
],
}
)
if url.endswith("snap_shot"):
return _Resp(
{
"errorcode": 0,
"tables": [
{
"time": ["2024-09-02 09:21:00"],
"thscode": ["000001.SZ"],
"table": {"latest": [10.5], "preClose": [10]},
}
],
}
)
raise AssertionError(url)
adapter = IfindAdapter("refresh-token", urlopen=urlopen)
rows = adapter.fetch("wencai", {"query": "涨停"})
self.assertEqual(rows[0]["涨停原因"], "重组")
snaps = adapter.fetch(
"snapshots",
{
"codes": ["000001.SZ"],
"indicators": ["latest", "preClose"],
"start_time": "2024-09-02 09:15:00",
"end_time": "2024-09-02 09:22:00",
},
)
self.assertEqual(snaps[0]["latest"], 10.5)
self.assertTrue(any("quantapi.51ifind.com" in item for item in calls))
def test_steward_ifind_status_and_unconfigured_fetch(self) -> None:
class _Api:
ifind = IfindAdapter()
payload = steward_query(_Api(), {"api_name": "ifind_status", "params": {}})
self.assertFalse(payload["data"][0]["configured"])
with self.assertRaises(ApiError):
steward_query(_Api(), {"api_name": "ifind_wencai", "params": {"query": "涨停"}})
if __name__ == "__main__":
unittest.main()
+10 -1
View File
@@ -19,11 +19,20 @@ class LayoutTests(unittest.TestCase):
def test_reserved_adapters_present(self) -> None: def test_reserved_adapters_present(self) -> None:
from datahub.adapters import RESERVED from datahub.adapters import RESERVED
for name in ("eastmoney", "tencent", "ths", "xgb", "akshare", "ifind"): for name in ("ths", "xgb", "akshare"):
self.assertIn(name, RESERVED) self.assertIn(name, RESERVED)
probe = RESERVED[name].probe() probe = RESERVED[name].probe()
self.assertEqual(probe["state"], "reserved") self.assertEqual(probe["state"], "reserved")
self.assertFalse(probe["configured"]) self.assertFalse(probe["configured"])
for name in ("eastmoney", "tencent"):
self.assertIn(name, RESERVED)
probe = RESERVED[name].probe()
self.assertIn(probe["state"], {"ok", "empty", "error"})
self.assertTrue(probe["configured"])
self.assertIn("ifind", RESERVED)
ifind = RESERVED["ifind"].probe()
self.assertIn(ifind["state"], {"unconfigured", "ok", "empty", "error"})
self.assertEqual(ifind["configured"], ifind["state"] != "unconfigured")
if __name__ == "__main__": if __name__ == "__main__":
@@ -0,0 +1,292 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from datahub.adapters.base import AdapterError
from datahub.adapters.eastmoney import HIS_TRENDS_URL, TRENDS_URL, EastmoneyAdapter
from datahub.db import HubDB
from datahub.realtime_serve import fetch_intraday
from datahub.serving import ApiError, V1API
from datahub.timeutil import now_shanghai, yyyymmdd
class FakeEastmoney(EastmoneyAdapter):
def __init__(self) -> None:
super().__init__(timeout=2)
self.urls: list[str] = []
def _get_json(self, url, params, referer):
self.urls.append(f"{url}|{params.get('ndays')}")
if url == TRENDS_URL:
return {"data": {"name": "中国平安", "code": "601318", "preClose": 56.36, "trends": []}}
if url == HIS_TRENDS_URL:
return {
"data": {
"name": "中国平安",
"code": "601318",
"preClose": 55.8,
"trends": [
"2026-09-07 09:30,55.80,55.90,56.00,55.70,100,5580.00,55.900",
"2026-09-07 15:00,56.10,56.20,56.30,56.00,200,11240.00,56.150",
"2026-09-08 09:30,0,0,0,0,0,0.00,0",
],
}
}
raise AdapterError(f"unexpected url {url}")
class EastmoneyIntradayLookbackTests(unittest.TestCase):
def test_empty_today_uses_latest_available_session(self):
adapter = FakeEastmoney()
payload = adapter.fetch_intraday("601318.SH")
self.assertEqual(adapter.urls, [f"{TRENDS_URL}|1", f"{TRENDS_URL}|5", f"{HIS_TRENDS_URL}|5"])
self.assertEqual(payload["trade_date"], "2026-09-07")
self.assertEqual([point["time"] for point in payload["points"]], ["09:30", "15:00"])
self.assertEqual(payload["points"][0]["close"], 55.9)
def test_preferred_date_keeps_that_session(self):
adapter = FakeEastmoney()
payload = adapter.fetch_intraday("601318.SH", "20260907")
self.assertEqual(payload["trade_date"], "2026-09-07")
self.assertEqual(len(payload["points"]), 2)
class IntradayLkgTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
self.db = HubDB(Path(self.tmp.name) / "hub.db")
def tearDown(self) -> None:
self.tmp.cleanup()
def test_source_failure_returns_last_known_good(self):
from datahub.realtime_serve import _envelope, _write_cache
payload = _envelope(
{
"entity_type": "stock",
"ts_code": "601318.SH",
"trade_date": "2026-09-07",
"previous_close": 55.8,
"points": [{"date": "2026-09-07", "time": "09:30", "close": 55.9}],
},
{
"tier": "provisional",
"trade_date": "20260907",
"source": "eastmoney:trends2",
"stale": False,
},
)
_write_cache(self.db, "intraday:601318.SH:today", payload, 20, "eastmoney:trends2")
self.db.execute(
"UPDATE rt_cache SET expires_at = ? WHERE cache_key = ?",
("2000-01-01T00:00:00+08:00", "intraday:601318.SH:today"),
)
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_intraday.side_effect = AdapterError("down")
recovered = fetch_intraday(self.db, "601318.SH")
self.assertTrue(recovered["meta"]["stale"])
self.assertEqual(recovered["data"]["points"][0]["close"], 55.9)
def test_source_failure_without_lkg_raises(self):
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_intraday.side_effect = AdapterError("down")
with self.assertRaises(Exception) as ctx:
fetch_intraday(self.db, "000001.SZ")
self.assertIn("intraday unavailable", str(ctx.exception))
class ServingIntradayDateTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
self.db = HubDB(Path(self.tmp.name) / "hub.db")
self.api = V1API(self.db, pipeline=None, settings=None)
def tearDown(self) -> None:
self.tmp.cleanup()
def _assert_usable_intraday(self, payload: dict) -> None:
data = payload["data"]
points = [point for point in data.get("points") or [] if float(point.get("close") or 0) > 0]
self.assertGreaterEqual(len(points), 1)
self.assertTrue(str(data.get("trade_date") or ""))
self.assertFalse((payload.get("meta") or {}).get("stale"))
def test_serving_omitted_or_empty_date_uses_today_and_returns_points(self) -> None:
today = yyyymmdd(now_shanghai())
omitted = self.api.handle("/v1/intraday/points", {"code": ["601318"]})
empty = self.api.handle("/v1/intraday/points", {"code": ["601318"], "date": [""]})
explicit = self.api.handle("/v1/intraday/points", {"code": ["601318"], "date": [today]})
self._assert_usable_intraday(omitted)
self._assert_usable_intraday(empty)
self._assert_usable_intraday(explicit)
self.assertEqual(omitted["data"]["trade_date"], empty["data"]["trade_date"])
self.assertEqual(explicit["data"]["trade_date"], omitted["data"]["trade_date"])
def test_serving_normalizes_empty_date_to_today_and_keeps_history(self) -> None:
today = yyyymmdd(now_shanghai())
captured: list[str] = []
def fake_fetch(db, code, date=""):
captured.append(date)
return {
"schema_version": 1,
"data": {
"trade_date": f"{date[:4]}-{date[4:6]}-{date[6:8]}",
"points": [{"date": f"{date[:4]}-{date[4:6]}-{date[6:8]}", "time": "09:30", "close": 55.9}],
},
"meta": {"stale": False, "trade_date": date},
}
with patch("datahub.realtime_serve.fetch_intraday", side_effect=fake_fetch):
omitted = self.api.handle("/v1/intraday/points", {"code": ["601318"]})
empty = self.api.handle("/v1/intraday/points", {"code": ["601318"], "date": [" "]})
history = self.api.handle("/v1/intraday/points", {"code": ["601318"], "date": ["20260907"]})
self.assertEqual(captured, [today, today, "20260907"])
self.assertEqual(omitted["data"]["trade_date"], f"{today[:4]}-{today[4:6]}-{today[6:8]}")
self.assertEqual(empty["data"]["trade_date"], omitted["data"]["trade_date"])
self.assertEqual(history["data"]["trade_date"], "2026-09-07")
def test_serving_invalid_date_is_invalid_argument(self) -> None:
with self.assertRaises(ApiError) as ctx:
self.api.handle("/v1/intraday/points", {"code": ["601318"], "date": ["not-a-date"]})
self.assertEqual(ctx.exception.code, "INVALID_ARGUMENT")
self.assertIn("invalid trade_date", ctx.exception.message)
def test_serving_missing_code_is_invalid_argument(self) -> None:
with self.assertRaises(ApiError) as ctx:
self.api.handle("/v1/intraday/points", {"date": [yyyymmdd(now_shanghai())]})
self.assertEqual(ctx.exception.code, "INVALID_ARGUMENT")
self.assertIn("code is required", ctx.exception.message)
def test_serving_no_data_keeps_source_unavailable(self) -> None:
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_intraday.side_effect = AdapterError("No intraday chart data returned")
with self.assertRaises(ApiError) as ctx:
self.api.handle("/v1/intraday/points", {"code": ["000001"]})
self.assertEqual(ctx.exception.code, "SOURCE_UNAVAILABLE")
self.assertIn("intraday unavailable", ctx.exception.message)
class MarketQuotesTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
self.db = HubDB(Path(self.tmp.name) / "hub.db")
self.api = V1API(self.db, None, None) # type: ignore[arg-type]
def tearDown(self) -> None:
self.tmp.cleanup()
def test_empty_codes_returns_full_market_snapshot(self) -> None:
rows = [
{
"ts_code": f"{600000 + index:06d}.SH",
"name": f"股票{index}",
"pre_close": 10.0,
"close": 10.2,
"open": 10.1,
"high": 10.3,
"low": 10.0,
"vol": 1000,
"amount": 2000000,
}
for index in range(220)
]
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_market_quotes.return_value = rows
omitted = self.api.handle("/v1/quotes/latest", {})
empty = self.api.handle("/v1/quotes/latest", {"codes": [""]})
self.assertEqual(len(omitted["data"]), 220)
self.assertEqual(omitted["meta"]["scope"], "market")
self.assertEqual(omitted["meta"]["source"], "eastmoney:clist")
self.assertEqual(len(empty["data"]), 220)
def test_explicit_codes_still_use_named_quote_path(self) -> None:
named = [
{
"ts_code": "600000.SH",
"name": "浦发银行",
"price": 10.2,
"previous_close": 10.0,
}
]
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_quotes.return_value = named
payload = self.api.handle("/v1/quotes/latest", {"codes": ["600000.SH"]})
mocked.return_value.fetch_market_quotes.assert_not_called()
self.assertEqual(payload["data"][0]["ts_code"], "600000.SH")
self.assertEqual(payload["meta"]["source"], "eastmoney:ulist")
def test_named_quotes_page_beyond_sixty_codes(self) -> None:
codes = [f"{index:06d}.SZ" for index in range(70)]
def fake_fetch(chunk):
return [{"ts_code": code, "close": 10, "pre_close": 9} for code in chunk]
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
mocked.return_value.fetch_quotes.side_effect = fake_fetch
payload = self.api.handle("/v1/quotes/latest", {"codes": [",".join(codes)]})
self.assertEqual(mocked.return_value.fetch_quotes.call_count, 2)
self.assertEqual(len(payload["data"]), 70)
def test_named_quotes_fail_over_to_tencent(self) -> None:
named = [
{
"ts_code": "000737.SZ",
"name": "北方铜业",
"close": 12.3,
"pre_close": 11.2,
}
]
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
"datahub.realtime_serve.TencentAdapter"
) as tencent:
eastmoney.return_value.fetch_quotes.side_effect = AdapterError("HTTP 503")
tencent.return_value.fetch_quotes.return_value = named
payload = self.api.handle("/v1/quotes/latest", {"codes": ["000737.SZ"]})
self.assertEqual(payload["data"][0]["ts_code"], "000737.SZ")
self.assertEqual(payload["meta"]["source"], "tencent:qt")
self.assertTrue(payload["meta"]["failover"])
self.assertFalse(payload["meta"]["stale"])
def test_both_quote_sources_return_last_known_good(self) -> None:
from datahub.realtime_serve import _envelope, _write_cache
cache_key = "quotes:placeholder:1"
payload = _envelope(
[{"ts_code": "000737.SZ", "close": 12.3, "pre_close": 11.2}],
{"source": "eastmoney:ulist", "stale": False, "trade_date": "20260908"},
)
_write_cache(self.db, cache_key, payload, 60, "eastmoney:ulist")
self.db.execute(
"UPDATE rt_cache SET expires_at = ? WHERE cache_key = ?",
("2000-01-01T00:00:00+08:00", cache_key),
)
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
"datahub.realtime_serve.TencentAdapter"
) as tencent, patch("datahub.realtime_serve.hashlib.sha1") as sha1:
eastmoney.return_value.fetch_quotes.side_effect = AdapterError("HTTP 503")
tencent.return_value.fetch_quotes.side_effect = AdapterError("timeout")
sha1.return_value.hexdigest.return_value = "placeholder"
recovered = self.api.handle("/v1/quotes/latest", {"codes": ["000737.SZ"]})
self.assertTrue(recovered["meta"]["stale"])
self.assertIn("真实快照", recovered["meta"]["delay_notice"])
self.assertEqual(recovered["data"][0]["close"], 12.3)
self.assertNotEqual(recovered["data"][0]["close"], 0)
def test_market_unavailable_stays_source_error(self) -> None:
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
"datahub.realtime_serve.TencentAdapter"
) as tencent:
eastmoney.return_value.fetch_market_quotes.side_effect = AdapterError("too small")
tencent.return_value.fetch_quotes.side_effect = AdapterError("empty master")
with self.assertRaises(ApiError) as ctx:
self.api.handle("/v1/quotes/latest", {})
self.assertEqual(ctx.exception.code, "SOURCE_UNAVAILABLE")
if __name__ == "__main__":
unittest.main()
+13 -1
View File
@@ -151,7 +151,19 @@ class RevisionReviewTests(unittest.TestCase):
return None if row is None else row["turnover_rate"] return None if row is None else row["turnover_rate"]
def _batch_ids(self, db: HubDB, day: str) -> set[str]: def _batch_ids(self, db: HubDB, day: str) -> set[str]:
return {str(row["batch_id"]) for row in db.fetchall("SELECT batch_id FROM batches WHERE trade_date=?", (day,))} # HEL-478/HEL-463 integration: extended soft datasets (limit_events,
# dragon_tiger, sector_daily, popularity) publish on their own schedule
# inside the same tick and are outside the valuation review boundary.
# Scope assertions to the A-group + stocks boundary the review republishes.
boundary = ("daily", "valuation", "moneyflow", "auction", "stocks")
placeholders = ",".join("?" for _ in boundary)
return {
str(row["batch_id"])
for row in db.fetchall(
f"SELECT batch_id FROM batches WHERE trade_date=? AND dataset IN ({placeholders})",
(day, *boundary),
)
}
def test_no_change_does_not_create_a_new_batch(self) -> None: def test_no_change_does_not_create_a_new_batch(self) -> None:
tmp, db, transport, pipe, sched = make_revision_env() tmp, db, transport, pipe, sched = make_revision_env()
+161
View File
@@ -0,0 +1,161 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from datahub.adapters.tushare import TushareAdapter
from datahub.crypto import SecretVault
from datahub.hub import Hub
from datahub.settings import Settings
from tests.fixtures import TRADE_DATE, fake_transport
class StewardQueryTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
settings = Settings(
host="127.0.0.1",
port=0,
encryption_key=SecretVault.generate_key(),
api_token="k" * 32,
admin_password="StartPass1",
tushare_token="tushare-secret-token-xyz",
db_path=Path(self.tmp.name) / "hub.db",
backup_dir=Path(self.tmp.name) / "backups",
scheduler_enabled=False,
quality={"daily_row_ratio": 0.5, "null_rate_max": 0.5, "list_limit_default": 5000, "list_limit_max": 5000},
)
adapter = TushareAdapter("tushare-secret-token-xyz", transport=fake_transport)
self.hub = Hub(settings, adapter=adapter)
self.hub.pipeline.ingest_reference(TRADE_DATE)
for dataset in ("daily", "valuation", "moneyflow", "auction", "index_daily"):
self.hub.pipeline.run_dataset(dataset, TRADE_DATE)
def tearDown(self) -> None:
self.hub.stop()
self.tmp.cleanup()
def test_published_daily_is_tushare_native(self) -> None:
payload = self.hub.api.query_api(
{"api_name": "daily", "params": {"trade_date": TRADE_DATE}, "fields": "ts_code,close,vol,amount"}
)
rows = payload["data"]
by_code = {row["ts_code"]: row for row in rows}
self.assertEqual(by_code["600000.SH"]["vol"], 1000.0)
self.assertEqual(by_code["600000.SH"]["amount"], 2000.0)
self.assertEqual(payload["meta"]["row_shape"], "tushare")
def test_live_stk_limit_uses_internal_tushare(self) -> None:
payload = self.hub.api.query_api(
{"api_name": "stk_limit", "params": {"trade_date": TRADE_DATE}, "fields": "ts_code,up_limit,down_limit"}
)
self.assertEqual(payload["meta"]["source"], "tushare")
self.assertEqual(payload["data"][0]["ts_code"], "600000.SH")
def test_stock_filter_missing_from_active_snapshot_falls_back_inside_hub(self) -> None:
calls = []
def transport(api_name, params, fields):
calls.append((api_name, dict(params)))
if api_name == "stock_basic" and params.get("list_status") == "D":
return [
{
"ts_code": "000627.SZ",
"symbol": "000627",
"name": "退市天茂",
"list_status": "D",
"list_date": "19961112",
}
]
return fake_transport(api_name, params, fields)
self.hub.pipeline.adapter._transport = transport
payload = self.hub.api.query_api(
{
"api_name": "stock_basic",
"params": {"list_status": "D"},
"fields": "ts_code,name,list_status,list_date",
}
)
self.assertEqual(payload["meta"]["source"], "tushare")
self.assertEqual(payload["data"][0]["list_status"], "D")
self.assertIn(("stock_basic", {"list_status": "D"}), calls)
def test_rt_sw_k_is_blocked(self) -> None:
from datahub.serving import ApiError
with self.assertRaises(ApiError):
self.hub.api.query_api({"api_name": "rt_sw_k", "params": {"ts_code": "801074.SI"}})
def test_missing_published_sw_family_falls_back_inside_hub(self) -> None:
self.hub.pipeline.run_eod_batch_e(TRADE_DATE)
batch_id = self.hub.pipeline.active_batch("sector_daily", TRADE_DATE)
with self.hub.db.write() as connection:
connection.execute(
"DELETE FROM eod_sector_daily WHERE trade_date = ? AND batch_id = ? AND family = 'sw'",
(TRADE_DATE, batch_id),
)
payload = self.hub.api.query_api(
{
"api_name": "sw_daily",
"params": {"ts_code": "801780.SI", "trade_date": TRADE_DATE},
"fields": "ts_code,trade_date,name,pct_change",
}
)
self.assertEqual(payload["meta"]["source"], "tushare")
self.assertEqual(payload["data"][0]["ts_code"], "801780.SI")
def test_rt_k_uses_free_quotes_not_tushare(self) -> None:
quotes = [
{
"ts_code": "600000.SH",
"name": "浦发银行",
"close": 10.2,
"pre_close": 10.0,
"open": 10.1,
"high": 10.3,
"low": 9.9,
"vol": 1000,
"amount": 2000000,
}
]
with patch("datahub.steward.fetch_quotes", return_value={"data": quotes, "meta": {"source": "eastmoney:ulist", "stale": False}}):
payload = self.hub.api.query_api({"api_name": "rt_k", "params": {"ts_code": "600000.SH"}})
self.assertEqual(payload["data"][0]["close"], 10.2)
self.assertEqual(payload["meta"]["source"], "eastmoney:ulist")
def test_shenwan_quote_uses_eastmoney_90_prefix(self) -> None:
from datahub.adapters.eastmoney import EastmoneyAdapter
with patch.object(EastmoneyAdapter, "_get_json") as get_json:
get_json.return_value = {
"data": {
"diff": [
{
"f12": "801074",
"f14": "工业金属",
"f2": 1234.5,
"f3": 2.88,
"f18": 1200,
"f17": 1205,
"f15": 1240,
"f16": 1198,
"f6": 1,
"f124": 1757319000,
}
]
}
}
quote = EastmoneyAdapter().fetch_shenwan_quote("801074.SI")
self.assertEqual(quote["source"], "eastmoney_sw")
self.assertAlmostEqual(quote["change"], 2.88)
self.assertEqual(get_json.call_args.args[1]["secids"], "90.801074")
if __name__ == "__main__":
unittest.main()