fix(HEL-494): 盘中双免费源故障切换并禁止问天假0覆盖
主源东财失败后自动改走腾讯行情，成功结果写入缓存；两源都失败时返回最近真实快照并标明延迟，不再显示假0。 Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
总工
co-authored by
Cursor
multica-agent
Cursor
multica-agent
parent
b5d65ecb41
commit
ef13d6feb5
@@ -1,7 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
@@ -10,6 +9,8 @@ from datahub.adapters.base import AdapterError, MarketAdapter
|
||||
from datahub.numbers import finite_number, round4
|
||||
|
||||
TENCENT_INDEX_URL = "https://qt.gtimg.cn/q=sh000001,sz399001,sz399006"
|
||||
TENCENT_QUOTE_URL = "https://qt.gtimg.cn/q="
|
||||
TENCENT_QUOTE_BATCH = 80
|
||||
BROWSER_UA = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||
@@ -45,6 +46,16 @@ class TencentAdapter(MarketAdapter):
|
||||
def fetch(self, dataset: str, params: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
if dataset in {"indexes_quotes", "index_quotes"}:
|
||||
return self.fetch_indices()
|
||||
if dataset in {"quotes", "quotes_latest"}:
|
||||
codes = params.get("codes") or []
|
||||
if isinstance(codes, str):
|
||||
codes = [item.strip() for item in codes.split(",") if item.strip()]
|
||||
return self.fetch_quotes(list(codes))
|
||||
if dataset in {"quotes_market", "market_quotes"}:
|
||||
codes = params.get("codes") or []
|
||||
if isinstance(codes, str):
|
||||
codes = [item.strip() for item in codes.split(",") if item.strip()]
|
||||
return self.fetch_quotes(list(codes))
|
||||
raise AdapterError(f"{self.name} unsupported dataset: {dataset}")
|
||||
|
||||
def normalize(self, dataset: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
@@ -97,3 +108,111 @@ class TencentAdapter(MarketAdapter):
|
||||
if len(result) != 3:
|
||||
raise AdapterError(f"Tencent returned {len(result)}/3 indices")
|
||||
return result
|
||||
|
||||
def fetch_quotes(self, codes: list[str]) -> list[dict[str, Any]]:
|
||||
symbols: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw in codes:
|
||||
symbol = _tencent_symbol(str(raw or ""))
|
||||
if not symbol or symbol in seen:
|
||||
continue
|
||||
seen.add(symbol)
|
||||
symbols.append(symbol)
|
||||
if not symbols:
|
||||
return []
|
||||
result: list[dict[str, Any]] = []
|
||||
errors: list[str] = []
|
||||
for index in range(0, len(symbols), TENCENT_QUOTE_BATCH):
|
||||
batch = symbols[index:index + TENCENT_QUOTE_BATCH]
|
||||
try:
|
||||
raw = self._get_text(f"{TENCENT_QUOTE_URL}{','.join(batch)}")
|
||||
except AdapterError as exc:
|
||||
errors.append(str(exc))
|
||||
continue
|
||||
for line in raw.splitlines():
|
||||
quote = _parse_tencent_stock_quote(line)
|
||||
if quote:
|
||||
result.append(quote)
|
||||
if not result:
|
||||
detail = f"；{'; '.join(errors[:3])}" if errors else ""
|
||||
raise AdapterError(f"Tencent quotes empty{detail}")
|
||||
return result
|
||||
|
||||
def _get_text(self, url: str) -> str:
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": BROWSER_UA, "Referer": "https://gu.qq.com/"},
|
||||
method="GET",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=self.timeout) as response:
|
||||
return response.read().decode("gb18030", errors="ignore")
|
||||
except Exception as exc:
|
||||
raise AdapterError(f"tencent request failed: {exc}") from exc
|
||||
|
||||
|
||||
def _tencent_symbol(code: str) -> str:
|
||||
raw = str(code or "").strip().upper()
|
||||
if not raw:
|
||||
return ""
|
||||
symbol = raw.split(".")[0]
|
||||
if not symbol.isdigit() or len(symbol) != 6:
|
||||
return ""
|
||||
if raw.endswith(".SH") or symbol.startswith(("5", "6", "9")):
|
||||
return f"sh{symbol}"
|
||||
if raw.endswith(".BJ") or symbol.startswith(("4", "8")):
|
||||
return f"bj{symbol}"
|
||||
return f"sz{symbol}"
|
||||
|
||||
|
||||
def _parse_tencent_stock_quote(line: str) -> dict[str, Any] | None:
|
||||
if '="' not in line:
|
||||
return None
|
||||
prefix, payload = line.split('="', 1)
|
||||
fields = payload.rsplit('";', 1)[0].split("~")
|
||||
if len(fields) < 38:
|
||||
return None
|
||||
symbol = str(fields[2] or "")
|
||||
if not symbol.isdigit() or len(symbol) != 6:
|
||||
return None
|
||||
close = round4(finite_number(fields[3]))
|
||||
previous = round4(finite_number(fields[4]))
|
||||
if not close or not previous or close <= 0 or previous <= 0:
|
||||
return None
|
||||
marker = prefix.lower()
|
||||
if "sh" in marker:
|
||||
ts_code = f"{symbol}.SH"
|
||||
elif "bj" in marker:
|
||||
ts_code = f"{symbol}.BJ"
|
||||
else:
|
||||
ts_code = f"{symbol}.SZ"
|
||||
quote_stamp = ""
|
||||
quote_date = ""
|
||||
epoch = 0
|
||||
try:
|
||||
parsed = datetime.strptime(fields[30], "%Y%m%d%H%M%S")
|
||||
quote_date = parsed.strftime("%Y%m%d")
|
||||
epoch = int(parsed.timestamp())
|
||||
quote_stamp = parsed.astimezone().isoformat(timespec="seconds")
|
||||
except ValueError:
|
||||
pass
|
||||
return {
|
||||
"ts_code": ts_code,
|
||||
"name": fields[1] or symbol,
|
||||
"price": close,
|
||||
"close": close,
|
||||
"pct_chg": round4(finite_number(fields[32])),
|
||||
"change_amount": round4(finite_number(fields[31])),
|
||||
"open": round4(finite_number(fields[5])),
|
||||
"high": round4(finite_number(fields[33])),
|
||||
"low": round4(finite_number(fields[34])),
|
||||
"pre_close": previous,
|
||||
"previous_close": previous,
|
||||
"volume": round4(finite_number(fields[6]) * 100),
|
||||
"vol": round4(finite_number(fields[6]) * 100),
|
||||
"amount": round4(finite_number(fields[37]) * 10000),
|
||||
"quote_date": quote_date,
|
||||
"quote_time_epoch": epoch,
|
||||
"quote_time": quote_stamp,
|
||||
"source": "tencent_qt",
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import time
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from datahub.adapters.base import AdapterError
|
||||
from datahub.adapters.eastmoney import EastmoneyAdapter
|
||||
from datahub.adapters.tencent import TencentAdapter
|
||||
from datahub.codes import resolve_code
|
||||
@@ -71,24 +72,32 @@ def fetch_market_quotes(db: HubDB) -> dict[str, Any]:
|
||||
cached = _read_cache(db, cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
adapter = EastmoneyAdapter()
|
||||
errors: list[str] = []
|
||||
rows: list[dict[str, Any]] = []
|
||||
source = ""
|
||||
try:
|
||||
rows = adapter.fetch_market_quotes()
|
||||
rows = EastmoneyAdapter().fetch_market_quotes()
|
||||
source = "eastmoney:clist"
|
||||
except Exception as exc:
|
||||
raise RealtimeApiError("SOURCE_UNAVAILABLE", f"market quotes unavailable: {exc}") from exc
|
||||
payload = _envelope(
|
||||
rows,
|
||||
{
|
||||
"tier": "provisional",
|
||||
"trade_date": yyyymmdd(now_shanghai()),
|
||||
"source": source,
|
||||
"stale": False,
|
||||
"staleness_seconds": 0,
|
||||
"published_at": isoformat(now_shanghai()),
|
||||
"scope": "market",
|
||||
},
|
||||
)
|
||||
errors.append(f"eastmoney:{exc}")
|
||||
try:
|
||||
listed = _listed_ts_codes(db)
|
||||
if not listed:
|
||||
raise AdapterError("no local stock master for tencent market snapshot")
|
||||
rows = TencentAdapter().fetch_quotes(listed)
|
||||
if len(rows) < 200:
|
||||
raise AdapterError(f"Tencent market snapshot too small: {len(rows)}")
|
||||
source = "tencent:qt"
|
||||
except Exception as backup_exc:
|
||||
errors.append(f"tencent:{backup_exc}")
|
||||
recovered = _load_quotes_lkg(db, cache_key)
|
||||
if recovered is not None:
|
||||
return recovered
|
||||
raise RealtimeApiError(
|
||||
"SOURCE_UNAVAILABLE",
|
||||
"market quotes unavailable: " + "；".join(errors),
|
||||
) from backup_exc
|
||||
payload = _quote_payload(rows, source, scope="market")
|
||||
_write_cache(db, cache_key, payload, QUOTE_TTL, source)
|
||||
return payload
|
||||
|
||||
@@ -110,29 +119,98 @@ def fetch_quotes(db: HubDB, codes: list[str]) -> dict[str, Any]:
|
||||
cached = _read_cache(db, cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
adapter = EastmoneyAdapter()
|
||||
minimum = max(1, int(len(resolved) * 0.5))
|
||||
errors: list[str] = []
|
||||
rows: list[dict[str, Any]] = []
|
||||
source = ""
|
||||
try:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for index in range(0, len(resolved), QUOTE_BATCH):
|
||||
rows.extend(adapter.fetch_quotes(resolved[index:index + QUOTE_BATCH]))
|
||||
rows = _eastmoney_named_quotes(resolved)
|
||||
if len(rows) < minimum:
|
||||
raise AdapterError(f"Eastmoney named quotes too small: {len(rows)}/{len(resolved)}")
|
||||
source = "eastmoney:ulist"
|
||||
except Exception as exc:
|
||||
raise RealtimeApiError("SOURCE_UNAVAILABLE", f"quotes unavailable: {exc}") from exc
|
||||
payload = _envelope(
|
||||
rows,
|
||||
{
|
||||
"tier": "provisional",
|
||||
"trade_date": yyyymmdd(now_shanghai()),
|
||||
"source": source,
|
||||
"stale": False,
|
||||
"staleness_seconds": 0,
|
||||
"published_at": isoformat(now_shanghai()),
|
||||
},
|
||||
)
|
||||
errors.append(f"eastmoney:{exc}")
|
||||
try:
|
||||
rows = TencentAdapter().fetch_quotes(resolved)
|
||||
if len(rows) < minimum:
|
||||
raise AdapterError(f"Tencent named quotes too small: {len(rows)}/{len(resolved)}")
|
||||
source = "tencent:qt"
|
||||
except Exception as backup_exc:
|
||||
errors.append(f"tencent:{backup_exc}")
|
||||
recovered = _load_quotes_lkg(db, cache_key)
|
||||
if recovered is not None:
|
||||
return recovered
|
||||
raise RealtimeApiError(
|
||||
"SOURCE_UNAVAILABLE",
|
||||
"quotes unavailable: " + "；".join(errors),
|
||||
) from backup_exc
|
||||
payload = _quote_payload(rows, source)
|
||||
_write_cache(db, cache_key, payload, QUOTE_TTL, source)
|
||||
return payload
|
||||
|
||||
|
||||
def _eastmoney_named_quotes(codes: list[str]) -> list[dict[str, Any]]:
|
||||
adapter = EastmoneyAdapter()
|
||||
rows: list[dict[str, Any]] = []
|
||||
for index in range(0, len(codes), QUOTE_BATCH):
|
||||
rows.extend(adapter.fetch_quotes(codes[index:index + QUOTE_BATCH]))
|
||||
return rows
|
||||
|
||||
|
||||
def _listed_ts_codes(db: HubDB) -> list[str]:
|
||||
try:
|
||||
rows = db.fetchall(
|
||||
"SELECT ts_code FROM stock_master WHERE list_status = 'L' ORDER BY ts_code"
|
||||
)
|
||||
except Exception:
|
||||
return []
|
||||
return [str(row.get("ts_code") or "") for row in rows if row.get("ts_code")]
|
||||
|
||||
|
||||
def _quote_payload(
|
||||
rows: list[dict[str, Any]],
|
||||
source: str,
|
||||
scope: str = "",
|
||||
) -> dict[str, Any]:
|
||||
meta: dict[str, Any] = {
|
||||
"tier": "provisional",
|
||||
"trade_date": yyyymmdd(now_shanghai()),
|
||||
"source": source,
|
||||
"stale": False,
|
||||
"staleness_seconds": 0,
|
||||
"published_at": isoformat(now_shanghai()),
|
||||
"failover": source.startswith("tencent"),
|
||||
"delay_notice": "",
|
||||
}
|
||||
if scope:
|
||||
meta["scope"] = scope
|
||||
return _envelope(rows, meta)
|
||||
|
||||
|
||||
def _load_quotes_lkg(db: HubDB, cache_key: str) -> dict[str, Any] | None:
|
||||
store = LastKnownGood(db)
|
||||
item = store.load(cache_key)
|
||||
payload = item.get("payload") if item else None
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
data = payload.get("data")
|
||||
if not isinstance(data, list) or not data:
|
||||
return None
|
||||
stamped = dict(payload)
|
||||
meta = dict(stamped.get("meta") or {})
|
||||
stored = str((item or {}).get("stored_at") or "")
|
||||
try:
|
||||
age = max(0, int(time.time() - datetime.fromisoformat(stored).timestamp()))
|
||||
except Exception:
|
||||
age = QUOTE_TTL
|
||||
meta["stale"] = True
|
||||
meta["staleness_seconds"] = age
|
||||
meta["delay_notice"] = f"主备免费行情均暂不可用，显示 {age} 秒前的真实快照"
|
||||
meta["lkg_source"] = str((item or {}).get("source") or meta.get("source") or "")
|
||||
stamped["meta"] = meta
|
||||
return stamped
|
||||
|
||||
|
||||
def fetch_intraday(db: HubDB, code: str, date: str = "") -> dict[str, Any]:
|
||||
ts_code = resolve_code(db, code) or _guess_ts_code(code)
|
||||
if not ts_code:
|
||||
|
||||
@@ -232,9 +232,57 @@ class MarketQuotesTests(unittest.TestCase):
|
||||
self.assertEqual(mocked.return_value.fetch_quotes.call_count, 2)
|
||||
self.assertEqual(len(payload["data"]), 70)
|
||||
|
||||
def test_named_quotes_fail_over_to_tencent(self) -> None:
|
||||
named = [
|
||||
{
|
||||
"ts_code": "000737.SZ",
|
||||
"name": "北方铜业",
|
||||
"close": 12.3,
|
||||
"pre_close": 11.2,
|
||||
}
|
||||
]
|
||||
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
|
||||
"datahub.realtime_serve.TencentAdapter"
|
||||
) as tencent:
|
||||
eastmoney.return_value.fetch_quotes.side_effect = AdapterError("HTTP 503")
|
||||
tencent.return_value.fetch_quotes.return_value = named
|
||||
payload = self.api.handle("/v1/quotes/latest", {"codes": ["000737.SZ"]})
|
||||
self.assertEqual(payload["data"][0]["ts_code"], "000737.SZ")
|
||||
self.assertEqual(payload["meta"]["source"], "tencent:qt")
|
||||
self.assertTrue(payload["meta"]["failover"])
|
||||
self.assertFalse(payload["meta"]["stale"])
|
||||
|
||||
def test_both_quote_sources_return_last_known_good(self) -> None:
|
||||
from datahub.realtime_serve import _envelope, _write_cache
|
||||
|
||||
cache_key = "quotes:placeholder:1"
|
||||
payload = _envelope(
|
||||
[{"ts_code": "000737.SZ", "close": 12.3, "pre_close": 11.2}],
|
||||
{"source": "eastmoney:ulist", "stale": False, "trade_date": "20260908"},
|
||||
)
|
||||
_write_cache(self.db, cache_key, payload, 60, "eastmoney:ulist")
|
||||
self.db.execute(
|
||||
"UPDATE rt_cache SET expires_at = ? WHERE cache_key = ?",
|
||||
("2000-01-01T00:00:00+08:00", cache_key),
|
||||
)
|
||||
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
|
||||
"datahub.realtime_serve.TencentAdapter"
|
||||
) as tencent, patch("datahub.realtime_serve.hashlib.sha1") as sha1:
|
||||
eastmoney.return_value.fetch_quotes.side_effect = AdapterError("HTTP 503")
|
||||
tencent.return_value.fetch_quotes.side_effect = AdapterError("timeout")
|
||||
sha1.return_value.hexdigest.return_value = "placeholder"
|
||||
recovered = self.api.handle("/v1/quotes/latest", {"codes": ["000737.SZ"]})
|
||||
self.assertTrue(recovered["meta"]["stale"])
|
||||
self.assertIn("真实快照", recovered["meta"]["delay_notice"])
|
||||
self.assertEqual(recovered["data"][0]["close"], 12.3)
|
||||
self.assertNotEqual(recovered["data"][0]["close"], 0)
|
||||
|
||||
def test_market_unavailable_stays_source_error(self) -> None:
|
||||
with patch("datahub.realtime_serve.EastmoneyAdapter") as mocked:
|
||||
mocked.return_value.fetch_market_quotes.side_effect = AdapterError("too small")
|
||||
with patch("datahub.realtime_serve.EastmoneyAdapter") as eastmoney, patch(
|
||||
"datahub.realtime_serve.TencentAdapter"
|
||||
) as tencent:
|
||||
eastmoney.return_value.fetch_market_quotes.side_effect = AdapterError("too small")
|
||||
tencent.return_value.fetch_quotes.side_effect = AdapterError("empty master")
|
||||
with self.assertRaises(ApiError) as ctx:
|
||||
self.api.handle("/v1/quotes/latest", {})
|
||||
self.assertEqual(ctx.exception.code, "SOURCE_UNAVAILABLE")
|
||||
|
||||
Reference in New Issue
Block a user