Skip to content

Commit d57961d

Browse files
authored
Merge pull request #147 from duanyytop/refactor/cleanup
refactor: cleanup code
2 parents 674ad64 + 4d26832 commit d57961d

File tree

16 files changed

+1074
-835
lines changed

16 files changed

+1074
-835
lines changed

CLAUDE.md

Lines changed: 21 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ agents-radar is a daily digest generator for the AI open-source ecosystem. A Git
88

99
```bash
1010
pnpm start # run the full digest locally
11+
pnpm test # vitest (unit tests)
1112
pnpm typecheck # tsc --noEmit
1213
pnpm lint # ESLint
1314
pnpm lint:fix # ESLint --fix
@@ -44,24 +45,29 @@ The pipeline runs in four sequential phases, each implemented as a named async f
4445
1. **`fetchAllData`** — all network I/O in parallel: GitHub API (issues/PRs/releases) for 17 repos, Claude Code Skills, Anthropic/OpenAI sitemaps, GitHub Trending HTML + Search API, Hacker News Algolia API.
4546
2. **`generateSummaries`** — per-repo LLM calls, all in parallel, rate-limited to 5 concurrent requests by a queue in `src/report.ts`.
4647
3. **Comparisons** — two LLM calls: cross-tool CLI comparison and OpenClaw cross-ecosystem comparison.
47-
4. **Save phase**`buildCliReportContent` / `buildOpenclawReportContent` build Markdown strings; `saveWebReport` / `saveTrendingReport` / `saveHnReport` call LLM + write file + create GitHub Issue.
48+
4. **Save phase**`buildCliReportContent` / `buildOpenclawReportContent` (in `src/report-builders.ts`) build Markdown strings; `saveWebReport` / `saveTrendingReport` / `saveHnReport` (in `src/report-savers.ts`) call LLM + write file + create GitHub Issue.
4849

4950
## Source files
5051

5152
| File | Responsibility |
5253
|------|---------------|
5354
| `src/index.ts` | Orchestration: repo config, phase functions, `main()` |
55+
| `src/i18n.ts` | Centralized bilingual strings: `Lang` type, report titles, issue labels, footer text, `REPORT_LABELS`, `NOTIFY_LABELS` |
5456
| `src/github.ts` | GitHub API helpers: `fetchRecentItems`, `fetchRecentReleases`, `fetchSkillsData`, `createGitHubIssue`; shared `RepoFetch` type |
55-
| `src/prompts.ts` | LLM prompt builders (one per report type) and `formatItem` |
56-
| `src/date.ts` | CST (UTC+8) date helpers: `toCstDateStr`, `toUtcStr` |
57+
| `src/prompts.ts` | LLM prompt builders for repo reports: `buildCliPrompt`, `buildPeerPrompt`, `buildComparisonPrompt`, `buildPeersComparisonPrompt`, `buildSkillsPrompt` |
58+
| `src/prompts-data.ts` | LLM prompt builders for data-source reports: `buildTrendingPrompt`, `buildWebReportPrompt`, `buildHnPrompt`, `buildWeeklyPrompt`, `buildMonthlyPrompt` |
59+
| `src/report.ts` | `callLlm` (with concurrency limiter), `saveFile`, `autoGenFooter` (uses i18n), LLM token budget constants |
60+
| `src/report-builders.ts` | `buildCliReportContent`, `buildOpenclawReportContent` — assemble final Markdown strings for CLI and OpenClaw reports |
61+
| `src/report-savers.ts` | `saveWebReport`, `saveTrendingReport`, `saveHnReport` — LLM call + file save + optional GitHub issue |
62+
| `src/date.ts` | Date and timing utilities: `toCstDateStr`, `toUtcStr`, `sleep` |
63+
| `src/rollup.ts` | Weekly and monthly rollup report generator |
5764
| `src/providers/types.ts` | `LlmProvider` interface, `ProviderName` type, `VALID_PROVIDER_NAMES` |
5865
| `src/providers/openai-compatible.ts` | `OpenAICompatibleProvider` — shared base class for OpenAI-compatible providers |
5966
| `src/providers/anthropic.ts` | `AnthropicProvider` — Anthropic SDK wrapper |
6067
| `src/providers/openai.ts` | `OpenAIProvider` — extends `OpenAICompatibleProvider` |
6168
| `src/providers/github-copilot.ts` | `GitHubCopilotProvider` — extends `OpenAICompatibleProvider` |
6269
| `src/providers/openrouter.ts` | `OpenRouterProvider` — extends `OpenAICompatibleProvider` |
6370
| `src/providers/index.ts` | `createProvider` factory + barrel re-exports |
64-
| `src/report.ts` | `callLlm` (with concurrency limiter), `saveFile`, `autoGenFooter`, LLM token budget constants |
6571
| `src/web.ts` | Sitemap-based web content fetching; state persisted to `digests/web-state.json` |
6672
| `src/trending.ts` | GitHub Trending HTML scraper + Search API topic queries |
6773
| `src/hn.ts` | Hacker News top AI stories via Algolia HN Search API |
@@ -90,7 +96,8 @@ Files written to `digests/YYYY-MM-DD/`:
9096

9197
## Key conventions
9298

93-
- All LLM prompts are in `src/prompts.ts`. Each report type has its own builder function. Prompts are written in Chinese and produce Chinese output.
99+
- All bilingual strings (titles, labels, footers, messages) are centralized in `src/i18n.ts`. Use the `Lang` type (`"zh" | "en"`) and `Record<Lang, string>` maps. Do not add inline bilingual ternaries elsewhere.
100+
- LLM prompt builders are split across two files: `src/prompts.ts` (repo-level prompts) and `src/prompts-data.ts` (data-source and rollup prompts). Each report type has its own builder function.
94101
- `callLlm(prompt, maxTokens?)` defaults to 4096 tokens. Web report uses 8192, trending uses 6144. HN report uses the default 4096.
95102
- On 429 rate-limit errors `callLlm` retries up to 3 times with exponential backoff (5 s / 10 s / 20 s); the concurrency slot is released during the wait.
96103
- The concurrency limiter (`LLM_CONCURRENCY = 5`) prevents 429s when many parallel LLM calls fire. Do not bypass it by calling SDK clients directly.
@@ -105,14 +112,16 @@ Files written to `digests/YYYY-MM-DD/`:
105112
- Web UI: `index.html` reads `manifest.json` to build the sidebar, then fetches `digests/YYYY-MM-DD/report.md` on demand.
106113
- RSS Feed: `feed.xml` at the repo root. Generated by `src/generate-manifest.ts` in the same `pnpm manifest` step. Contains the latest 30 items (newest first) across all report types. Item links use hash routing: `https://duanyytop.github.io/agents-radar/#YYYY-MM-DD/report`.
107114
- Both `manifest.json` and `feed.xml` are committed together in the "Commit manifest and feed" GHA step.
108-
- The `REPORT_LABELS` map in `generate-manifest.ts` must be kept in sync with the `LABELS` object in `index.html` when adding new report types.
115+
- The `REPORT_LABELS` map in `src/i18n.ts` must be kept in sync with the `LABELS` object in `index.html` when adding new report types.
109116

110117
## Adding a new report type
111118

112119
1. Create a data fetcher (or add to an existing one).
113-
2. Add a `buildXxxPrompt` function in `src/prompts.ts`.
114-
3. Wire into `fetchAllData`, `generateSummaries`, and a `saveXxxReport` function in `src/index.ts`.
115-
4. Add a label color entry in `LABEL_COLORS` in `src/github.ts`.
116-
5. Add the report ID and label to `REPORT_LABELS` in `src/generate-manifest.ts` and `LABELS` in `index.html`.
117-
6. Add the report file name to `REPORT_FILES` in `src/generate-manifest.ts`.
118-
7. Update both README files and this file.
120+
2. Add a `buildXxxPrompt` function in `src/prompts-data.ts` (for data-source prompts) or `src/prompts.ts` (for repo-level prompts).
121+
3. Add bilingual strings (titles, labels, issue title function) to `src/i18n.ts`.
122+
4. Add a `saveXxxReport` function in `src/report-savers.ts`.
123+
5. Wire into `fetchAllData`, `generateSummaries`, and the save phase in `src/index.ts`.
124+
6. Add a label color entry in `LABEL_COLORS` in `src/github.ts`.
125+
7. Add the report ID and label to `REPORT_LABELS` in `src/i18n.ts` and `LABELS` in `index.html`.
126+
8. Add the report file name to `REPORT_FILES` in `src/generate-manifest.ts`.
127+
9. Update both README files and this file.

src/__tests__/i18n.test.ts

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
import { describe, it, expect } from "vitest";
2+
import {
3+
MSG,
4+
CLI_REPORT,
5+
OPENCLAW_REPORT,
6+
WEB_REPORT,
7+
TRENDING_REPORT,
8+
HN_REPORT,
9+
WEEKLY_REPORT,
10+
MONTHLY_REPORT,
11+
ISSUE_LABELS,
12+
CLI_ISSUE_TITLE,
13+
OPENCLAW_ISSUE_TITLE,
14+
FOOTER,
15+
NOTIFY_LABELS,
16+
} from "../i18n.ts";
17+
18+
// ---------------------------------------------------------------------------
19+
// Static bilingual strings
20+
// ---------------------------------------------------------------------------
21+
22+
describe("bilingual string maps", () => {
23+
const maps = [
24+
{ name: "MSG.noActivity", obj: MSG.noActivity },
25+
{ name: "MSG.summaryFailed", obj: MSG.summaryFailed },
26+
{ name: "MSG.skillsFailed", obj: MSG.skillsFailed },
27+
{ name: "MSG.trendingNoData", obj: MSG.trendingNoData },
28+
{ name: "MSG.trendingFailed", obj: MSG.trendingFailed },
29+
{ name: "CLI_REPORT.title", obj: CLI_REPORT.title },
30+
{ name: "CLI_REPORT.skillsHeading", obj: CLI_REPORT.skillsHeading },
31+
{ name: "CLI_REPORT.comparison", obj: CLI_REPORT.comparison },
32+
{ name: "CLI_REPORT.detail", obj: CLI_REPORT.detail },
33+
{ name: "OPENCLAW_REPORT.title", obj: OPENCLAW_REPORT.title },
34+
{ name: "OPENCLAW_REPORT.deepDive", obj: OPENCLAW_REPORT.deepDive },
35+
{ name: "WEB_REPORT.title", obj: WEB_REPORT.title },
36+
{ name: "WEB_REPORT.firstCrawl", obj: WEB_REPORT.firstCrawl },
37+
{ name: "TRENDING_REPORT.title", obj: TRENDING_REPORT.title },
38+
{ name: "HN_REPORT.title", obj: HN_REPORT.title },
39+
{ name: "WEEKLY_REPORT.title", obj: WEEKLY_REPORT.title },
40+
{ name: "MONTHLY_REPORT.title", obj: MONTHLY_REPORT.title },
41+
{ name: "FOOTER.autoGen", obj: FOOTER.autoGen },
42+
];
43+
44+
for (const { name, obj } of maps) {
45+
it(`${name} has both zh and en`, () => {
46+
expect(obj).toHaveProperty("zh");
47+
expect(obj).toHaveProperty("en");
48+
expect(obj.zh).toBeTruthy();
49+
expect(obj.en).toBeTruthy();
50+
expect(obj.zh).not.toBe(obj.en);
51+
});
52+
}
53+
});
54+
55+
// ---------------------------------------------------------------------------
56+
// Dynamic title functions
57+
// ---------------------------------------------------------------------------
58+
59+
describe("issue title functions", () => {
60+
it("CLI_ISSUE_TITLE produces zh and en titles", () => {
61+
expect(CLI_ISSUE_TITLE("2026-03-12", "zh")).toContain("AI CLI");
62+
expect(CLI_ISSUE_TITLE("2026-03-12", "zh")).toContain("2026-03-12");
63+
expect(CLI_ISSUE_TITLE("2026-03-12", "en")).toContain("AI CLI Tools Digest");
64+
});
65+
66+
it("OPENCLAW_ISSUE_TITLE produces zh and en titles", () => {
67+
expect(OPENCLAW_ISSUE_TITLE("2026-03-12", "zh")).toContain("OpenClaw");
68+
expect(OPENCLAW_ISSUE_TITLE("2026-03-12", "en")).toContain("OpenClaw Ecosystem Digest");
69+
});
70+
71+
it("WEB_REPORT.issueTitle includes first crawl flag", () => {
72+
expect(WEB_REPORT.issueTitle("2026-03-12", true, "zh")).toContain("首次全量");
73+
expect(WEB_REPORT.issueTitle("2026-03-12", false, "zh")).not.toContain("首次全量");
74+
expect(WEB_REPORT.issueTitle("2026-03-12", true, "en")).toContain("First Crawl");
75+
});
76+
77+
it("TRENDING_REPORT.issueTitle produces zh and en", () => {
78+
expect(TRENDING_REPORT.issueTitle("2026-03-12", "zh")).toContain("开源趋势");
79+
expect(TRENDING_REPORT.issueTitle("2026-03-12", "en")).toContain("Open Source Trends");
80+
});
81+
82+
it("HN_REPORT.issueTitle produces zh and en", () => {
83+
expect(HN_REPORT.issueTitle("2026-03-12", "zh")).toContain("Hacker News");
84+
expect(HN_REPORT.issueTitle("2026-03-12", "en")).toContain("Hacker News");
85+
});
86+
87+
it("WEEKLY_REPORT.issueTitle includes week string", () => {
88+
expect(WEEKLY_REPORT.issueTitle("2026-W11")).toContain("2026-W11");
89+
});
90+
91+
it("MONTHLY_REPORT.issueTitle includes month string", () => {
92+
expect(MONTHLY_REPORT.issueTitle("2026-02")).toContain("2026-02");
93+
});
94+
});
95+
96+
// ---------------------------------------------------------------------------
97+
// Dynamic content functions
98+
// ---------------------------------------------------------------------------
99+
100+
describe("dynamic content helpers", () => {
101+
it("CLI_REPORT.meta produces zh and en metadata", () => {
102+
const zh = CLI_REPORT.meta("12:00", 5, "zh");
103+
expect(zh).toContain("12:00");
104+
expect(zh).toContain("5 个");
105+
106+
const en = CLI_REPORT.meta("12:00", 5, "en");
107+
expect(en).toContain("12:00");
108+
expect(en).toContain("Tools covered: 5");
109+
});
110+
111+
it("WEB_REPORT.newContent formats count", () => {
112+
expect(WEB_REPORT.newContent(10, "zh")).toContain("10 篇");
113+
expect(WEB_REPORT.newContent(10, "en")).toContain("10 articles");
114+
});
115+
116+
it("WEB_REPORT.generated formats timestamp", () => {
117+
expect(WEB_REPORT.generated("12:00", "zh")).toContain("12:00 UTC");
118+
expect(WEB_REPORT.generated("12:00", "en")).toContain("12:00 UTC");
119+
});
120+
});
121+
122+
// ---------------------------------------------------------------------------
123+
// ISSUE_LABELS
124+
// ---------------------------------------------------------------------------
125+
126+
describe("ISSUE_LABELS", () => {
127+
it("maps report types to label names", () => {
128+
expect(ISSUE_LABELS.cli.zh).toBe("digest");
129+
expect(ISSUE_LABELS.cli.en).toBe("digest-en");
130+
expect(ISSUE_LABELS.openclaw.zh).toBe("openclaw");
131+
expect(ISSUE_LABELS.trending.en).toBe("trending-en");
132+
expect(ISSUE_LABELS.hn.en).toBe("hn-en");
133+
});
134+
});
135+
136+
// ---------------------------------------------------------------------------
137+
// NOTIFY_LABELS
138+
// ---------------------------------------------------------------------------
139+
140+
describe("NOTIFY_LABELS", () => {
141+
it("covers all report types", () => {
142+
const expected = ["ai-cli", "ai-agents", "ai-web", "ai-trending", "ai-hn", "ai-weekly", "ai-monthly"];
143+
for (const key of expected) {
144+
expect(NOTIFY_LABELS[key]).toBeDefined();
145+
expect(NOTIFY_LABELS[key]!.zh).toBeTruthy();
146+
expect(NOTIFY_LABELS[key]!.en).toBeTruthy();
147+
}
148+
});
149+
});

src/__tests__/prompt-builders.test.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,14 @@ import {
55
buildComparisonPrompt,
66
buildPeersComparisonPrompt,
77
buildSkillsPrompt,
8+
} from "../prompts.ts";
9+
import {
810
buildTrendingPrompt,
911
buildWebReportPrompt,
1012
buildWeeklyPrompt,
1113
buildMonthlyPrompt,
1214
buildHnPrompt,
13-
} from "../prompts.ts";
15+
} from "../prompts-data.ts";
1416
import type { RepoConfig, GitHubItem, GitHubRelease } from "../github.ts";
1517
import type { RepoDigest } from "../prompts.ts";
1618
import type { TrendingData } from "../trending.ts";

src/date.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Date utilities — CST (UTC+8) conversions used across the pipeline.
2+
* Date and timing utilities used across the pipeline.
33
*/
44

55
const CST_OFFSET_MS = 8 * 60 * 60 * 1000;
@@ -13,3 +13,8 @@ export function toCstDateStr(date: Date): string {
1313
export function toUtcStr(date: Date): string {
1414
return date.toISOString().slice(0, 16).replace("T", " ");
1515
}
16+
17+
/** Promise-based delay. */
18+
export function sleep(ms: number): Promise<void> {
19+
return new Promise((r) => setTimeout(r, ms));
20+
}

src/generate-manifest.ts

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import fs from "fs";
22
import path from "path";
3+
import { REPORT_LABELS } from "./i18n.ts";
34

45
const DIGESTS_DIR = "digests";
56
const MANIFEST_PATH = "manifest.json";
@@ -24,23 +25,6 @@ const REPORT_FILES = [
2425
] as const;
2526
const MAX_FEED_ITEMS = 30;
2627

27-
const REPORT_LABELS: Record<string, string> = {
28-
"ai-cli": "AI CLI 工具社区动态日报",
29-
"ai-cli-en": "AI CLI Tools Digest",
30-
"ai-agents": "AI Agents 生态日报",
31-
"ai-agents-en": "AI Agents Ecosystem Digest",
32-
"ai-web": "AI 官方内容追踪报告",
33-
"ai-web-en": "Official AI Content Report",
34-
"ai-trending": "AI 开源趋势日报",
35-
"ai-trending-en": "AI Open Source Trends",
36-
"ai-hn": "Hacker News AI 社区动态日报",
37-
"ai-hn-en": "Hacker News AI Community Digest",
38-
"ai-weekly": "AI 工具生态周报",
39-
"ai-weekly-en": "AI Tools Weekly Digest",
40-
"ai-monthly": "AI 工具生态月报",
41-
"ai-monthly-en": "AI Tools Monthly Digest",
42-
};
43-
4428
interface DateEntry {
4529
date: string;
4630
reports: string[];

src/github.ts

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,20 @@ export async function fetchSkillsData(repo: string): Promise<{ prs: GitHubItem[]
191191
const GITHUB_ISSUE_BODY_LIMIT = 65536;
192192
const TRUNCATION_NOTICE = "\n\n---\n> ⚠️ 内容超过 GitHub Issue 上限,完整报告见提交的 Markdown 文件。";
193193

194+
/** GitHub label colors by label name. Default: "0075ca". */
195+
const LABEL_COLORS: Record<string, string> = {
196+
openclaw: "e11d48",
197+
trending: "f9a825",
198+
hn: "ff6600",
199+
weekly: "7c3aed",
200+
monthly: "0d9488",
201+
"digest-en": "1d76db",
202+
"openclaw-en": "f472b6",
203+
"web-en": "6366f1",
204+
"trending-en": "fbbf24",
205+
"hn-en": "fb923c",
206+
};
207+
194208
/**
195209
* Break GitHub URLs in issue body to prevent cross-repository references.
196210
* Inserts a zero-width space in "github.com" so GitHub's auto-linker
@@ -206,18 +220,6 @@ export async function createGitHubIssue(title: string, body: string, label: stri
206220
if (body.length > GITHUB_ISSUE_BODY_LIMIT) {
207221
body = body.slice(0, GITHUB_ISSUE_BODY_LIMIT - TRUNCATION_NOTICE.length) + TRUNCATION_NOTICE;
208222
}
209-
const LABEL_COLORS: Record<string, string> = {
210-
openclaw: "e11d48",
211-
trending: "f9a825",
212-
hn: "ff6600",
213-
weekly: "7c3aed",
214-
monthly: "0d9488",
215-
"digest-en": "1d76db",
216-
"openclaw-en": "f472b6",
217-
"web-en": "6366f1",
218-
"trending-en": "fbbf24",
219-
"hn-en": "fb923c",
220-
};
221223
await ensureLabel(label, LABEL_COLORS[label] ?? "0075ca");
222224
const resp = await fetch(`https://api.github.com/repos/${digestRepo}/issues`, {
223225
method: "POST",

0 commit comments

Comments
 (0)