fix(route/twitter): dynamically resolve GraphQL query IDs and fix production CookieAgent#21544
Open
yuguorui wants to merge 2 commits intoDIYgod:masterfrom
Open
fix(route/twitter): dynamically resolve GraphQL query IDs and fix production CookieAgent#21544yuguorui wants to merge 2 commits intoDIYgod:masterfrom
yuguorui wants to merge 2 commits intoDIYgod:masterfrom
Conversation
…r JS bundles Twitter rotates GraphQL query IDs every 2-4 weeks, causing 404 errors. Instead of relying solely on hardcoded IDs, fetch and cache the latest query IDs from Twitter's client-web JS bundles at runtime. - Add gql-id-resolver module to extract queryId/operationName pairs - Cache resolved IDs for 24 hours via RSSHub cache (Redis supported) - Fall back to hardcoded IDs if dynamic resolution fails - Update init() to trigger resolution before first API call Signed-off-by: yuguorui <yuguorui@pku.edu.cn>
… dispatcher Both ofetch and the global wrappedFetch (request-rewriter) drop the dispatcher option when constructing a new Request object, which prevents the CookieAgent from attaching cookies in production builds. Use undici.fetch directly for Twitter API requests to preserve the dispatcher. Signed-off-by: yuguorui <yuguorui@pku.edu.cn>
github-actions
bot
added
route
auto: route no found
Contributor
|
Please use actual values in |
Contributor
Auto ReviewNo clear rule violations found in the current diff. |
TonyRL
reviewed
Mar 29, 2026
| async function fetchTwitterPage(): Promise<string> { | ||
| const response = await ofetch('https://x.com', { | ||
| headers: { | ||
| 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36', |
Collaborator
There was a problem hiding this comment.
Why use this fixed version of UA instead of RSSHub's auto-generated UA string? Does the site only works with this specific version of UA string?
| scriptUrls.map(async (url) => { | ||
| const content = await ofetch(url, { | ||
| headers: { | ||
| 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36', |
Collaborator
There was a problem hiding this comment.
Why use this fixed version of UA instead of RSSHub's auto-generated UA string? Does the site only works with this specific version of UA string?
|
|
||
| if (Object.keys(ids).length > 0) { | ||
| try { | ||
| await cache.set(CACHE_KEY, JSON.stringify(ids), CACHE_TTL); |
Collaborator
There was a problem hiding this comment.
Do not hard-coded cache duration. Users are expected to change it to their likings through CACHE_CONTENT_EXPIRE
Comment on lines
+136
to
+140
| try { | ||
| await cache.set(CACHE_KEY, JSON.stringify(ids), CACHE_TTL); | ||
| } catch { | ||
| logger.debug('twitter gql-id-resolver: cache write failed, IDs will be refetched next time'); | ||
| } |
Collaborator
There was a problem hiding this comment.
Do not wrap cache.set with try-catch.
| } catch { | ||
| logger.debug('twitter gql-id-resolver: failed to parse chunk map'); | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
main.hash.js has already got those 10 graphql hashes. Why other files are still needed?
Comment on lines
+132
to
+143
| logger.info('twitter gql-id-resolver: fetching fresh query IDs from Twitter JS bundles'); | ||
| const ids = await fetchAndExtractIds(); | ||
|
|
||
| if (Object.keys(ids).length > 0) { | ||
| try { | ||
| await cache.set(CACHE_KEY, JSON.stringify(ids), CACHE_TTL); | ||
| } catch { | ||
| logger.debug('twitter gql-id-resolver: cache write failed, IDs will be refetched next time'); | ||
| } | ||
| const found = operationNames.filter((name) => ids[name]); | ||
| const missing = operationNames.filter((name) => !ids[name]); | ||
| logger.info(`twitter gql-id-resolver: resolved ${found.length}/${operationNames.length} query IDs. Missing: ${missing.join(', ') || 'none'}`); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Involved Issue / 该 PR 相关 Issue
Close #18894
Example for the Proposed Route(s) / 路由地址示例
New RSS Route Checklist / 新 RSS 路由检查表
PuppeteerNote / 说明
Two fixes for the Twitter web API routes:
1. Dynamically resolve GraphQL query IDs (
gql-id-resolver.ts)Twitter rotates GraphQL query IDs every 2-4 weeks, causing 404 errors on all Twitter routes. This adds a
gql-id-resolvermodule that:queryId/operationNamepairs via regexapi.init()so resolution happens before the first API call2. Fix production build CookieAgent dispatcher loss (
utils.ts)In production builds, the Twitter API returns 404 with empty body because cookies are never sent. Root cause: both
ofetchand the globalwrappedFetch(request-rewriter) drop thedispatcheroption when constructingnew Request(input, init)— the standardRequestclass does not supportdispatcher, so theCookieAgentis silently discarded. This meansundici.fetchis called without the cookie jar.Fix: use
undici.fetchdirectly intwitterGotto preserve theCookieAgentdispatcher. All rate-limit and auth error handling logic is preserved.Testing
/twitter/homeand/twitter/home_latestwork in dev mode/twitter/homeand/twitter/home_latestwork in production build (was 404 before)oxlintandeslintpre-commit checks