Skip to content

Commit 17599cb

Browse files
jasonterandoJason Terandojj22ee
authored
Fetch (#590)
* Add support for including sql query in sql subsegment for MySQL * Update createSqlData to accept values and sql as arguments, and the call to createSqlData to send those values from the arguments made to the sql call * Add function comments to mysqL_p.createSqlData * Working Node 14 and 18 tests for fetch * Auto/manual fetch support * Moved fetch patcher to sdk_contrib * Added integration testing for captureFetch * Revert tests to Javascript * Add chai-as-promised to base package.json dev dependencies, because you really need it to cleanly test async with chai; also fixed some lint findings * Try fixing package*.json * Change var to const * Removed captureFetch per @jj22ee * Removed chai-as-promised and tsconfig.debug.json * wip * wip * remove docker files used for diag * Updates per jj22ee * Housekeeping * More housekeeping * Fix typescript type test * update version and package-lock, fix tests * remove duplicate code * Record fetch info in subsegment http property * Add type def file for subsegment addFetchRequestData method; if url or method are not set store '' in http data --------- Co-authored-by: Jason Terando <[email protected]> Co-authored-by: jjllee <[email protected]>
1 parent 4644c81 commit 17599cb

File tree

16 files changed

+14860
-15937
lines changed

16 files changed

+14860
-15937
lines changed

package-lock.json

Lines changed: 13837 additions & 15934 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,11 @@
99
}
1010
},
1111
"devDependencies": {
12+
"@hapi/hapi": "^20.0.0",
1213
"@smithy/config-resolver": "^2.0.15",
1314
"@smithy/middleware-stack": "^2.0.6",
1415
"@smithy/node-config-provider": "^2.1.3",
1516
"@smithy/smithy-client": "^2.1.11",
16-
"@hapi/hapi": "^20.0.0",
1717
"@types/chai": "^4.2.12",
1818
"@types/koa": "^2.11.3",
1919
"@types/mocha": "^8.0.0",
@@ -45,7 +45,7 @@
4545
"grunt-contrib-clean": "^1.0.0",
4646
"grunt-jsdoc": "^2.4.0",
4747
"koa": "^2.13.0",
48-
"lerna": "^5.5.2",
48+
"lerna": "^5.6.2",
4949
"mocha": "^10.2.0",
5050
"nock": "^13.2.9",
5151
"nyc": "^15.1.0",
@@ -66,6 +66,7 @@
6666
"aws-xray-sdk-core": "file:packages/core",
6767
"aws-xray-sdk-express": "file:packages/express",
6868
"aws-xray-sdk-fastify": "file:sdk_contrib/fastify",
69+
"aws-xray-sdk-fetch": "file:sdk_contrib/fetch",
6970
"aws-xray-sdk-hapi": "file:sdk_contrib/hapi",
7071
"aws-xray-sdk-koa2": "file:sdk_contrib/koa",
7172
"aws-xray-sdk-mysql": "file:packages/mysql",

packages/core/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,4 +55,4 @@
5555
],
5656
"license": "Apache-2.0",
5757
"repository": "https://github.com/aws/aws-xray-sdk-node/tree/master/packages/core"
58-
}
58+
}

sdk_contrib/fetch/.eslintignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
dist

sdk_contrib/fetch/.eslintrc.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"extends": "../../.eslintrc.json"
3+
}

sdk_contrib/fetch/README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# fetch-xray
2+
3+
A patcher for AWSXRay to support fetch, implemented via either the [node-fetch module](https://www.npmjs.com/package/node-fetch) or the built-in
4+
global fetch support starting with [NodeJS 18](https://nodejs.org/en/blog/announcements/v18-release-announce).
5+
6+
## Usage
7+
8+
```js
9+
const { captureFetchGlobal } = require('aws-xray-sdk-fetch');
10+
11+
// To use globally defined fetch (available in NodeJS 18+)
12+
const fetch = captureFetchGloba();
13+
const result = await fetch('https://foo.com');
14+
15+
// To use node-fetch module
16+
const { captureFetchModule } = require('aws-xray-sdk-fetch');
17+
const fetchModule = require('node-fetch');
18+
const fetch = captureFetchModule(fetchModule); // Note, first parameter *must* be the node-fetch module
19+
const result = await fetch('https://foo.com');
20+
```
21+
22+
There are two optional parameters you can pass when calling `captureFetchGlobal` / `captureFetchModule`:
23+
24+
* **downstreamXRayEnabled**: when True, adds a "traced:true" property to the subsegment so the AWS X-Ray service expects a corresponding segment from the downstream service (default = False)
25+
* **subsegmentCallback**: a callback that is called with the subsegment, the fetch request, the fetch response and any error issued, allowing custom annotations and metadata to be added
26+
27+
TypeScript bindings for the capture functions are included.
28+
29+
## Testing
30+
31+
Unit and integration tests can be run using `npm run test`. Typings file tess can be run using `npm run test-d`.
32+
33+
## Errata
34+
35+
1. This package CommonJS to conform with the rest of the AWSXRay codebase. As such, it is incompatible with node-fetch 3, which is ESM only. As such, it is written
36+
to be compatible with [node-fetch version 2](https://www.npmjs.com/package/node-fetch#CommonJS), which should still receive critical fixes. If you are using global
37+
fetch (available in NodeJS 18+) then this isn't an issue for you.
38+
39+
2. This package is designed working under the assumption that the NodeJS implementation of fetch is compatible with node-fetch, albeit with its own separate,
40+
built-in typings. If NodeJS takes fetch in a different direction (breaks compatibility) then that would most likely break this package. There is no indication that
41+
I could find that this will happen, but throwing it out there "just in case".
42+
43+
## Contributors
44+
45+
- [Jason Terando](https://github.com/jasonterando)

sdk_contrib/fetch/lib/fetch_p.d.ts

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
/* eslint-disable @typescript-eslint/no-unused-vars */
2+
import AWSXRay from 'aws-xray-sdk-core';
3+
import * as fetchModule from 'node-fetch';
4+
5+
type FetchModuleType = typeof fetchModule;
6+
7+
type fetchModuleFetch = (url: URL | fetchModule.RequestInfo, init?: fetchModule.RequestInit | undefined) => Promise<fetchModule.Response>;
8+
9+
export function captureFetchGlobal(
10+
downstreamXRayEnabled?: boolean,
11+
subsegmentCallback?: (subsegment: AWSXRay.Subsegment, req: Request, res: Response | null, error: Error) => void):
12+
typeof globalThis.fetch;
13+
14+
export function captureFetchModule(
15+
fetch: FetchModuleType,
16+
downstreamXRayEnabled?: boolean,
17+
subsegmentCallback?: (subsegment: AWSXRay.Subsegment, req: fetchModule.Request, res: fetchModule.Response | null, error: Error) => void):
18+
(url: URL | fetchModule.RequestInfo, init?: fetchModule.RequestInit | undefined) => Promise<fetchModule.Response>;
19+

sdk_contrib/fetch/lib/fetch_p.js

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
/**
2+
* @module fetch_p
3+
*/
4+
5+
/**
6+
* This module patches the global fetch instance for NodeJS 18+
7+
*/
8+
const AWSXRay = require('aws-xray-sdk-core');
9+
const utils = AWSXRay.utils;
10+
const getLogger = AWSXRay.getLogger;
11+
require('./subsegment_fetch');
12+
13+
/**
14+
* Wrap fetch global instance for recent NodeJS to automatically capture information for the segment.
15+
* This patches the built-in fetch function globally.
16+
* @param {boolean} downstreamXRayEnabled - when true, adds a "traced:true" property to the subsegment
17+
* so the AWS X-Ray service expects a corresponding segment from the downstream service.
18+
* @param {function} subsegmentCallback - a callback that is called with the subsegment, the fetch request,
19+
* the fetch response and any error issued, allowing custom annotations and metadata to be added.
20+
* @alias module:fetch_p.captureFetchGlobal
21+
*/
22+
function captureFetchGlobal(downstreamXRayEnabled, subsegmentCallback) {
23+
if (globalThis.fetch === undefined) {
24+
throw new Error('Global fetch is not available in NodeJS');
25+
}
26+
if (!globalThis.__fetch) {
27+
globalThis.__fetch = globalThis.fetch;
28+
globalThis.fetch = enableCapture(globalThis.__fetch, globalThis.Request,
29+
downstreamXRayEnabled, subsegmentCallback);
30+
}
31+
return globalThis.fetch;
32+
}
33+
34+
/**
35+
* Wrap fetch module to capture information for the segment.
36+
* This patches the fetch function distributed in node-fetch package.
37+
* @param {fetch} module - The fetch module
38+
* @param {boolean} downstreamXRayEnabled - when true, adds a "traced:true" property to the subsegment
39+
* so the AWS X-Ray service expects a corresponding segment from the downstream service.
40+
* @param {function} subsegmentCallback - a callback that is called with the subsegment, the fetch request,
41+
* the fetch response and any error issued, allowing custom annotations and metadata to be added.
42+
* @alias module:fetch_p.captureFetchModule
43+
*/
44+
function captureFetchModule(module, downstreamXRayEnabled, subsegmentCallback) {
45+
if (!module.default) {
46+
getLogger().warn('X-ray capture did not find fetch function in module');
47+
return null;
48+
}
49+
if (!module.__fetch) {
50+
module.__fetch = module.default;
51+
module.default = enableCapture(module.__fetch, module.Request,
52+
downstreamXRayEnabled, subsegmentCallback);
53+
}
54+
return module.default;
55+
}
56+
57+
const enableCapture = function enableCapture(baseFetchFunction, requestClass, downstreamXRayEnabled, subsegmentCallback) {
58+
59+
const overridenFetchAsync = async (...args) => {
60+
const thisDownstreamXRayEnabled = !!downstreamXRayEnabled;
61+
const thisSubsegmentCallback = subsegmentCallback;
62+
// Standardize request information
63+
const request = typeof args[0] === 'object' ?
64+
args[0] :
65+
new requestClass(...args);
66+
67+
// Facilitate the addition of Segment information via the request arguments
68+
const params = args.length > 1 ? args[1] : {};
69+
70+
// Short circuit if the HTTP is already being captured
71+
if (request.headers.has('X-Amzn-Trace-Id')) {
72+
return await baseFetchFunction(...args);
73+
}
74+
75+
const url = new URL(request.url);
76+
const isAutomaticMode = AWSXRay.isAutomaticMode();
77+
78+
const parent = AWSXRay.resolveSegment(AWSXRay.resolveManualSegmentParams(params));
79+
const hostname = url.hostname || url.host || 'Unknown host';
80+
81+
if (!parent) {
82+
let output = '[ host: ' + hostname +
83+
(request.method ? (', method: ' + request.method) : '') +
84+
', path: ' + url.pathname + ' ]';
85+
86+
if (isAutomaticMode) {
87+
getLogger().info('RequestInit for request ' + output +
88+
' is missing the sub/segment context for automatic mode. Ignoring.');
89+
} else {
90+
getLogger().info('RequestInit for request ' + output +
91+
' requires a segment object on the options params as "XRaySegment" for tracing in manual mode. Ignoring.');
92+
}
93+
94+
// Options are not modified, only parsed for logging. We can pass in the original arguments.
95+
return await baseFetchFunction(...args);
96+
}
97+
98+
let subsegment;
99+
if (parent.notTraced) {
100+
subsegment = parent.addNewSubsegmentWithoutSampling(hostname);
101+
} else {
102+
subsegment = parent.addNewSubsegment(hostname);
103+
}
104+
105+
subsegment.namespace = 'remote';
106+
107+
request.headers.set('X-Amzn-Trace-Id',
108+
'Root=' + (parent.segment ? parent.segment : parent).trace_id +
109+
';Parent=' + subsegment.id +
110+
';Sampled=' + (subsegment.notTraced ? '0' : '1'));
111+
112+
// Set up fetch call and capture any thrown errors
113+
const capturedFetch = async () => {
114+
const requestClone = request.clone();
115+
let response;
116+
try {
117+
response = await baseFetchFunction(requestClone);
118+
119+
if (thisSubsegmentCallback) {
120+
thisSubsegmentCallback(subsegment, requestClone, response);
121+
}
122+
123+
const statusCode = response.status;
124+
if (statusCode === 429) {
125+
subsegment.addThrottleFlag();
126+
}
127+
128+
const cause = utils.getCauseTypeFromHttpStatus(statusCode);
129+
if (cause) {
130+
subsegment[cause] = true;
131+
}
132+
133+
subsegment.addFetchRequestData(requestClone, response, thisDownstreamXRayEnabled);
134+
subsegment.close();
135+
return response;
136+
} catch (e) {
137+
if (thisSubsegmentCallback) {
138+
thisSubsegmentCallback(subsegment, requestClone, response, e);
139+
}
140+
const madeItToDownstream = (e.code !== 'ECONNREFUSED');
141+
subsegment.addErrorFlag();
142+
subsegment.addFetchRequestData(requestClone, response, madeItToDownstream && thisDownstreamXRayEnabled);
143+
subsegment.close(e);
144+
throw (e);
145+
}
146+
};
147+
148+
if (isAutomaticMode) {
149+
const session = AWSXRay.getNamespace();
150+
return await session.runPromise(async () => {
151+
AWSXRay.setSegment(subsegment);
152+
return await capturedFetch();
153+
});
154+
} else {
155+
return await capturedFetch();
156+
}
157+
};
158+
159+
return overridenFetchAsync;
160+
};
161+
162+
module.exports.captureFetchGlobal = captureFetchGlobal;
163+
module.exports.captureFetchModule = captureFetchModule;
164+
module.exports._fetchEnableCapture = enableCapture;
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import { Request, Response } from 'node-fetch';
2+
3+
declare module 'aws-xray-sdk-core' {
4+
interface Subsegment {
5+
/**
6+
* Extends Subsegment to append remote request data to subsegment, similar to what
7+
* Subsegment.prototype.addRemoteRequestData does in core/lib/segments/attributes/subsegment.js
8+
* @param {Fetch Request} request
9+
* @param {Fetch Request or null|undefined} response
10+
* @param {boolean} downstreamXRayEnabled
11+
*/
12+
addFetchRequestData(
13+
request: Request,
14+
response: Response,
15+
downstreamXRayEnabled: boolean): void;
16+
}
17+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
const {Subsegment} = require('aws-xray-sdk-core');
2+
3+
/**
4+
* Extends Subsegment to append remote request data to subsegment, similar to what
5+
* Subsegment.prototype.addRemoteRequestData does in core/lib/segments/attributes/subsegment.js
6+
* @param {Subsegment} subsegment
7+
* @param {Fetch Request} request
8+
* @param {Fetch Request or null|undefined} response
9+
* @param {boolean} downstreamXRayEnabled
10+
*/
11+
Subsegment.prototype.addFetchRequestData = function addFetchRequestData(request, response, downstreamXRayEnabled) {
12+
this.http = {
13+
request: {
14+
url: request.url?.toString() ?? '',
15+
method: request.method ?? ''
16+
}
17+
};
18+
19+
if (downstreamXRayEnabled) {
20+
this.traced = true;
21+
}
22+
23+
if (response) {
24+
this.http.response = {
25+
status: response.status
26+
};
27+
if (response.headers) {
28+
const clength = response.headers.get('content-length');
29+
if (clength) {
30+
const v = parseInt(clength);
31+
if (! Number.isNaN(v)) {
32+
this.http.response.content_length = v;
33+
}
34+
}
35+
}
36+
}
37+
};

0 commit comments

Comments
 (0)