Skip to content

feat: etag support #746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

feat: etag support #746

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
eee8640
temporariy disable files
privatenumber Oct 27, 2020
768d0c0
temporariy disable files
privatenumber Oct 27, 2020
bf79de7
refactor: use native req/res methods
privatenumber Oct 27, 2020
6a28828
feat: added etag support
privatenumber Oct 27, 2020
203c767
revert: pkg files
privatenumber Oct 27, 2020
7aca742
style: added eslint-ignore for res statusCode
privatenumber Oct 27, 2020
494728c
test: fixed req/res methods
privatenumber Oct 27, 2020
a2b418d
revert: refactor
privatenumber Oct 27, 2020
0129e2c
revert: eslint-disable
privatenumber Oct 27, 2020
c9f954f
fix: memory leak in etagRegistry
privatenumber Oct 27, 2020
1bd3270
feat: etag option
privatenumber Oct 27, 2020
420c6fb
refactor: removed compiler variable
privatenumber Oct 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 30 additions & 3 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
"webpack": "^4.0.0 || ^5.0.0"
},
"dependencies": {
"etag": "^1.8.1",
"mem": "^8.0.0",
"memfs": "^3.2.0",
"mime-types": "^2.1.27",
Expand Down
39 changes: 39 additions & 0 deletions src/middleware.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,35 @@ import path from 'path';

import mime from 'mime-types';

import getETag from 'etag';

import getFilenameFromUrl from './utils/getFilenameFromUrl';
import handleRangeHeaders from './utils/handleRangeHeaders';
import ready from './utils/ready';

const { hasOwnProperty } = Object.prototype;

export default function wrapper(context) {
let etagRegistry;

if (context.options.etags) {
etagRegistry = new Map();
context.compiler.hooks.done.tap('webpack-dev-middleware', (stats) => {
etagRegistry.clear();

try {
const { assets } = stats.compilation;
for (const assetId in assets) {
if (hasOwnProperty.call(assets, assetId)) {
const { existsAt: fsPath } = assets[assetId];
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems existsAt doesn't exist in Webpack 5 either

const etag = getETag(assets[assetId].source());
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this is no longer possible in Webpack 5, due to error:

[webpack-dev-middleware] Error: Content and Map of this Source is not available (only size() is supported)
at SizeOnlySource._error (/node_modules/webpack-sources/lib/SizeOnlySource.js:16:10)
at SizeOnlySource.source (/node_modules/webpack-sources/lib/SizeOnlySource.js:26:14)

@alexander-akait Do you have a recommended work around for this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up leveraging chunkHashs, which is actually more convenient because I can just use the hash as the etag instead of generating one from the file source:

const { chunkHashs: assetHashes } = stats.compilation.records;
for (const assetId in assetHashes) {
	if (hasOwnProp(assetHashes, assetId)) {
		etagRegistry.set(assetId, JSON.stringify(assetHashes[assetId]));
	}
}

If there's an alternative you recommend, LMK.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @sokra What do you think?

etagRegistry.set(fsPath, etag);
}
}
} catch (_ignoreError) {} // eslint-disable-line no-empty
});
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No hooks here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already can load assets here, please look at logic

Copy link
Author

@privatenumber privatenumber Oct 27, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw that there's access to the memfs instance, but is there a manifest object (eg. build stats) I can iterate over to get all the asset paths?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you need iterate?

Copy link
Member

@alexander-akait alexander-akait Oct 28, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway if you need this, you have context, put this logic inside done hooks and collect them context.etags

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.

I want to iterate a list of distribution files to efficiently calculate etags. If I only have the memfs instance, I'd have to traverse the entire fs, which isn't optimal.


return async function middleware(req, res, next) {
const acceptedMethods = context.options.methods || ['GET', 'HEAD'];
// fixes #282. credit @cexoso. in certain edge situations res.locals is undefined.
Expand Down Expand Up @@ -49,6 +73,21 @@ export default function wrapper(context) {
return;
}

if (etagRegistry) {
const assetEtag = etagRegistry.get(filename);
if (assetEtag) {
const { 'if-none-match': ifNoneMatch } = req.headers;
if (ifNoneMatch) {
if (assetEtag === ifNoneMatch) {
res.status(304).end();
return;
}
} else {
res.set('ETag', assetEtag);
}
}
}

try {
content = context.outputFileSystem.readFileSync(filename);
} catch (_ignoreError) {
Expand Down
3 changes: 3 additions & 0 deletions src/options.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
"headers": {
"type": "object"
},
"etags": {
"type": "boolean"
},
"publicPath": {
"type": "string"
},
Expand Down