Skip to content

refactor: esbuild refactor for readability #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Feb 2, 2023

Conversation

torresxb1
Copy link
Contributor

@torresxb1 torresxb1 commented Jan 28, 2023

Issue #, if available:

Description of changes:
I originally intended to add build-in-source support for the esbuild workflow but I found it hard with the way it was currently set up. I'm hoping my changes make it more readable and easier to work with in general, but let me know if you disagree with my changes or you think there could be further improvements.

Main changes done:

  • Changed the bundle action:
    • parameter names were kind of coupled to scratch and artifacts directory, so I changed them to accept a working directory and an output directory
    • it now performs the esbuild version checking as part of this action (when needed)
  • workflow conditionals/IFs cleanup: my main issue with the workflow was the accelerate workflow stuff was a bit complicated, with a bunch of conditions and nested ifs that went a few levels deep. In my proposed changes for the workflow, IFs would only go a single level deep.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@torresxb1 torresxb1 changed the title refactor: esbuild workflow refactor for readability refactor: esbuild refactor for readability Jan 28, 2023
Comment on lines +83 to +86
minimum_version_required=MINIMUM_VERSION_FOR_EXTERNAL,
working_directory=self._working_directory,
subprocess_esbuild=self._subprocess_esbuild,
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This used to be a separate action. Since we need to check the version every time that we want to mark things as --external then I think we should do the check when/before marking things as external (here), instead of having a separate action that would always have to be appended before this one. This cleans things up a bit, but it's also safer as we don't have to remember that we need a check action before this bundle action.

Comment on lines +31 to +32
working_directory: str,
output_directory: str,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to make more general

@@ -5,20 +5,17 @@
import logging
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file changed quite a bit. Might be easier to open the new and the old file in different windows (without the file diffs), and comparing.

osutils,
self.options,
self.actions.append(
CopySourceAction(source_dir=self.source_dir, dest_dir=self.scratch_dir, excludes=self.EXCLUDED_FILES)
Copy link
Contributor Author

@torresxb1 torresxb1 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using named arguments can be more readable in cases where it's not immediately obvious what the parameter(s) would be for. E.g. here we would immediately know which folder we copy from and which one we copy to, without having to check the CopySourceAction definition.

osutils=self.osutils,
subprocess_esbuild=self.subprocess_esbuild,
manifest=self.manifest_path,
skip_deps=not self.combine_dependencies,
Copy link
Contributor Author

@torresxb1 torresxb1 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

combine_dependencies determines whether we should bundle dependencies or not

self.actions += [
NodejsNpmWorkflow.get_install_action(
source_dir=source_dir,
artifacts_dir=self.scratch_dir,
Copy link
Contributor Author

@torresxb1 torresxb1 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be better if this argument was named install_dir, as it actually controls where dependencies are installed. Will rename in another PR, when I introduce building-in-source to esbuild

@@ -244,11 +244,10 @@ def test_workflow_sets_up_esbuild_actions_without_download_dependencies_with_dep
experimental_flags=[],
)

self.assertEqual(len(workflow.actions), 4)
self.assertEqual(len(workflow.actions), 3)
Copy link
Contributor Author

@torresxb1 torresxb1 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple tests will have one action less because I moved the action to check esbuild version into the bundling action itself.

Comment on lines +271 to +272
self.assertIsInstance(workflow.actions[2], EsbuildBundleAction)
self.assertIsInstance(workflow.actions[3], CleanUpAction)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the order did change, to simplify the code logic. There should be no impact in the end-result, however. We just need to clean up the dependencies dir before we move the updated dependencies in there, but the bundling itself happens on another folder.

@torresxb1 torresxb1 marked this pull request as ready for review January 31, 2023 18:15
@torresxb1 torresxb1 requested a review from a team as a code owner January 31, 2023 18:15
@torresxb1 torresxb1 requested review from mildaniel, hnnasit and mndeveci and removed request for hnnasit January 31, 2023 18:15
Copy link
Contributor

@mndeveci mndeveci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the effort! Left some comments but will definitely do another round.

@torresxb1 torresxb1 requested a review from mndeveci February 1, 2023 18:38
Copy link
Contributor

@mildaniel mildaniel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Rup, this is a great QoL change. Left a couple small comments but generally agree with the direction you took combining the bundle action and version check action since they are always bundled together.

# if we're reusing dependencies, then we need to symlink them before bundling
self.actions += [
LinkSourceAction(self.dependencies_dir, self.scratch_dir),
bundle_action,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that the bundle action is already included in line 103, so this would run the bundler twice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one part that can lead to confusion and wasn't able to figure out how to make it better. It would never bundle twice because this bundle on this line happens when self.download_dependencies is False, while the one in line 103 happens when self.download_dependencies is True

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in other words, if we're reusing dependencies, then we didn't download them and we should first symlink and then bundle. If we downloaded dependencies, then we should bundle after downloading, (and we won't symlink or bundle again)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm seeing it if it's true for both cases:

if self.download_dependencies:
            self.actions += [
                NodejsNpmWorkflow.get_install_action(
                    source_dir=source_dir,
                    artifacts_dir=self.scratch_dir,
                    subprocess_npm=self.subprocess_npm,
                    osutils=self.osutils,
                    build_options=self.options,
                ),
                bundle_action,
            ]

        if not self.dependencies_dir:
            # we need a dependencies dir for the logic below
            return

        if self.download_dependencies:
            # if we downloaded dependencies, we have to update dependencies_dir
            self.actions += [
                CleanUpAction(self.dependencies_dir),
                MoveDependenciesAction(self.source_dir, self.scratch_dir, self.dependencies_dir),
            ]
        else:
            # if we're reusing dependencies, then we need to symlink them before bundling
            self.actions += [
                LinkSourceAction(self.dependencies_dir, self.scratch_dir),
                bundle_action,
            ]

Copy link
Contributor Author

@torresxb1 torresxb1 Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your indentation might be off? 😅 . The first if in the code block you're posting should be in the same level as the other two if's below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or otherwise not sure what you mean

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just made an additional change related to this, hopefully it's clearer now.

@torresxb1 torresxb1 requested a review from mildaniel February 1, 2023 22:40
@torresxb1
Copy link
Contributor Author

@mndeveci added tests to increase coverage and changed to numpy docstrings

@torresxb1 torresxb1 requested a review from mndeveci February 2, 2023 03:25
Copy link
Contributor

@mndeveci mndeveci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the effort that you put here @torresxb1! Approving the changes since we confirm that expected actions are been setup with possible combinations through tests.

@torresxb1 torresxb1 merged commit 755a99b into aws:develop Feb 2, 2023
@torresxb1 torresxb1 deleted the refactor_esbuild_workflow branch February 2, 2023 19:53
calavera pushed a commit to calavera/aws-lambda-builders that referenced this pull request Feb 8, 2023
hawflau pushed a commit that referenced this pull request Feb 9, 2023
* feat: support rust via cargo

* make black reformat

* use os.path.join in path assertion tests to address windows paths

* address pylint ci errors

* fix reformatted list comprehensions

* try passing rust-lld linker on windows

* update rust cargo design doc. windows support was added and tested

* add test for cargo workspaces project

* reformat test sources

* build with musl on linux because glibc may differ on lambda

* update linux copy and bin paths

* update make test again, use the version appveyor is complaining about

* update integration tests for rust cargo with latest aws rust runtime interfaces

* align TestCustomMakeWorkflow integ test assumptions with appveyor reality

* make x86_64-unknown-linux-musl a const for the rust cargo workflow

* add integ test for failing cargo rust build

* Update Rust workflow to use cargo-lambda

Cargo-lambda takes care of cross compiling using Zig as linker.
This works on Windows, Linux, and MacOS natively.

Signed-off-by: David Calavera <[email protected]>

* Fix deprecation warnings.

Signed-off-by: David Calavera <[email protected]>

* Install Zig on Windows manually

Signed-off-by: David Calavera <[email protected]>

* Print Zig version on Windows

Signed-off-by: David Calavera <[email protected]>

* Update version of cargo-lambda

Signed-off-by: David Calavera <[email protected]>

* Run windows tests in powershell

Signed-off-by: David Calavera <[email protected]>

* Fix powershell env notation

Signed-off-by: David Calavera <[email protected]>

* Print clang version on windows

Signed-off-by: David Calavera <[email protected]>

* Fix env variable name

Signed-off-by: David Calavera <[email protected]>

* Try new version of zigbuild that fixes some linker issues on Windows.

Signed-off-by: David Calavera <[email protected]>

* Update releases URL.

Signed-off-by: David Calavera <[email protected]>

* Upgrade LLVM and clang

Signed-off-by: David Calavera <[email protected]>

* Update visual studio image

To check if that makes any difference.

Signed-off-by: David Calavera <[email protected]>

* Change the visual studio image everywhere.

Signed-off-by: David Calavera <[email protected]>

* Revert upgrade changes

They didn't fix the problem

Signed-off-by: David Calavera <[email protected]>

* Print environment

Signed-off-by: David Calavera <[email protected]>

* Update to Visual Studio 2022

Signed-off-by: David Calavera <[email protected]>

* Fix python variable

Signed-off-by: David Calavera <[email protected]>

* Fix package urls

Signed-off-by: David Calavera <[email protected]>

* Update missing vs 2019 reference.

Signed-off-by: David Calavera <[email protected]>

* Change windows package suffix

Signed-off-by: David Calavera <[email protected]>

* Update cargo-lambda to version 0.9.0

Signed-off-by: David Calavera <[email protected]>

* Go back to the original VS version.

Signed-off-by: David Calavera <[email protected]>

* Cleanup options

- Use architecture to setup the right build target.
- Don't require `--bin` flag, the default behaviour should work for the majority of functions.
- Add flags option to provide a list of additional flags for projects that need extra configuration, like projects within a workspace.

Signed-off-by: David Calavera <[email protected]>

* Add integration test with cargo_lambda_flags

Signed-off-by: David Calavera <[email protected]>

* Detect binaries when the project only includes one function.

Signed-off-by: David Calavera <[email protected]>

* Bring back handler as an optional argument.

It saves some duplicated flags for working with workspaces.

Signed-off-by: David Calavera <[email protected]>

* Ignore errors if directory doesn't exist.

Signed-off-by: David Calavera <[email protected]>

* Add debug logs

Set RUST_LOG=debug when debug is enabled.

Signed-off-by: David Calavera <[email protected]>

* Log the artifact destination path.

Signed-off-by: David Calavera <[email protected]>

* Add missing comma

Signed-off-by: David Calavera <[email protected]>

* Print out and err in the log when debug is enabled

Signed-off-by: David Calavera <[email protected]>

* Only set RUST_LOG when it's not already set

Log it's value, so users know what's set at.

Signed-off-by: David Calavera <[email protected]>

* Add experimentalCargoLambda feature flag.

Signed-off-by: David Calavera <[email protected]>

* Add integration test for multi-function projects.

Signed-off-by: David Calavera <[email protected]>

* Remove type hints

They're causing false positives in Python 3.9 with pylint.

Signed-off-by: David Calavera <[email protected]>

* Add new CI steps for GHA

Signed-off-by: David Calavera <[email protected]>

* Add build_in_source_support

Signed-off-by: David Calavera <[email protected]>

* Test rust logger

Signed-off-by: David Calavera <[email protected]>

* Fix assertion

Signed-off-by: David Calavera <[email protected]>

* Don't fail fast

Signed-off-by: David Calavera <[email protected]>

* fix: Fix failing esbuild integration tests (#423)

* fix: Fix failing esbuild integration tests

* Replace npm ci with npm install

* Remove default shell

Signed-off-by: David Calavera <[email protected]>

* Revert "Remove default shell"

This reverts commit 478c3b6.

* Add check to ensure that Cargo Lambda is installed.

Include a link to the gettings started guide that gives direct installation instructions based on the platform.

Signed-off-by: David Calavera <[email protected]>

* feat: Add support for mjs files with esbuild (#427)

* feat: Add support for mjs files with esbuild

* Black reformat

* Test Cargo Lambda check

Signed-off-by: David Calavera <[email protected]>

* Don't redefine which

Signed-off-by: David Calavera <[email protected]>

* Organize code in more modules

This structure follows other workflows, and provides better testeability.

Signed-off-by: David Calavera <[email protected]>

* Add more documentation

Signed-off-by: David Calavera <[email protected]>

* Format code

Signed-off-by: David Calavera <[email protected]>

* Capture exception

Signed-off-by: David Calavera <[email protected]>

* chore: Remove type/bug label for Bug Issue Template (#425)

Co-authored-by: Jacob Fuss <[email protected]>
Co-authored-by: Mehmet Nuri Deveci <[email protected]>

* Clean design doc and tests

Signed-off-by: David Calavera <[email protected]>

* chore: Version bump to 1.25.0 (#429)

* refactor: assign each workflow a default build directory (#428)

* fix: remove unused symlinking (#432)

* chore: Move to ruff from pylint (#435)

When we started the project we defaulted to use pylint. Pylint has served it's
purpose but it pretty slow. Ruff is a newer linter in the python
ecosystem but is written in Rust. This makes Ruff was faster than pylint.
On my machine (while testing in SAM CLI), pylint took about 70s to lint the repo
but with ruff it took .04s.

Co-authored-by: Jacob Fuss <[email protected]>

* chore: Enable pylint within ruff (#436)

Co-authored-by: Jacob Fuss <[email protected]>

* refactor: esbuild refactor for readability (#433)

* feat: use build_dir in esbuild workflow to support building in source (#437)

* Fix formatting

Signed-off-by: David Calavera <[email protected]>

* Update build in source settings

Signed-off-by: David Calavera <[email protected]>

* fix: remove python3.6 support (#434)

* chore: bump version to 1.26.0 (#441)

* Remove default value for subprocess_cargo_lambda

Signed-off-by: David Calavera <[email protected]>

* feat: Pin ruff version, add dependabot config (#442)

* feat: Pin ruff version, add dependabot config to keep our dependencies up to date

* Exlude isort and flake8 from dependabot updates

* feat: Add sources content flag to supported esbuild options (#439)

* Better process management

Signed-off-by: David Calavera <[email protected]>

* Make release mode the default

Signed-off-by: David Calavera <[email protected]>

* Remove already default None

Signed-off-by: David Calavera <[email protected]>

* Turn debug message into warning

Signed-off-by: David Calavera <[email protected]>

* Update documentation format

Signed-off-by: David Calavera <[email protected]>

* Fix formatting

Signed-off-by: David Calavera <[email protected]>

* Preserve binary permissions

Use copy2 instead of copyfile

Signed-off-by: David Calavera <[email protected]>

---------

Signed-off-by: David Calavera <[email protected]>
Co-authored-by: softprops <[email protected]>
Co-authored-by: Daniel Mil <[email protected]>
Co-authored-by: Jacob Fuss <[email protected]>
Co-authored-by: Jacob Fuss <[email protected]>
Co-authored-by: Mehmet Nuri Deveci <[email protected]>
Co-authored-by: Ruperto Torres <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants