How to use this gem from within rails engines?

[fiedl]

I'm not sure if this is the right approach and maybe I've not understood this new part of the pipeline, yet, but:

Suppose, the major part of the app code lives inside a rails engine including all javascripts. And there are several main apps using that engine. The main apps only contain some layout changes and some minor patches.

Therefore, in order to integrate the webpacker gem into this setup, I'm trying to do the heavy lifting inside the engine, i.e. keep the work that has to be done inside all the main apps as little as possible.

How would I do that? Any pointers or suggestions are appreciated.

I'm not sure if this is a duplicate of #21. But in any case, I'd like to conclude this issue with a step-by-step guide how to approach this, for others facing the same use case.

What to do in the engine

1. Include webpacker in the *.gemspec file.
2. require 'webpacker' in the lib/foo/engine.rb.
3. ...

What to do in each main app

1. bundle install
2. Include ./bin/webpack-dev-server in the Procfile if using Foreman.
3. ...

[gauravtiwari]

@fiedl Out-of-the-box webpacker doesn't support engines yet. There are quite a few moving pieces that needs to be considered for this setup to work properly and usually people will have different use cases. I started doing some work on this but it's going to take time.

Feel free to leave any ideas/suggestions 👍

[fiedl]

@gauravtiwari Yes, after diving into this, I see your point. One would have to decide for each component where it belongs---to the engine, the main app, or both---and tie the pieces together in the right manner.

Some thoughts on this:

• The bin/webpack-dev-server is needed in the main app. One could add it to the Procfile of the main app to start the dev server with Foreman. Is there already a way to have the webpack dev server started by Pow?
• But one would need the ability to add a package from the engine.
• The app/javascript files have to be loaded from both, the engine and the main app. The main app files need to have the final say. I guess if there are several engines, the files from all engines need to be loaded.
• There are some files like .babelrc, .postcssrc.yml that would seem right in the main app root folder; but I'm not sure what these files are doing, yet.
• The config/webpack files appear to be tricky. In my use case, I would like to minimise the work to be done in the main app. But for other use cases, I imagine, the main app would need more control over the configuration. I really like how initializers work. I have most of them in the engine, but those of the main app have the last say in case of a conflict. Maybe we could utilise this mechanism: Would it make sense to tell webpacker in an initializer where to look for the webpack config? If no initializer is present, just use config/webpack from the rails root folder.
• For package.json and yarn.lock, one would need a mechanism similar to what is used with the *.gemspec, Gemfile and Gemfile.lock. If the engine decides that a certain package version is required, the package will be updated together with the engine when running bundle update my_engine from the main app. But the last say regarding the package versions has the main app as they are locked in the lock file. After all, the yarn.lock looks like a Gemfile.lock.
• This leads to the question how the upgrade process would look like from the view of the main app. I guess, just running bundle install from the main app would not cover any javascript package updates from the engine.

[himdel]

@gauravtiwari I started adding webpacker support to ManageIQ, where the UI is just an engine (so, app root != UI root), and we may need to support reading assets from multiple engines, and we need to output them to the rails root folder, not engine root.

To achieve that, I needed to override most of the webpacker methods to use the engine root instead of hardcoding Rails.root, I also needed to add a rake task that outputs Rails.root and call that from webpack config (didn't find another way of providing that path to webpack). (This also means that we need to include the initializer that overrides those paths from our Rakefile, which is a bit unfortunate :).)

.. And I still haven't figured out what would be needed to be able to call webpacker:compile, so far it it works if you manually call Webpacker.bootstrap but only when called from the root app, not when called from the engine.

I still haven't finished work on the "read assets from multiple engines" bit, but so far, I have a rake task that outputs a json of all the engines and their root paths (and filters those engines by existence of /app/javascript, until there's a proper way to register this), and intending to read that in the webpack config.

So.. I think at least these changes would be needed:

• don't assume all paths are relative to the current directory
• don't hardcode Rails.root anywhere
• come up with a sane way of passing those engine paths all the way to JS

If you're interested in the changes that we needed (and feel free to criticize and/or make suggestions), ManageIQ/manageiq-ui-classic#1132 .

---

Furthermore (and these are only my personal opinion):

• we should default to expecting the assets compiled, UI devels can enable webpack-dev-server, and for the rest, it should Just Work as long as they don't make UI changes.
• we should never deal with yarn, yarn and webpack are separate tools, and webpacker's job starts when you have all the assets, not before
• all those safety checks (compile depends on verify_install, verify_install depends on check_node and check_yarn, ..) are out of scope and needlessly complicate these tasks. All we need is to see the real error when it fails.

EDIT: Oh, and if there was a nice way of disabling the assets:precompile hook, it would be lovely, when wrapping that task, you may want the other taks to run on precompile :).

EDIT2: Aand making all the compile-time yarn deps as --dev, so that non-dev is reserved for actual UI dependencies, but I guess that's a separate issue :)

[zacksiri]

Right now i'm doing this, but obviously the solution is quite incomplete. Here is what I'm doing, what I see as the potential solution

by default wepbacker will always look for the manifest in the /public/packs folder of the Root application, that's because the default webpacker.yml is configured as such. However when coming from an engine environment it should respect the setting of the engine's webpacker.yml as well. which means webpacker needs to be aware of multiple namespaces.

For webpacker to be successful in an engine environment we need things to be isolated like every other part.

Which means, when we install the engine into an app, it needs to have an 'install step' that basically installs the pre-built modules into the rails app /public/packs/engine/name/manifest.json

We will probably need to modify the stylesheet_pack_tag and javascript_pack_tag helper to be aware of namespaces. So basically if you want to use the engine's assets you should be able to do something like javascript_pack_tag 'engine/name/vendor' etc... or javascript_pack_tag 'vendor', module: 'Engine::Name' which resolves to /public/packs/engine/name/vendor.js

I will have more clarity on this as I develop the solution. right now for me developing the engine with JS code I am just using a symlink from the dummy app's public/packs folder to the engine's public/packs folder, which works for development purposes.

A question for @dhh is also that should the Root app using the engine have to worry about compiling an engine's assets? If not it should make thing simpler since we would expect the JS code from the engine to already be compiled and usable which would mean we just need to make the javascript_pack_tag and stylesheet_pack_tag aware of those assets based on namespaces. or the module: 'Engine::Name' option that gets passed in.

[ohadlevy]

I've made a POC using webpack and rails engines that I believe that can be reused to solve this request: see ohadlevy/foreman@2afa796 for a reference.

[soundasleep]

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

[dhh]

Please Do Investigate 👍

…

On Jul 19, 2017, at 23:03, Jevon Wright ***@***.***> wrote: I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ohadlevy]

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman project.
they key was to create a simple ruby script that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths

[dhh]

Sounds like something we should be able to automate 👍

…

On Thu, Jul 20, 2017 at 7:01 AM, Ohad Levy ***@***.***> wrote: we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman <https://github.com/theforeman/foreman> project. they key was to create a simple ruby script <https://github.com/theforeman/foreman/blob/develop/script/plugin_webpack_directories.rb> that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths <https://github.com/theforeman/foreman/blob/develop/config/webpack.config.js#L30> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#348 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKtcL8JHDO8sv-eFyRzPOOtBxPNcmoks5sP0GAgaJpZM4NUgTi> .

[mltsy]

Should the Root app using the engine have to worry about compiling an engine's assets?

On the surface, it's a fair argument that an asset engine (which is essentially what a webpack app gem is) could be expected to have all the assets pre-compiled, because... once it's released the assets aren't going to change... but looking at how assets in engines currently work, that is not the case, probably to allow including uncompiled assets within other uncompiled assets in the Root app (e.g. application.scss could use mixins from an engine's scss files) if that's the intent of the engine. I'm not totally sure whether that would ever be the case with a webpack engine I guess... is it feasible that there might be a use case for an engine that provides pieces to be used within a larger webpack app? Perhaps the engine would expose some Elm module or React component, and the Root app's UI would utilize that component from its own Elm/React app?

In Elm at least, that actually doesn't make sense as a use case, because all modules have to be explicitly included in the elm-package.json, and will then be added to the dependency graph independently of any Rails Engine. But I'm not sure the same can be said of every potential webpacked app - for instance, what if your Engine's app is just straight ES6 that exposes some modules? Could the Root app import those modules from the engine to be used as potential internal components of it's reactive UI?

If that kind of integration is possible (and useful, which seems likely to me if it's possible), then in order to facilitate it, a Root app obviously needs to be able to compile the engine's assets (after they are included and used however they are going to be used).

My main argument against allowing this type of integration, through a Rails Engine specifically, would be that it's mixing what are becoming ever more the separate responsibilities of the server-side app and the client-side app. That is, if your reactive UI requires some component, it should import it using a javascript dependency manager like npm, whereas the Gemfile should manage dependencies for your server-side Rails app (which may include some compiled reactive UI app to be used on the front-end, like any other asset to be sent to the client)

My question is: is that a good enough argument to make it impossible using webpacker (by assuming there will be no precompiled integration between the UI components of an engine and its root app, and hence not providing any considerations to facilitate such integration)?

[tomprats]

I was originally patiently waiting for this kind of support so I could upgrade a gem I manage to be used with webpacker, but @mltsy's argument resonated pretty well with me.

Javascript provided by an engine to be used through webpacker could just be put in (an) npm package(s). This would also allow the javascript to become part of the npm ecosystem, becoming a dependency or depending on other packages.

The engine could still provide CSS and JS when needed through sprockets. One downside is the potential overlap of an engine providing a page that depends on javascript that is also provided in its npm package. It could lead to a few use-cases of an end user having to download the same javascript code from multiple locations

[tomasc]

@mltsy @tomprats I partially agree with this view.

However, let's say the engine is a pluggable CMS that defines its own JS dependencies (similarly to gems in .gemspec). I would hope there would be a way for the main Rails app to pickup these dependencies as well.

As of now, in my case, most of the JS dependencies defined in a Rails engine are facilitated via https://rails-assets.org. It would be fantastic to be able to define them via webpacker.