Webpack doesn’t work well with wasm modules created with Emscripten

See original GitHub issue

Feature request

What is the current behavior? The modularized JS emitted by Emscripten registers a global with a given name that loads the wasm file on invocation, initializes the wasm runtime and returns a Module.

Making it work with Webpack is quite hard as there seems to be interference with Webpack 4 defaults. This is the webpack.config.js that I came up with:

const webpack = require("webpack");
const path = require("path");

module.exports = {
  mode: "development",
  entry: "./index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "bundle.js"
  },
  module: {
    defaultRules: [
      {
        type: "javascript/auto",
        resolve: {}
      }
    ],
    rules: [
      {
        test: /fibonacci\.js$/,
        loader: "exports-loader"
      },
      {
        test: /fibonacci\.wasm$/,
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
  // This is necessary due to the fact that emscripten puts both Node and
  // web code into one file. The node part uses Node’s `fs` module to load
  // the wasm file.
  // Issue: https://github.com/kripken/emscripten/issues/6542.
  plugins: [new webpack.IgnorePlugin(/(fs)/)]
};

(Here is a minimal test project in a gist that you can clone and build with npm start. Docker required!)

edit: In the meantime, @sokra informed that that I can simplify the configuration a bit (and make it less like a sledgehammer):

module.exports = {
  /* ... */
  browser: { 
    "fs": false // ← !!
  },
  module: {
    rules: [
      /* ... */
      {
        test: /fibonacci\.wasm$/,
        type: "javascript/auto", // ← !!
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
};

Unexpected things I had to do

  • I needed to overwrite defaultRules as otherwise some sort of default rule will run in addition to the ones I specified and making webpack error “Module parse failed: magic header not detected” (try it!)
  • I needed to specify file-loader for the wasm file as otherwise webpack tries to resolve the names of the wasm module’s import object like env, which are provided by the JS file.
  • I needed to set a locateFile() function as webpack changes the file (and potentially path) of the wasm file and Emscripten hardcodes that name (not visible here but in the gist)

I am not sure what the right course of action here is, but considering that most wasm projects are going to be built with Emscripten, I feel like it’s worth making it easier.

Happy to answer Qs to give y’all a clearer picture.

What is the expected behavior?

Ideally, Webpack would recognize the typical Emscripten JS files and automatically bundle the accomodating wasm module and make paths work.

Other relevant information: webpack version: 4.8.3 Node.js version: 10 Operating System: Mac OS 10.13.4 Additional tools:

Issue Analytics

  • State:open
  • Created 5 years ago
  • Reactions:39
  • Comments:10 (3 by maintainers)

github_iconTop GitHub Comments

7reactions
mikeheddescommented, Mar 18, 2019

I also came across this problem when using a JS library I made, that uses WebAssembly, in a React web project.

I solved it by adding some hacky pre.js code to basically hijack the wasm module instantiation from the Emscripten created glue code. This way I have all the control over initialising the WASM module. The pre.js file overrides the createWasm function and adds a custom module initialisation function which it exports using ES6. Note that I compile with the default JS glue code settings (see cpp build script) so the EXPORT_ES6 and modularize options are not set but because the pre.js code has an ES6 export statement the generated glue code is an ES6 module.

After that I combine the generated JS glue code with the JS API interface code bundled with Rollup. The entry of this can be found in index.js.

Then finally when using the library I need to provide it an arrayBuffer (node.js) or a fetch instance (browsers) of the generated dcgp.wasm file. With webpack that means I needed the following rule:

{
  test: /\.wasm$/,
  type: 'javascript/auto',
  loader: 'file-loader',
}

Some implementation references:

So my conclusion being… I agree with @surma that Webpack and Emscripten don’t work well together but I think the main solution should be found in the way Emscripten generates the JS glue code and not how Webpack should handle them. I’m also not sure if @surma’s suggestion in #6542 to make separate platform outputs will fix the root cause of this problem. For me the following implementation would make more sense in the JS ecosystem:

// node.js
import { initializer } from './wasm-glue-code'
import { fileToArrayBuffer } from 'emscripten-helpers'

const myLibrary = initializer(fileToArrayBuffer('./wasm-file.wasm'))

// web
import { initializer } from './wasm-glue-code'

const myLibrary = initializer(fetch('./wasm-file.wasm'))

Hope this was helpful in some way.

5reactions
VirtualTimcommented, Apr 7, 2020

Hi @surma, @sokra, I know this is fairly old, but it seems like:

browser: { 
  "fs": false // ← !!
},

no longer works, and I can’t see any reference to browser in the webpack documentation. Any ideas what we should be using instead?

Edit: figured it out. Use this instead:

node: { 
  fs: "empty"
},
Read more comments on GitHub >

github_iconTop Results From Across the Web

Webpack throws error with Emscripten: Can't resolve 'fs'
One way to avoid the problem is that to use ENVIRONMENT flag in emcc command. This will remove some problematic codes such as...
Read more >
FAQ — Emscripten 3.1.26-git (dev) documentation
Why do I get multiple errors building basic code and the tests? I tried something: why doesn't it work? Do I need to...
Read more >
How To Use Emscripten & WASM in Web Apps
I wondered when I first started working with WebAssembly. To put things into perspective: I had just recently switched from Webpack to ...
Read more >
CPP to WASM Webpack Loader - Awesome JS
The only C/C++ webpack loader that can inject the full emscripten ... and Webassembly code is the module variable returned from the promise...
Read more >
Building Performant Web Apps with Rust, WebAssembly, and ...
However, your Javascript application can't use this binary directly since the compiled Rust code doesn't implement the module export syntax ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found