Syntax error when using webpack 5 and karma 6

Apologies if this isn’t the correct place to report this, but getting a bit desparate. I’ve converted my app to work with webpack 5, but I’m unable to get the tests running. All configs look correct. I’ve also tried more or less plugging in the example config from here, but still getting the same error: https://github.com/appzuka/karma-webpackv5-test

Getting a strong feeling that something related to webpack or karma is completely off, but I’m unable to figure out what it is. As described below, the weird message I get no matter what I do is this:

Chrome 89.0.4389.114 (Windows 10) ERROR
  Uncaught SyntaxError: Unexpected token ':'
  at C:/Users/einarq/AppData/Local/Temp/_karma_webpack_1/commons.js:3:66

I’ve pasted the beginning of that generated file below.

• Operating System: Windows 10
• Node Version: 14.15.4
• NPM Version: 6.14.10
• webpack Version: 5.31.0
• karma-webpack Version: 5.0.0

Expected Behavior

Tests should run fine

Actual Behavior

08 04 2021 20:38:17.964:INFO [karma-server]: Karma v6.3.2 server started at http://localhost:9999/ 08 04 2021 20:38:17.965:INFO [launcher]: Launching browsers ChromeNoSandbox with concurrency unlimited 08 04 2021 20:38:17.971:INFO [launcher]: Starting browser Chrome 08 04 2021 20:38:20.213:INFO [Chrome 89.0.4389.114 (Windows 10)]: Connected on socket Arz-bQzd3v3C9CF0AAAB with id 68760601 Chrome 89.0.4389.114 (Windows 10) ERROR Uncaught SyntaxError: Unexpected token ‘:’ at C:/Users/einarq/AppData/Local/Temp/_karma_webpack_1/commons.js:3:66

SyntaxError: Unexpected token ‘:’

Code

{
      mode: 'development',
      output: {
        publicPath: '/',
      },
      cache: true,
      devtool: false,
      resolve: {
        modules: [path.resolve('./www/js/app'), 'www/js/libs', 'www/svg', 'node_modules', 'test/jasmine-ui', 'server'],
        extensions: ['.js', '.jsx', '.html', '.css', '.svg'],
        fallback: {stream: require.resolve('stream-browserify')},
      },
      plugins: [
        new webpack.ProvidePlugin({
          process: 'process/browser',
        }),
      ],
      module: {
        rules: [
          {
            test: /\.(jsx|js)?$/,
            include: [path.resolve(__dirname, 'www/js/app'), path.resolve(__dirname, 'server')],
            use: [
              {
                loader: 'babel-loader',
                options: {
                  cacheDirectory: true,
                },
              },
            ],
          },
        ],
      },
    },
    webpackMiddleware: {
      noInfo: true
    },
  }

Geneated file contents:

{
/***/ "./node_modules/@babel/runtime-corejs2/core-js/date/now.js":
/*!*****************************************************************!*\
  !*** ./node_modules/@babel/runtime-corejs2/core-js/date/now.js ***!
  \*****************************************************************/
/*! dynamic exports */
/*! export __esModule [maybe provided (runtime-defined)] [no usage info] [provision prevents renaming (no use info)] -> ./node_modules/@babel/runtime-corejs2/node_modules/core-js/library/fn/date/now.js .__esModule */
/*! other exports [maybe provided (runtime-defined)] [no usage info] -> ./node_modules/@babel/runtime-corejs2/node_modules/core-js/library/fn/date/now.js */
/*! runtime requirements: module, __webpack_require__ */
/***/ ((module, __unused_webpack_exports, __webpack_require__) => {

module.exports = __webpack_require__(/*! core-js/library/fn/date/now */ "./node_modules/@babel/runtime-corejs2/node_modules/core-js/library/fn/date/now.js");

/***/ }),

/***/ "./node_modules/@babel/runtime-corejs2/core-js/number/is-integer.js":
/*!**************************************************************************!*\
  !*** ./node_modules/@babel/runtime-corejs2/core-js/number/is-integer.js ***!