JaCoCo not working with `@QuarkusIntegrationTest`s

Describe the bug

After upgrading to Quarkus 2 we saw a test coverage loss of 6% for both unit and integration tests.

I’ve tried to change how we have everything setup following https://quarkus.io/guides/tests-with-coverage (since we were still using offline instrumentation), starting with our integration tests, but I can’t get things to work right (JaCoCo complaints about classes not matching execution data). I’ve created a small reproducer (linked below) by setting up the necessary parts as in https://quarkus.io/guides/tests-with-coverage and then having the following test (since we can’t use @QuarkusIntegrationTest for a couple of reasons):

public class AppIT {

    @Test
    void myTest() throws Exception {
        new ProcessBuilder()
                .command("java", System.getProperty("quarkus.test.argLine"), "-jar", "target/quarkus-app/quarkus-run.jar")
                .inheritIO()
                .start()
                .waitFor();
    }

}

I also tried then with a REST resource and a proper @QuarkusIntegrationTest (reproducer also linked below) and got the same result: JaCoCo still complaining and showing a coverage of 0%.

Expected behavior

Can get JaCoCo working together with Quarkus, finding the right classes and reporting the right coverage.

Actual behavior

[INFO] --- jacoco-maven-plugin:0.8.7:report (report-it) @ app ---
[INFO] Loading execution data file /Users/xtaixe/dev/quarkus-reproducer/app/target/jacoco-quarkus.exec
[INFO] Analyzed bundle 'app' with 2 classes
[WARNING] Classes in bundle 'app' do not match with execution data. For report generation the same class files must be used as at runtime.

And JaCoCo reports an incorrect coverage (0% in the reproducers).

To Reproduce

quarkus-reproducer-1.zip quarkus-reproducer-2.zip

Environment (please complete the following information):

Output of `uname -a` or `ver`

Darwin xtaixe 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Output of `java -version`

openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05)
OpenJDK 64-Bit Server VM GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05, mixed mode, sharing)

Quarkus version or git rev

2.0.1.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /Users/xtaixe/.mvnvm/apache-maven-3.8.1
Java version: 11.0.11, vendor: GraalVM Community, runtime: /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.1.0/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.4", arch: "x86_64", family: "mac"

Issue Analytics

State:
Created 2 years ago
Comments:24 (12 by maintainers)

Top GitHub Comments

1reaction

xtaixecommented, Mar 17, 2022

Hey, finally got time to properly look into this again and I got it working, but quarkus.package.write-transformed-bytecode-to-build-output seems to be incompatible with the Quarkus JaCoCo extension. When using both, JaCoCo complains again about classes not being the same and in this case it fails to report the proper coverage for classes covered by NON integration tests.

The only way I got both unit and integration tests reporting the right coverage together (on a very simple maven single module project) was by:

Using quarkus.package.write-transformed-bytecode-to-build-output.
Removing the Quarkus JaCoCo extension.
Removing <exclClassLoaders>*QuarkusClassLoader</exclClassLoaders> from the jacoco-maven-plugin configurations.

Attaching working single module project for reference: tests-with-coverage-quickstart-single-module.zip

On a multi module project with an aggregated JaCoCo report though, there’s always some incompatibilities between the classes that were unit tested and the classes that JaCoCo sees when doing the report (or the other way around if for example I tell Quarkus to not remove “unused” classes). After coming to know that the classes are copied during the package phase, I’m not sure it’s going to be possible to aggregate unit and integration test reports at all, and I have no idea why everything just works in a single module project.

cc @geoand and @stuartwdouglas in case you have some comments/suggestions or think it’s worth to create some issues.

0reactions

pipinetcommented, Aug 21, 2022

Hey, finally got time to properly look into this again and I got it working, but quarkus.package.write-transformed-bytecode-to-build-output seems to be incompatible with the Quarkus JaCoCo extension. When using both, JaCoCo complains again about classes not being the same and in this case it fails to report the proper coverage for classes covered by NON integration tests.

The only way I got both unit and integration tests reporting the right coverage together (on a very simple maven single module project) was by:

Using quarkus.package.write-transformed-bytecode-to-build-output.

Removing the Quarkus JaCoCo extension.

Removing <exclClassLoaders>*QuarkusClassLoader</exclClassLoaders> from the jacoco-maven-plugin configurations.

Attaching working single module project for reference: tests-with-coverage-quickstart-single-module.zip

On a multi module project with an aggregated JaCoCo report though, there’s always some incompatibilities between the classes that were unit tested and the classes that JaCoCo sees when doing the report (or the other way around if for example I tell Quarkus to not remove “unused” classes). After coming to know that the classes are copied during the package phase, I’m not sure it’s going to be possible to aggregate unit and integration test reports at all, and I have no idea why everything just works in a single module project.

cc @geoand and @stuartwdouglas in case you have some comments/suggestions or think it’s worth to create some issues.