S3: Data read has a different checksum than expected

Hi. Unfortunately, I am still able to reproduce the issue on the SDK version 2.15.11. I prepared a demo project to show the issue.

The project has 3 tests. First to upload ~100kB array with checksum validation. This is the only green test in the scope. Both other tests upload ~1mB array with and without checksum validation. In the test with checksum validation, I get mentioned “Data read has a different checksum than expected” error. Without checksum validation bytes are uploaded to s3 but they are different from the original byte array.

Looks like the packets still be reordered as @marc-christian-schulze mentioned.

FWIW, we’ve encountered this issue too when calling getObject, which was due to S3AsyncClient.clientConfiguration having a duplicated set of default interceptors.

Apparently there can’t be more than one AsyncChecksumValidationInterceptor in the chain, as it will remove the checksum part and break the second check, which is why all those exceptions were expecting 0.

In DefaultS3BaseClientBuilder, it calls finalizeServiceConfiguration when building the client:

ClasspathInterceptorChainFactory interceptorFactory = new ClasspathInterceptorChainFactory();
List<ExecutionInterceptor> interceptors = interceptorFactory
        .getInterceptors("software/amazon/awssdk/services/s3/execution.interceptors");
interceptors = CollectionUtils.mergeLists(interceptors, config.option(SdkClientOption.EXECUTION_INTERCEPTORS));
return config.toBuilder().option(SdkClientOption.EXECUTION_INTERCEPTORS, interceptors).build();

ClasspathInterceptorChainFactory.getInterceptors uses its classloader’s getResources to fetch a list of ExecutionInterceptor, and for some reason it returned duplicated set of interceptors.

This was verified in our side by calling

Collections.list(
    new ClasspathInterceptorChainFactory().getClass().getClassLoader()
          .getResources("software/amazon/awssdk/services/s3/execution.interceptors")
).forEach(System.out::println);

Which prints out two same lines.

Looks like it’s a not-so-common bug, we had our servlet running as ROOT webapp in tomcat, renaming ROOT to something else fixed it.

However I would suggest changing the behavior of ClasspathInterceptorChainFactory.createExecutionInterceptorsFromClasspath, to check for duplication before returning. @millems