VipsJpeg: Invalid SOS parameters for sequential JPEG / with sharp(imageBuffer, originalFileData, { failOnError: false })

I am experiencing same error as described earlier: https://github.com/lovell/sharp/issues/1578 But I am using with image data from an S3 like storage as Buffer

Error on latest versions: sharp@0.28.2, sharp@0.28.3

This is how I use sharp, I added faileOnError: false according to #1578

sharp(imageBuffer, { failOnError: false })
        .rotate()
        .resize({
          width: dimension,
          height: dimension,
          fit: sharp.fit.cover,
          position: sharp.strategy.entropy,
        })
        .jpeg({ quality: 90, mozjpeg: true })
        .toBuffer()