limits.fileSize not working when greater than a specified value

See original GitHub issue

Prerequisites

  • I have written a descriptive issue title
  • I have searched existing issues to ensure it has not already been reported

Fastify version

3.8.1

Plugin version

4.0.7

Node.js version

14.17.1

Operating system

Linux

Operating system version (i.e. 20.04, 11.3, 10)

Manjaro 21

Description

When fileSize is greater than a specified value, it will not throw exception when upload large file.

Steps to Reproduce

Here is the sample code just from README

const fastify = require('fastify')()
const fs = require('fs')
const util = require('util')
const path = require('path')
const { pipeline } = require('stream')
const pump = util.promisify(pipeline)

fastify.register(require('fastify-multipart'))

fastify.post('/', async function (req, reply) {
  // process a single file
  // also, consider that if you allow to upload multiple files
  // you must consume all files otherwise the promise will never fulfill
  console.log('before req file');
  // {limits: {fileSize: 1024}}
  const data = await req.file({limits: {fileSize: 40 * 1024}})
  console.log(data);
  console.log('after req file');

  // to accumulate the file in memory! Be careful!
  //
  // await data.toBuffer() // Buffer
  //
  // or

  console.log('before pump')
  await pump(data.file, fs.createWriteStream(data.filename))
  console.log('after pump')
  // be careful of permission issues on disk and not overwrite
  // sensitive files that could cause security risks
  
  // also, consider that if the file stream is not consumed, the promise will never fulfill

  reply.send()
})

fastify.listen(3000, err => {
  if (err) throw err
  console.log(`server listening on ${fastify.server.address().port}`)
})

Test with curl:

curl -v -F 'file=@BIG_FILE' localhost:3000/

When fileSize is a small one like 40 * 1024(40KB), it’s working, response:

{"statusCode":413,"code":"FST_REQ_FILE_TOO_LARGE","error":"Payload Too Large","message":"request file too large, please check multipart config"}

But when fileSize is a large one like 40 * 1024 * 1024(40MB), it’s not working, the file is uploaded successfully, but the uploaded file is truncated to 40MB only.

Expected Behavior

limits.fileSize should working as expect.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

3reactions
leorossicommented, Jul 28, 2021

So here’s my conclusions on this: the library work as expected but maybe docs should be slightly more clear on this.

 await pump(data.file, fs.createWriteStream(data.filename))

this line does NOT guarantee that the file has been completely written on disk. In fact, after this line you should check data.file.truncated, if it’s false then all the file has been streamed, otherwise the limit has been reached.

const filename = data.filename + 'output';
await pump(data.file, fs.createWriteStream(filename))
if (data.file.truncated) {
  console.log('file is truncated');
  fs.unlinkSync(filename);
  return reply.send(new fastify.multipartErrors.FilesLimitError());
} else {
  return reply.send({ message: 'file accepted' });
}

Alternatively you can also use req.saveRequestFiles({ limits: { fileSize: YOUR_FILESIZE_LIMIT } }) and that would handle the error for you.

I will update the docs and create a PR to underline this fact.

1reaction
abcfy2commented, Jul 1, 2021

Seems that when fileSize is large, the file.truncated is false before the stream reading.

// ...
  const data = await req.file({limits: {fileSize: 1024*1024*40}, throwFileSizeLimit: true})
  const file = data.file;
  console.log('file.truncated before pump: ' + file.truncated)
  await pump(data.file, fs.createWriteStream(data.filename))
  console.log('file.truncated after pump: ' + file.truncated);
// ...

Output:

file.truncated before pump: false
file.truncated after pump: true
Read more comments on GitHub >

github_iconTop Results From Across the Web

How to limit the file size when uploading with multer?
This all works fine and the file gets uploaded. The only thing that is not working is the limit on the max size....
Read more >
How to Fix the uploaded file exceeds the upload_max_filesize ...
A step-by-step guide for how to fix the "the uploaded file exceeds the upload_max_filesize directive in php.ini" error in WordPress.
Read more >
Attachment size exceeds the allowable limit error - Outlook
The file you're attaching is bigger than the server allows. Try putting the file in a shared location and sending a link instead....
Read more >
How to Fix the upload_max_filesize Error in WordPress
Running into the uploaded file exceeds the upload_max_filesize directive in php.ini error? Read and learn how to fix it easily here!
Read more >
ulimit - Set process limits - IBM
The hard limit may be lowered to any value that is greater than or equal to the soft limit. ... -f: Set or...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found