Fits memmap, "buffer is too small for requested array" Error

See original GitHub issue

Hi Astropy developers, I am reading a big fits file using memmap method. However, I got a TypeError. has anyone seen this before? What is the correct way to read a big file (> 2GB)?

Thank you. Useage:

>>> x = fits.open("big_fits_file.fits", memmap=True)
>>> x
[<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7fc1830c9908>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7fc1830cf128>]
>>> x[1].data[0]
TypeError                                 Traceback (most recent call last)
<ipython-input-87-2cd5f5ec1899> in <module>()
----> 1 x[1].data[0]

~/Projects/astropy/astropy/utils/decorators.py in __get__(self, obj, owner)
    742                 return val
    743             else:
--> 744                 val = self.fget(obj)
    745                 obj.__dict__[self._key] = val
    746                 return val

~/Projects/astropy/astropy/io/fits/hdu/table.py in data(self)
    404     @lazyproperty
    405     def data(self):
--> 406         data = self._get_tbdata()
    407         data._coldefs = self.columns
    408         data._character_as_bytes = self._character_as_bytes

~/Projects/astropy/astropy/io/fits/hdu/table.py in _get_tbdata(self)
    170         else:
    171             raw_data = self._get_raw_data(self._nrows, columns.dtype,
--> 172                                           self._data_offset)
    173             if raw_data is None:
    174                 # This can happen when a brand new table HDU is being created

~/Projects/astropy/astropy/io/fits/hdu/base.py in _get_raw_data(self, shape, code, offset)
    473                               offset=offset)
    474         elif self._file:
--> 475             return self._file.readarray(offset=offset, dtype=code, shape=shape)
    476         else:
    477             return None

~/Projects/astropy/astropy/io/fits/file.py in readarray(self, size, offset, dtype, shape)
    332 
    333                 return np.ndarray(shape=shape, dtype=dtype, offset=offset,
--> 334                                   buffer=self._mmap)
    335             else:
    336                 count = reduce(operator.mul, shape)

TypeError: buffer is too small for requested array

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:18 (9 by maintainers)

github_iconTop GitHub Comments

2reactions
thrivethcommented, Feb 13, 2021

It was indeed corrupted. Sorry for the noise.

0reactions
suvenduatcommented, Jun 19, 2022

I am having the same issue while trying to append some keywords to the header using astropy.io.fits. WARNING: File may have been truncated: actual file length (2100032) is smaller than the expected size (2102400) [astropy.io.fits.file]

and the error: TypeError: buffer is too small for requested array

The files can be viewed using DS9 without any issue.

I got the following from fitsinfo: fitsinfo J0114p1325_R_300_1024x1024_4.fits WARNING: File may have been truncated: actual file length (2100032) is smaller than the expected size (2102400) [astropy.io.fits.file] Filename: J0114p1325_R_300_1024x1024_4.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 29 (1024, 1024) int16 (rescales to uint16) I posted this issue here: https://github.com/astropy/astropy/issues/13356

Any solution? I can post the code here that I using to append.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Error "buffer is too small for requested array" when modify a ...
fits has the table of the voice 1 with 3915 rows, instead, my new file, must have more then 50000 rows. The part...
Read more >
[Numpy-discussion] Bug in memmap/python allocation code?
... order=order) TypeError: buffer is too small for requested array >>> If I have a small number of frames (z=800 rather than 2011),...
Read more >
astropy.io.fits FAQ
I am opening many FITS files in a loop and getting OSError: Too many open ... module written in C that is required...
Read more >
pyfits 3.4 - PyPI
Reads FITS images and tables into numpy arrays and manipulates FITS headers. ... PyFITS separately in the short term, including any critical bug...
Read more >
"TypeError: buffer is too small for the requested array" while ...
The convert.py file was converting the Yolov4 weights to Keras. After converting 92 layers, I encountered the TypeError: buffer is too small ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found