OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

问题

I’m trying to load into memory a few 2 000 FITS using astropy.io.fits:

def readfits(filename):
    with fits.open(filename) as ft:
        # the fits contain a single HDU
        data = ft[0].data
    return data

data_sci = []
for i in range(2000):
    data_sci.append(readfits("filename_{}.fits".format(i)))

However, when reaching the 1015th file, OSError: [Errno 24] Too many open files is raised.

I have the same issue with:

def readfits(filename):
    ft = fits.open(filename) as ft:
    data = ft[0].data
    ft.close()
    return data

I suspect that astropy.io.fits does not properly close the file. Is there a way I can force the files to be closed?

回答1:

Your readfits function actually leaves the file handle open in order to keep access to the data, because by default it creates a mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Incidentally, if you just want a function that reads the data out of the first HDU this is already built in: http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata

It's not necessary to reinvent the wheel.

回答2:

After taking a look at the astropy documentation i found this: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

回答3:

You can call this function and store its output as long as you have memory. I thought it worths mentioning the answer explicitly but the credit goes to Iguananaut, bkaf, and this page.

def get_single_fits_data(fits_dir):

    hdul  = fits.open(fits_dir)
    for hdu in hdul:
        image_data = hdu.data.copy()
    hdul.close()
    gc.collect()

    return image_data

来源：https://stackoverflow.com/questions/32952428/oserror-24-too-many-open-files-when-reading-bunch-of-fits-with-astropy-io

标签

python

fits

astropy