io - Silent failure to write large files in python -

io - Silent failure to write large files in python -

i have programme uses cpickle dump big (several gb) files. appears work fine smaller files (the largest file worked 5.6 gb) fail silently (!) on larger files.

not sure if it's relevant, i'm using multiprocessing generate , write these files on several nodes simultaneously. suspect problem not specific (c)pickle i've experienced similar issues trying write out big files in hdf5 using h4py.

any thought might going on?

some additional info: i'm running on linux ext4 file system. should have more plenty ram (60 gb + 120 gb swap ... no memoryerror) , each process works on separate file. no process gets killed ... have loop calculates big file on each iteration , dumps it. if file's big doesn't dumped , process goes on the next iteration of loop without error message or other indication went wrong (except missing file).

here's minimal sample reproduces behavior:

import numpy np import cpickle import multiprocessing mp def test(): test = np.random.random(1000000000) # test = np.random.random(10000) # works cpickle.dump(test, open('/home/ctw/tmp/test.pickle','w'), -1) po = mp.pool(1) po.apply_async(test) po.close()

the result creation of empty file. when run outside multiprocessing environment, usual result creation of little (127b) file , next error message (i never see error when run using multiprocessing):

in [32]: cpickle.dump(test, open('/home/ctw/tmp/test.pickle','w'), -1) --------------------------------------------------------------------------- systemerror traceback (most recent phone call last) <ipython-input-32-62283b59a617> in <module>() ----> 1 cpickle.dump(test, open('/home/ctw/tmp/test.pickle','w'), -1) systemerror: error homecoming without exception set

another update: if utilize 0 instead of -1 protocol, file gets written.

apparently there's inherent limit on cpickle file sizes python <3.3 of 2**31-1 byte protocol. still annoying/strange there's no error message during multiprocessing.

python io multiprocessing large-files pickle