我正在尝试使用multiprocessing.RawArray
在共享内存中分配一组图像缓冲区。它适用于少量图像。但是,当我到达一定数量的缓冲区时,我得到一个OSError
指示我的内存不足。
一个明显的问题,我实际上是内存不足吗?根据我的计算,我要分配的缓冲区应该大约有1 GB的内存,并且根据Windows Task Manager,我有大约20 GB的可用空间。我看不到我实际上怎么可能会内存不足!
我是否达到了某种我可以增加的人造内存消耗限制?如果没有,为什么会这样,我该如何解决?
我正在使用Windows 10,Python 3.7、64位体系结构,总共32 GB RAM。
这是一个最小的可重现示例:
import multiprocessing as mp
import ctypes
imageDataType = ctypes.c_uint8
imageDataSize = 1024*1280*3 # 3,932,160 bytes
maxBufferSize = 300
buffers = []
for k in range(maxBufferSize):
print("Creating buffer #",k)
buffers.append(mp.RawArray(imageDataType,imageDataSize))
输出:
Creating buffer # 0
Creating buffer # 1
Creating buffer # 2
Creating buffer # 3
Creating buffer # 4
Creating buffer # 5
...等等...
Creating buffer # 278
Creating buffer # 279
Creating buffer # 280
Traceback (most recent call last):
File ".\Cruft\memoryErrorTest.py",line 10,in <module>
buffers.append(mp.RawArray(imageDataType,imageDataSize))
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py",line 129,in RawArray
return RawArray(typecode_or_type,size_or_initializer)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py",line 61,in RawArray
obj = _new_value(type_)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py",line 41,in _new_value
wrapper = heap.BufferWrapper(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py",line 263,in __init__
block = BufferWrapper._heap.malloc(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py",line 242,in malloc
(arena,start,stop) = self._malloc(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py",line 134,in _malloc
arena = Arena(length)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py",line 38,in __init__
buf = mmap.mmap(-1,size,tagname=name)
OSError: [WinError 8] Not enough memory resources are available to process this command