我有这个小脚本,可以将一堆文本数据以gzip的形式从源转储到磁盘。我从工作中获得的大多数资源都没有问题,但是我遇到了一个抛出JavaScript heap out of memory
的问题。
下面是它的摘要
const fs = require('fs');
const zlib = require('zlib');
const file = fs.createWriteStream('file.gz');
const gzip = zlib.createGzip();
gzip.pipe(file);
// ... code to connect to someDataSource would be here
someDataSource.on('data',(line) => { // feeding lines of text
gzip.write(line);
});
someDataSource.on('done',() => {
// crashes before this point
gzip.end();
});
我怀疑zlib
模块的缓冲方式比刷新到磁盘之前要多。崩溃时,gz文件只有4MB
大。就像我在其他数据源上面所说的那样,我从工作中提取了数据,所有这些数据源都在50MB
之上生成gz文件。
模块上的文档在这里:https://nodejs.org/api/zlib.html#zlib_class_options
我不确定如何调整选项以使其正常工作。
崩溃:
<--- Last few GCs --->
[33692:0x10264e000] 97556 ms: Scavenge 1370.6 (1411.7) -> 1363.3 (1412.2) MB,4.5 / 0.0 ms (average mu = 0.174,current mu = 0.137) allocation failure
[33692:0x10264e000] 97569 ms: Scavenge 1371.0 (1412.2) -> 1363.7 (1413.7) MB,current mu = 0.137) allocation failure
[33692:0x10264e000] 97582 ms: Scavenge 1371.3 (1413.7) -> 1364.0 (1430.2) MB,current mu = 0.137) allocation failure
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0xdd88f3dbe3d]
Security context: 0x32b80cc1e6e9 <JSObject>
1: /* anonymous */(aka /* anonymous */) [0x32b897904941] [/some/path/node_modules/tedious/lib/token/stream-parser.js:~154] [pc=0xdd88f6fbec4](this=0x32b8101826f1 <undefined>)
2: valueParse(aka valueParse) [0x32b8c73a8ab9] [/some/path/node_modules/tedious/lib/value-parser.js:~74] [pc=0xdd88f6c96d3](this=0x32b8101826f1 ...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x10003c597 node::Abort() [/usr/local/bin/node]
2: 0x10003c7a1 node::OnFatalError(char const*,char const*) [/usr/local/bin/node]
3: 0x1001ad575 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*,char const*,bool) [/usr/local/bin/node]
4: 0x100579242 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
5: 0x10057bd15 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long,double) [/usr/local/bin/node]
6: 0x100577bbf v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,v8::GCCallbackflags) [/usr/local/bin/node]
7: 0x100575d94 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,v8::internal::GarbageCollectionReason,v8::GCCallbackflags) [/usr/local/bin/node]
8: 0x100574998 v8::internal::Heap::HandleGCRequest() [/usr/local/bin/node]
9: 0x10052a1c8 v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
10: 0x1007d9bb1 v8::internal::Runtime_StackGuard(int,v8::internal::Object**,v8::internal::Isolate*) [/usr/local/bin/node]
11: 0xdd88f3dbe3d
12: 0xdd88f6fbec4
13: 0xdd88f6c96d3
14: 0xdd88f6c8870
[1] 33692 abort node app.js