这是我在stackoverflow中的第一个答案，所以请多多包涵。 我遇到了一个非常相似的问题，需要分析多个日志，其中一些日志非常庞大，无法完全放入内存中。 解决此问题的方法是创建一个数据处理管道，类似于unix / linux管道。背后的想法是将每个任务分解为各自的功能，并使用生成器来实现内存效率更高的方法。

import os
import gzip
import re
import fnmatch

def find_files(pattern,path):
    """
    Here you can find all the filenames that match a specific pattern
    using shell wildcard pattern that way you avoid hardcoding
    the file pattern i.e 'messages'
    """
    for root,dirs,files in os.walk(path):
        for name in fnmatch.filter(files,pattern):
            yield os.path.join(root,name)

def file_opener(filenames):
    """
    Open a sequence of filenames one at a time
    and make sure to close the file once we are done 
    scanning its content.
    """
    for filename in filenames:
        if filename.endswith('.gz'):
            f = gzip.open(filename,'rt')
        else:
            f = open(filename,'rt')
        yield f
        f.close()

def chain_generators(iterators):
    """
    Chain a sequence of iterators together
    """
    for it in iterators:
        # Look up yield from if you're unsure what it does
        yield from it

def grep(pattern,lines):
    """
    Look for a pattern in a line
    """
    pat = re.compile(pattern)
    for line in lines:
        if pat.search(line):
            yield line

# A simple way to use these functions together

logs = find_files('messages*','One/two/three')
files = file_opener(logs)
lines = chain_generators(files)
each_line = grep('keywords_1',lines)
for match in each_line:
    print(match)

如果您对我的回答有任何疑问，请告诉我