Python：我的脚本不允许我创建大文件

Question

I'm making a small python script which will create random files in all shapes and sizes but it will not let me create large files. I want to be able to create files up to around 8GB in size, I know this would take a long amount of time but I'm not concerned about that.

The problem is that Python 2.7 will not handle the large numbers I am throwing at it in order to create the random text that will fill my files.

The aim of my code is to create files with random names and extentions, fill the files with a random amount of junk text and save the files. It will keep on repeating this until I close the command line window.

import os
import string
import random


ext = ['.zip', '.exe', '.txt', '.pdf', '.msi', '.rar', '.jpg', '.png', '.html', '.iso']

min = raw_input("Enter a minimum file size eg: 112 (meaning 112 bytes): ")
minInt = int(min)

max = raw_input("Enter a maximum file size: ")
maxInt = int(max)

def name_generator(chars=string.ascii_letters + string.digits):
    return ''.join(random.choice(chars) for x in range(random.randint(1,10)))

def text_generator(chars=string.printable + string.whitespace):
    return ''.join(random.choice(chars) for x in range(random.randint(minInt,maxInt)))

def main():
    fileName = name_generator()
    extension = random.choice(ext)
    file = fileName + extension

    print 'Creating ==> ' + file
    fileHandle = open ( file, 'w' )
    fileHandle.write ( text_generator() )
    fileHandle.close()
    print file + ' ==> Was born!'

while 1:
    main()

Any help will be much appreciated!

Answer 1

Make it lazy, as per the following:

import string
import random
from itertools import islice

chars = string.printable + string.whitespace
# make infinite generator of random chars
random_chars = iter(lambda: random.choice(chars), '')
with open('output_file','w', buffering=102400) as fout:
    fout.writelines(islice(random_chars, 1000000)) # write 'n' many

Answer 2

The problem is not that python cannot handle large numbers. It can.

However, you try to put the whole file contents in memory at once - you might not have enough RAM for this and additionally do not want to do this anyway.

The solution is using a generator and writing the data in chunks:

def text_generator(chars=string.printable + string.whitespace):
    return (random.choice(chars) for x in range(random.randint(minInt,maxInt))

for char in text_generator():
    fileHandle.write(char)

This is still horribly inefficient though - you want to write your data in blocks of eg 10kb instead of single bytes.

Answer 3

A comment about performance: you could improve it by using os.urandom() to generates random bytes and str.translate() to translate them into the range of input characters:

import os
import string

def generate_text(size, chars=string.printable+string.whitespace):
    # make translation table from 0..255 to chars[0..len(chars)-1]
    all_chars = string.maketrans('', '')
    assert 0 < len(chars) <= len(all_chars)
    result_chars = ''.join(chars[b % len(chars)] for b in range(len(all_chars)))

    # generate `size` random bytes and translate them into given `chars`
    return os.urandom(size).translate(string.maketrans(all_chars, result_chars))

Example:

with open('output.txt', 'wb') as outfile: # use binary mode
    chunksize = 1 << 20  # 1MB
    N = 8 * (1 << 10)    # (N * chunksize) == 8GB
    for _ in xrange(N):
        outfile.write(generate_text(chunksize))

Note: to avoid skewing the random distribution, bytes larger than k*len(chars)-1 returned by os.urandom() should be discarded, where k*len(chars) <= 256 < (k+1)*len(chars) .