python regex：从文件读取时，regex失败

Question

My regex

table_names = ['check_channel_types', 'qualitycheckresult']
tables_group = '|'.join(table_names)
pattern = re.compile('(CREATE TABLE "({0})"(.*?);)'.format(tables_group), re.DOTALL)
match = pattern.findall(s)

works fine with this test-string:

s ="""CREATE TABLE "check_boundary_nodes" (
    "id" serial NOT NULL PRIMARY KEY,
    "test_name" varchar(120),
    "field_name" varchar(120),
    "original_pk" varchar(15),
    "check_result" varchar(255),
    "constraint" varchar(120),
    "the_geom" geometry(GEOMETRY,28992)
)
;
CREATE TABLE "check_channel_types" (
    "id" serial NOT NULL PRIMARY KEY,
    "original_pk" integer CHECK ("original_pk" >= 0) NOT NULL,
    "channel_inp_id" integer CHECK ("channel_inp_id" >= 0),
    "type" integer CHECK ("type" >= 0),
    "suggested_type" integer CHECK ("suggested_type" >= 0),
    "the_geom" geometry(LINESTRING,28992)
)
;
CREATE TABLE "qualitycheckresult" (
    "id" serial NOT NULL PRIMARY KEY,
    "qualitycheck" varchar(512) NOT NULL,
    "created" timestamp with time zone NOT NULL,
    "result" integer NOT NULL,
    "resultvalue" varchar(256) NOT NULL,
    "message" varchar(512) NOT NULL,
    "object_id" integer,
    "object_type" varchar(512) NOT NULL,
    "old_value" text NOT NULL
)
;"""  

Once I read the text from a file-like object, the regular expression fails (does not find any matches). I assume it has to do with the quote characters but find it hard to debug as the string I'm reading from the 'file' is very long. What really feels strange about this is that internally it should not make a difference if it is a triple quoted string or not. Any help is highly appreciated. This is how I retrieve the data in my app:

from StringIO import StringIO
content = StringIO()
call_command('sql', 'my_app', database=self.alias,
             stdout=content)
content.seek(0)
a = content.getvalue()
type(a)
>>> <type 'str'>

Answer 1

Try to write the output as utf8 encoding. And add the flags:

re.MULTILINE|re.DOTALL

To the regex.

f_stream = open( “my_dumpfile.txt”,     ‘w’, encoding=”utf-8″ )
call_command( “dumpdata”, indent=4, stdout=f_stream )

Or:

content = StringIO(content.read().decode('utf8'))

Or:

a = content.read().decode('utf8')

Answer 2

Ok, let's wrap this up.

The reason the regex fails when reading from the StringIO object is that the strings contain ANSI escape sequences (apparently to give certain lines a different color). This answer shows how to remove the escape sequences using a regular expression. Then everything works just like expected.