Python - 正则表达式“机器学习”

Question

I have thousands of lines of text where I need to find money-representations eg:

Lorem ipsum dolor sit amet, 100.000,00 USD sadipscing elitr, sed diam nonumy eirmod 
GBP 400 ut labore et dolore magna aliquyam erat, sed diam voluptua. At USD 20 eos et 
accusam et justo duo dolores et 100,000.00 USD  ea rebum. Stet 3,-- USD gubergren, no 

The Python script should return the amount converted to USD. (eg 100000USF, 400 GBP -> USD, etc)

What I did so far was manually creating Regular expressions for number - currency combinations to retreive the value, then compare the currency against a database and calculate the exchange.

However, this is neither efficient nor future proof (eg if another currency is added) So I'm wondering wether there is an efficient machine learning algorithm that I could "train" with some examples and it then tries to find sich "value - currency" combinations?

Answer 1

Can a human even learn if an acronym is a currency? if a new currency pops up then how is it distinguishable from any other arbitrary acronym? Say you come across something like "1000 CPU", how could you tell if that is (or isn't) currency if you don't know what a CPU is?

You could use natural language processing to look at the context around the number in question, but it's going to take more processing and you'll never know for sure.

My point is: for this problem machine learning is overkill, if it is even applicable.

Why do something the hard way when it is substantially easier and more accurate to do it another way?

Answer 2

Your problem is not well defined, but there is no need for machine learning. The set of possible currencies is finite and small, and the set of currency representations can not be so complicated as to not be expressible as a regular expression. You simply are not employing the full power of regular expressions.

For example, to match multiple currencies, use:

    currency = r"((USD)|(GBP)(...))"

You can then express the number part of the representation

    numbers = r"([0-9]+[0-9\.,]*)"

Compile the regular expression:

    matcher = re.compile(numbers+r"[\s]*+"currency)

You can create a second matcher that matches the currencies first. You might be able to use something clever with optional capture groups and such, but I would recommend a simple second matcher if performance isn't a big issue.

    matcher2 = re.compile(currency+r"[\s]*"+numbers)

Note that the 'currency' regex need not be created manually. Once you have a match, you can access the appropriate group number (1 or 3) to get the matched currency. For example:

    curren = m.group(1)
    amount = m.group(2)

This is possible since the entire 'currency' regex gets treated as a single group.

Unless there are infinite patterns of money representations in your input (probably impossible), your problem can definitely be tackled with appropriate regular expressions.

Answer 3

I would just use regex to crudely extract possible pairs:

import re

test = '''Lorem ipsum dolor sit amet, 100.000,00 USD sadipscing elitr, sed diam nonumy eirmod 
GBP 400 ut labore et dolore magna aliquyam erat, sed diam voluptua. At USD 20 eos et 
accusam et justo duo dolores et 100,000.00 USD  ea rebum. Stet 3,-- USD gubergren, no'''

number = r'([\d+.,]+)'
currency = r'([A-Z]{2,3})'

r1 = re.compile(number + r'\s+' + currency)
r2 = re.compile(currency + r'\s+' + number)

matches = r1.findall(test) + r2.findall(test)

print(matches)

I get:

[('100.000,00', 'USD'), ('100,000.00', 'USD'), ('GBP', '400'), ('USD', '20')]

From there, you can parse the numbers and filter out currencies that don't exist. You've got only five or six possible formats, so there's really nothing machine learning can do for you here.