从数据集中提取文本

Question

I am working on a dataset in which I need to extract all the available dates. Dates can be of the following format:

04/20/2009; 04/20/09; 4/20/09; 4/3/09
Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;
20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009
Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009
Feb 2009; Sep 2009; Oct 2010
6/2008; 12/2009
2009; 2010

I wrote the below code:

df['dates'] = df['text'].str.extract(r'((?:\d{1,2}[/ ])?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec[a-z.,]*[- ])?(?:\d{1,2}[a-z-, /]{1,4})?(?:\d{2,4}))')

It is giving me the correct result except for some text like:

TEXT OUTPUT

Lab: B12 969 2007\n 12 969 #should give 2007

for 35 years, sold in 1985\n 35 #should give 1985

x 14 yrs who died i... 14 #should not be considered

I tried to change the extract code to

r'((?:\d{1,2}[/ ])?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec[a-z.,]*[- ])?(?:\d{1,2}[a-z-, ]{1,4})?(?:[/]\d{2})?(?:\d{4})?)' 

But with this entire result got bad

Answer 1

The problem with your regex is that its constituents are all optional and it matches numbers that are not actually related to dates. You need to build a regex with obligatory parts to avoid matching arbitrary parts.

And this is tricky: there are different types of dates in your sample input. For those inputs, I'd recommend:

(?<!\d)((?<!\d[ \t])(?:A(?:pr(?:il)?|ug(?:ust)?)|Dec(?:ember)?|Feb(?:ruary)?|J(?:an(?:uary)?|u(?:ly|ne|[ln]))|Ma(?:rch|[ry])|Nov(?:ember)?|Oct(?:ober)?|Sep(?:tember)?)(?:-\d{1,2}-\d{4}|(?:\.?\s*\d{1,2}(?:st|[rn]d|th)?,?)?\s*\d{4})|\d{1,2}\s+(?:A(?:pr(?:il)?|ug(?:ust)?)|Dec(?:ember)?|Feb(?:ruary)?|J(?:an(?:uary)?|u(?:ly|ne|[ln]))|Ma(?:rch|[ry])|Nov(?:ember)?|Oct(?:ober)?|Sep(?:tember)?)\.?,?\s*\d{4}|(?:\d{1,2}/)?\d{1,2}/\d{2}(?:\d{2})?|(?:19|20)\d{2})(?!\d)

See the regex demo . It matches:

• (?<!\d) - a negative lookbehind: no digit is allowed immediately to the left of the current location
• ( - start of the outer capturing group (necessary for .str.extract )
  • (?<!\d[ \t]) - no digit followed with space or tab immediately to the left of the current location is allowed
  • (?:A(?:pr(?:il)?|ug(?:ust)?)|Dec(?:ember)?|Feb(?:ruary)?|J(?:an(?:uary)?|u(?:ly|ne|[ln]))|Ma(?:rch|[ry])|Nov(?:ember)?|Oct(?:ober)?|Sep(?:tember)?) - names of months with their abbreviations
  • (?:-\d{1,2}-\d{4}|(?:\.?\s*\d{1,2}(?:st|[rn]d|th)?,?)?\s*\d{4}) - either of the two alternatives:
    • -\d{1,2}-\d{4} - - , 1 or 2 digits, - and then 4 digits
    • | - or
    • (?:\.?\s*\d{1,2}(?:st|[rn]d|th)?,?)? - an optional non-capturing group that matches 1 or 0 occurrences of:
    • \.? - an optional .
    • \s* - 0+ whitespaces
    • \d{1,2} - 1 or 2 digits
    • (?:st|[rn]d|th)? - an optional sequence of chars: st , r or n followed with d , or th
    • ,? - an optional comma
    • \s*\d{4} - 0+ whitespaces and then 4 digits
• | - or
  • \d{1,2}\s+ - 1 or 2 digits and then 1+ whitespaces
  • (?:A(?:pr(?:il)?|ug(?:ust)?)|Dec(?:ember)?|Feb(?:ruary)?|J(?:an(?:uary)?|u(?:ly|ne|[ln]))|Ma(?:rch|[ry])|Nov(?:ember)?|Oct(?:ober)?|Sep(?:tember)?) - names of months with their abbreviations (same as above)
  • \.? - an optional dot
  • ,? - an optional comma
  • \s* - 0+ whitespaces
  • \d{4} - four digits
• | - or
  • (?:\d{1,2}/)? - an optional sequence of 1 or 2 digits and then /
  • \d{1,2} - 1 or 2 digits
  • / - /
  • \d{2}(?:\d{2})? - 2 digits and an optional sequence of 2 digits (it allows 2 or 4 digits but not 3)
• | - or
  • (?:19|20) - 19 or 20
  • \d{2} - two digits
• ) - end of the puter caprturing group
• (?!\d) - a negative lookahead: no digit is allowed immediately to the right of the current location.

In Python, you may define blocks for the pattern and build it dynamically:

months = r'(?:A(?:pr(?:il)?|ug(?:ust)?)|Dec(?:ember)?|Feb(?:ruary)?|J(?:an(?:uary)?|u(?:ly|ne|[ln]))|Ma(?:rch|[ry])|Nov(?:ember)?|Oct(?:ober)?|Sep(?:tember)?)'
pattern = rf'(?<!\d)((?<!\d[ \t]){months}(?:-\d{{1,2}}-\d{{4}}|(?:\.?\s*\d{{1,2}}(?:st|[rn]d|th)?,?)?\s*\d{{4}})|\d{{1,2}}\s+{months}\.?,?\s*\d{{4}}|(?:\d{{1,2}}/)?\d{{1,2}}/\d{{2}}(?:\d{{2}})?|(?:19|20)\d{{2}})(?!\d)'

Answer 2

Try using pandas.to_datetime() , it converts the most common date formats to datetime objects.

Answer 3

Try this pattern. My suggestion is, you should decompose the problem into pieces and try to match one pattern at a time. Because the regex for this problem is a bit messy and hard to cover all the edge cases with one expression.

I've included the sub regexps so you can refine them to validate the edge cases.