我需要删除括号以进行标记化吗？ 正则表达式分词器

Question

First attempt at tokenization using nltk's RegexpTokenizer for an assignment (necessary). Not sure if I should remove brackets?

You are required to extract the token and append them into the list 'token'

...not sure if I even did this right.

import re
import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.probability import *
from itertools import chain
from tqdm import tqdm
import codecs
from nltk.corpus import stopwords 
nltk.download('stopwords')

df_text = pd.read_csv(r"C:\Users\User\Downloads\JobPostings.csv")

lower = []
for item in df_text['job_description']:
    lower = [item]
    lower.append(item.lower())

tokenizer_test = RegexpTokenizer(r"\s+", gaps=True)
tokens_test = tokenizer_test.tokenize(item)

token = [tokens_test]
print(token)

Output is:

[['Data', 'Scientist,', '(Staff', 'or', 'Principal)', 'at', 'realtor.com', '(View', 'all', 'jobs)', 'Santa', 'Clara,', 'CA', 'At', 'realtor.com,', 'we', 'process', 'terabytes', 'of', 'data', 'every', 'day', 'and', 'transform', 'that', 'data', 'into', 'information', 'that', 'powers', 'decisions', 'for', 'millions', 'of', 'homebuyers,', 'renters,', 'dreamers,', 'and', 'real', 'estate', 'professionals.', 'We', 'aim', 'to', 'radically', 'simplify', 'home', 'buying/selling', 'and', 'help', 'more', 'people', 'achieve', 'the', 'American', 'dream', 'on', 'our', 'realtor.com', 'website', 'and', 'mobile', 'apps.', 'We', 'seek', 'a', 'highly', 'seasoned', 'Data', 'Scientist', 'to', 'join', 'our', 'data', 'science', 'program', 'and', 'help', 'develop', 'it', 'to', 'its', 'full', 'potential.', 'As', 'a', 'key', 'member', 'of', 'the', 'data', 'science', 'team,', 'you', 'will', 'be', 'responsible', 'for', 'the', 'development', 'of', 'innovative', 'concepts,', 'research, ', 'predictive', 'modeling,', 'and', 'machine', 'learning', 'algorithms.', 'Responsibilities:', 'Perform', 'exploratory', 'analysis', 'on', "realtor.com's", 'wealth', 'of', 'data', 'including', 'consumer', 'web', 'and', 'mobile', 'behavior', 'and', 'North', 'America's', 'most', 'comprehensive', 'and', 'up-to-date', 'listings', 'and', 'properties', 'data', 'set.', 'Effectively', 'partner', 'with', 'product', 'and', 'engineering', 'teams', 'to', 'build', 'new', 'data-driven', 'and', 'machine', 'learning-based', 'features', 'in', 'our', 'professional', 'software', 'and', 'lead', 'monetization', 'products', 'to', 'enable', 'real', 'state', 'professionals', 'to', 'be', 'more', 'productive', 'and', 'effective', 'in', 'serving', 'the', 'needs', 'of', 'home', 'shoppers.', 'Help', 'improve', 'the', 'scope', 'our', 'data', 'sets', 'by', 'identifying', 'new', 'data', 'collection', 'and', 'procurement', 'opportunities', 'on', 'an', 'ongoing', 'basis', 'Drive', 'A/B,', 'multivariate', 'tests', 'and ', 'design', 'of', 'experiments', 'to', 'facilitate', 'testing', 'of', 'new', 'product', 'and', 'design', 'features,', 'with', 'a', 'focus', 'on', 'improving', 'engagement,', 'retention,', 'and', 'conversion.', 'Select,', 'apply,', 'and', 'tune', 'a', 'diverse', 'set', 'of', 'tools', 'to', 'coherently', 'solve', 'challenging', 'business', 'goals', 'Create', 'automated', 'learning', 'systems', 'that', 'gracefully', 'scale', 'to', 'increasing', 'complexity', 'and', 'expectation', 'Develop', 'predictive,', 'explanatory', 'models', 'and', 'machine', 'learning', 'algorithms', 'Generate', 'descriptive', 'visualizations', 'and', 'presentations', 'to', 'communicate', 'insights', 'Mentor', 'a', 'team', 'of', 'data', 'scientists', 'on', 'data', 'exploration,', 'machine', 'learning', 'and', 'developing', 'data-based', 'products', 'Work', 'with', 'a', 'sense', 'of', 'ownership', 'and', 'urgency,', 'advocate', 'for', 'experimentation', 'based,', 'agile', 'culture.', 'Requirements:', 'MS', 'or', 'Ph .D.', 'in', 'statistics,', 'mathematics,', 'operations', 'research,', 'computer', 'science,', 'quantitative', 'analysis,', 'economics', 'or', 'related', 'field', 'is', 'required.', '7+', 'years', 'of', 'relevant', 'experience', 'in', 'data', 'science,', 'data', 'analytics,', 'or', 'applied', 'statistics,', 'Experience', 'with', 'machine', 'learning,', 'NLP,', 'data', 'mining,', 'statistical', 'modeling', 'tools,', 'and', 'underlying', 'algorithms', 'Experienced', 'in', 'R,', 'Perl,', 'Python,', 'Spark,', 'or', 'other', 'languages', 'and', 'frameworks', 'appropriate', 'for', 'large', 'scale', 'analysis', 'of', 'numerical,', 'textual,', 'image,', 'and', 'video', 'data', 'Strong', 'skills', 'in', 'data', 'gathering,', 'massaging', 'and', 'featurization', 'Working', 'experience', 'with', 'relational', 'databases', 'and', 'SQL', 'Experience', 'with', 'experiment', 'design', 'and', 'A/B', 'and', 'multivariate', 'tests', 'Experience', 'and', 'proven', 'track', 'record', 'developing', 'online' , 'data', 'products', 'Strong', 'creative', 'thinking', 'and', 'problem-solving', 'skills', 'Excellent', 'oral', 'and', 'written', 'communication', 'and', 'presentation', 'skills']]

edit: tried this out instead... thoughts?

df_text_jd = df_text.job_description

lower = []
for item in df_text_jd:
    lower.append(item.lower().replace('(','').replace(')',''))

l = []  
for token in item:
    tokenizer_test = RegexpTokenizer(r'\s+', gaps=True)
    token = tokenizer_test.tokenize(item)

    l.append(token)

l

Answer 1

You can replace the brackets by modifying the line where you append the lower case item to lower list:

lower.append(item.lower().replace('(','').replace(')',''))