从URL查找字符串中所有匹配的单词

Question

I'm trying to write my first python script. I want to write a program that will get information out of a website.

I managed to open the website, read all the data and transform the data from bytes to a string.

import urllib.request

response = urllib.request.urlopen('http://www.imdb.com/title/tt0413573/episodes?season=10')
website = response.read()
response.close()

html = website.decode("utf-8")

print(type(html))
print(html)

The string is massive, I don't know if I show transform it to a list and iterate over the list or just keep it as a string.

What I would like to do if find all the keyword airdate and them get the next line in the string.

When I scroll through the string this is the relevant bits:

<meta itemprop="episodeNumber" content="10"/>
<div class="airdate">
  Nov. 21, 2013
</div>

This happens lots of times inside the string. What I'm trying to do is to loop through the string and return this result:

"episodeNumber" = some number
"airdate" = what ever date

For overtime this happens in the string. I tried:

keywords = ["airdate","episodeNumber"]
for i in keywords:
    if i in html:
        print (something)

I hope I'm explaining myself in the right way. I will edit the question if needed.

Answer 1

When dealing with structured texts like HTML/XML it is a good idea to use existing tools that leverage this structure. Instead of using regex or searching by hand, this gives a much more reliable and readable solution. In this case, I suggest to install lxml to parse the HTML.

Applying this principle to your problem, try the following (I assume that you use Python 3 because you imported urllib.request):

import lxml.html as html
import urllib.request

resp = urllib.request.urlopen('http://www.imdb.com/title/tt0413573/episodes?season=10')

fragment = html.fromstring(resp.read())

for info in fragment.find_class('info'):
    print('"episodeNumber" = ', info.find('meta').attrib['content'])
    print('"airdate" =', info.find_class('airdate')[0].text_content().strip())

To make sure that the episode number and airdate are corresponding, I search for the surrounding element (a div with class 'info') and then extract the data you want.

I'm sure the code can be made prettier with a fancier selection of elements, but this should get you started.

---

[Added more information on the solution concerning the structure in the HTML.]

The string containing the data of one episode looks as follows:

<div class="info" itemprop="episodes" itemscope itemtype="...">
  <meta itemprop="episodeNumber" content="1"/>
  <div class="airdate">Sep. 26, 2013</div> <!-- already stripped whitespace -->
  <strong>
    <a href="/title/tt2911802/" title="Seal Our Fate" itemprop="name">...</a>
  </strong>
  <div class="item_description" itemprop="description">...</div>
  <div class="popoverContainer"></div>
  <div class="popoverContainer"></div>
</div>

You first select the div containing all data of one episode by its class 'info'. The first information you want is in a child of the div.info element, the meta element, stored in its property 'content'.

Next, you want the information stored in the div.airdate element, this time it is stored inside the element as text. To get rid of the whitespace around it, I then used the strip() method.

Answer 2

Would that work?

lines = website.splitlines()
lines.append('')
for index, line in enumerate(lines):
    for keyword in ["airdate","episodeNumber"]:
        if keyword in line:
            print(lines[index + 1])

It prints the next line if the keyword is found in the line.

Answer 3

If that is your first Python script, it is really impressive to see you have made so far.

You will use some legit parser to help you with your parsing.

Check out BeautifulSoup4

# intellectual property belongs to imdb    
import urllib2
from bs4 import BeautifulSoup

# get the SOUP: tree structure out of the HTML page
soup = BeautifulSoup(urllib2.urlopen("http://www.imdb.com/title/tt0413573/episodes?season=10"))

result = {}
for div in soup.find_all("div", {"class":"airdate"}):
    # get the date and number and store in a dictionary
    date = div.text.encode('utf-8').strip()
    number = div.find_previous_sibling()['content']
    result[number] = date

print result

output

{'10': 'Nov. 21, 2013', '1': 'Sep. 26, 2013', '3': 'Oct. 3, 2013', '2': 'Sep. 26, 2013', '5': 'Oct. 17, 2013', '4': 'Oct. 10, 2013', '7': 'Oct. 31, 2013', '6': 'Oct. 24, 2013', '9': 'Nov. 14, 2013', '8': 'Nov. 7, 2013'}

Let me know if I understood and answered your question correctly.