BeautifulSoup，尝试从包含作者姓名的锚标签中提取文本

Question

I am trying to scrape some data fromthis books site. I need to extract the title, and the author(s). I was able to extract the titles without much trouble. However, I am having issues to extract the authors when there are more than one, since they appear in the same line, and they belong to separate anchor tags within a header h4.

 <h4> "5 . " <a href="/items/705">The Elements of Style</a> " by " <a href="/authors/5107">William Strunk, Jr</a> ", " <a href="/authors/5108">EB White</a> </h4>

This is what I tried:

book_container = soup.find_all('li', class_='item pb-3 pt-3 border-bottom')

for container in book_container:

# title
title = container.h4.a.text
titles.append(title)

# author(s)
author_s = container.h4.find_all('a')
print('### SECOND FOR LOOP ###')
for a in author_s:
   
    if a['href'].startswith('/authors/'):
        
        print(a.text)
       

I'd like to have two authors in a tuple.

Answer 1

This might not be the most pythonic way, but it's a workaround.

newlist = []
for a in author_s:
    if a['href'].startswith('/authors/'):
        if len(author_s)>2:
            newlist.append(a.text)
            print(tuple(newlist))
        else:
            print(a.text)

I'm utilizing the fact that variable author_s would contain a list which we could check for more names. More than 2 in list, means more authors. (Alternatively, you could also check for the existence of newline in print)

You will also notice the printed output will have two tuples. Always extract the second one. The rest with one author will remain the same. Since this request do not have multiple lines of two authors, I couldn't check for complications.

Output:

[<a href="/items/705">The Elements of Style</a>, <a href="/authors/5107">William Strunk, Jr</a>, <a href="/authors/5108">E. B. White</a>]
### SECOND FOR LOOP ###
('William Strunk, Jr',)
('William Strunk, Jr', 'E. B. White')

Answer 2

You can extract all <a> links under <h4> (h4 is the tag where are title/authors). First <a> tag is the title, rest of <a> tags are the authors:

import requests
from bs4 import BeautifulSoup


url = 'https://thegreatestbooks.org/the-greatest-nonfiction-since/1900'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for item in soup.select('h4:has(>a)'):
    elements = [i.get_text(strip=True) for i in item.select('a')]
    title = elements[0]
    authors = elements[1:]
    print('{:<40} {}'.format(title, authors))

Prints:

The Diary of a Young Girl                ['Anne Frank']
The Autobiography of Malcolm X           ['Alex Haley']
Silent Spring                            ['Rachel Carson']
In Cold Blood                            ['Truman Capote']
The Elements of Style                    ['William Strunk, Jr', 'E. B. White']
The Double Helix: A Personal Account of the Discovery of the Structure of DNA ['James D. Watson']
Relativity                               ['Albert Einstein']
Look Homeward, Angel                     ['Thomas Wolfe']
Homage to Catalonia                      ['George Orwell']
Speak, Memory                            ['Vladimir Nabokov']
The General Theory of Employment, Interest and Money ['John Maynard Keynes']
The Second World War                     ['Winston Churchill']
The Education of Henry Adams             ['Henry Adams']
Out of Africa                            ['Isak Dinesen']
The Structure of Scientific Revolutions  ['Thomas Kuhn']
Dispatches                               ['Michael Herr']
The Gulag Archipelago                    ['Aleksandr Solzhenitsyn']
I Know Why the Caged Bird Sings          ['Maya Angelou']
The Civil War                            ['Shelby Foote']
If This Is a Man                         ['Primo Levi']
Collected Essays of George Orwell        ['George Orwell']
The Electric Kool-Aid Acid Test          ['Tom Wolfe']
Civilization and Its Discontents         ['Sigmund Freud']
The Death and Life of Great American Cities ['Jane Jacobs']
Selected Essays of T. S. Eliot           ['T. S. Eliot']
A Room of One's Own                      ['Virginia Woolf']
The Right Stuff                          ['Tom Wolfe']
The Road to Serfdom                      ['Friedrich von Hayek']
R. E. Lee                                ['Douglas Southall Freeman']
The Varieties of Religious Experience    ['Will James']
The Liberal Imagination                  ['Lionel Trilling']
Angela's Ashes: A Memoir                 ['Frank McCourt']
The Second Sex                           ['Simone de Beauvoir']
Mere Christianity                        ['C. S. Lewis']
Moveable Feast                           ['Ernest Hemingway']
The Autobiography of Alice B. Toklas     ['Gertrude Stein']
The Origins of Totalitarianism           ['Hannah Arendt']
Black Lamb and Grey Falcon               ['Rebecca West']
Orthodoxy                                ['G. K. Chesterton']
Philosophical Investigations             ['Ludwig Wittgenstein']
Night                                    ['Elie Wiesel']
The Affluent Society                     ['John Kenneth Galbraith']
Mythology                                ['Edith Hamilton']
The Open Society                         ['Karl Popper']
The Color of Water: A Black Man's Tribute to His White Mother ['James McBride']
The Seven Storey Mountain                ['Thomas Merton']
Hiroshima                                ['John Hersey']
Let Us Now Praise Famous Men             ['James Agee']
Pragmatism                               ['Will James']
The Making of the Atomic Bomb            ['Richard Rhodes']