尝试使用 Selenium 提取 Python 中的文本时无法定位元素

Question

我正在尝试获取 bloomberg 新闻库下的所有文本内容。 下面的代码使用一个示例 url。

from selenium import webdriver
from fake_useragent import UserAgent
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
  
# Chrome Path
ua = UserAgent()
userAgent = ua.random
options = Options()
#options.add_argument('--headless')
options.add_argument(f'user-agent={userAgent}')
browser = webdriver.Firefox(options=options)
input_url = "https://www.bloomberg.com/news/articles/2015-11-06/the-true-story-of-soul-city-a-utopian-town-built-for-african-americans-with-republican-support-in-the-early-1970s?srnd=citylab-housing"
print(input_url)
browser.get(input_url)
  
element = browser.find_element(by=By.CLASS_NAME, value='body-content fence-body')
  
# Get Text
print(element.text)
  
# Close the window
browser.close()

但是，当 HTML 文件中存在该标记时，我得到Unable to locate element: .body-content fence-body 。 这是为什么？

Answer 1

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 10)
input_url = "https://www.bloomberg.com/news/articles/2015-11-06/the-true-story-of-soul-city-a-utopian-town-built-for-african-americans-with-republican-support-in-the-early-1970s?srnd=citylab-housing"
driver.get(input_url) 
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@title='SP Consent Message']")))
wait.until(EC.element_to_be_clickable((By.XPATH,"//button[@title='Yes, I Accept']"))).click()
driver.switch_to.default_content()
body=wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".body-content.fence-body")))
print(body.text)

您只需单击全部接受，然后等待文本的可见性。 全部接受在 iframe 中，因此切换到它并退出，然后获取适当的元素。 （使用 Chrome 尚未与 Firefox 核对）

产出

The hip-hop radio ads coming out of Ben Carson’s presidential campaign this week, to much laughter and derision, represent what Republican outreach to African Americans often looks like these days. But there was a time when Republicans took diversifying their base much more seriously. In the early 1970s, even President Richard Nixon’s administration, with its uniquely conservative, “law and order” bent, experienced what Ohio State University professor Devin Fergus calls “paroxysms of progressivism.” Such progressive spurts from the Nixon administration were responsible for producing the Environmental Protection Agency, the Clean Water Act—and Soul City.
Soul City was a project dreamed up by the civil rights activist Floyd McKissick. The fact that it went forward with financial assistance from Nixon represented one of the disgraced president’s “lapsed moments of liberalism,” as Fergus calls it in his 2010 Journal of Policy History article, “Black Power, Soft Power.” The idea was a city built for African Americans (though not exclusively), steered by black interests, and funded upfront by the federal government. It was, in essence, a request for the federal government to make good on the unrealized 40 acres and a mule promise made to African Americans emancipated from slavery. And President Nixon was all for it.
Soul City still exists today in North Carolina, right near the Virginia border, but it never became the racial Xanadu McKissick sold it as. A number of problems contributed to its failures, many of them beyond city planners’ control. A look back at the rise and fall of Soul City offers a number of lessons useful today when considering what’s at stake when targeted federal funding is deployed to address problems that are racial or social in nature.  
Soul City, North Carolina
More from
Bloomberg
Citylab
What Should Ukraine’s Reconstruction Look Like?
Hong Kong’s Star Ferry Needs a Rescue Plan With No Tourists in Sight
What Does Affordable Housing Do to Nearby Property Values?
How Public Schools Are Going Net Zero
The deal for a black city made between McKissick and Nixon was, by all appearances, political. McKissick switched parties in the late 1960s to support Nixon for president, which was a surprise given his civil rights and labor union credentials. As president of the Congress of Racial Equality throughout the ‘60s, McKissick shifted the civil rights organization’s mission from nonviolent tactics to Black Power doctrine. After the 1968 assassination of Martin Luther King, Jr., McKissick came to believe that even violent self-defense was inadequate for African Americans to realize real empowerment. He began pushing the idea that African Americans needed a strategy built squarely on capitalism to counter the entrenched racism that fueled urban neglect and the destitute conditions of black neighborhoods.
To this end, McKissick began in 1969 drafting the plans for Soul City, which would be a place where African Americans would not be regularly subjected to racial discrimination, and where they could determine their own political and economic destinies. The city would grow through a regiment of intensive workforce training aimed at boosting the entrepreneurial zeal of its residents, a “pull yourself up by your bootstraps” strategy made popular in the early 20th century by Booker T. Washington.
McKissick had first taken his vision to outgoing President Lyndon B. Johnson, who gave it lip service, but candidate Nixon ate it up whole. When Nixon reached the White House in 1969, McKissick had a shovel-ready plan for Soul City that he’d created alongside black MIT grad Harvey Gantt, a giant in architecture and urban planning circles and the eventual first African-American mayor of Charlotte, North Carolina. The city would be developed over a few thousand acres of land in Warren County, North Carolina, a poor, rural region that was quickly growing poorer due to the steady hemorrhaging of residents fleeing the South in search of economic and social opportunity.
Timothy J. Minchin, The North Carolina Historical Review
For McKissick, the overstuffing of Northern urban enclaves was at the heart of the problem of racial tension and urban unrest. With Soul City, he touted the co-benefit of creating a place that could absorb African-American migration while keeping them in the South. As he once said at a press conference, “The roots of the urban crisis are in the migratory pattern of rural people seeking to leave areas of economic and racial oppression. … So in building a new city in a rural area, we help to solve this.”
The Washington Post wrote in 1972 that Soul City was “perhaps the most vital experiment yet in this country’s halting struggle against the cancer of hectic urbanization.”
That year, McKissick had wrangled $14 million from the Nixon administration to get the Soul City party started, making him officially the first African American to develop a new city with federal funding. The funds were awarded by the U.S. Department of Housing and Urban Development, through a program designed to help build 14 new towns around the nation. The primary mission for these new towns was to help decongest cities. McKissick’s Soul City was chosen as one of those new towns, and he raised additional $6 million from other federal and state agencies that were interested in making it a success.
Black Enterprise
Given the for blacks, by blacks mission of Soul City, the public investment served essentially as reparations, or at least a security deposit for reparations. The political landscape of the time—sullen from the assassinations of King and Robert F. Kennedy—was more sympathetic to racial causes and conducive to making amends. Cities were still riding the federally subsidized waves of the New Deal and Great Society eras. Much of those federal windfalls skipped right past African Americans, though, and Nixon was willing to pick up some of that slack by handing out grants to enterprising African Americans, especially if it meant bringing the Republican Party more black votes.
Nixon aide Robert Brown referred to this strategy as “grantsmanship,” and it fell in line with the advice Nixon was receiving from people like Daniel Patrick Moynihan, whom Nixon had appointed his urban affairs counselor. While Nixon was concerned about the “Soul City” tag sounding perhaps too much like black militancy, McKissick was adamant about keeping the name, telling The Black Scholar journal in defense of his vision, “We need not justify any demands for ‘separatism’ to anybody white. … The real separatists moved to the suburbs long ago.”
Soul City would not, in fact, become a racially segregated enclave. HUD officials reminded McKissick that it would be illegal to construct a city that excluded any other race than blacks, and also that the federal government could not fund such a project as it would violate the Civil Rights Act. So, McKissick did change his tune. As Fergus writes in his article, McKissick flipped it around when reporters asked him about Soul City’s separatist veneer, saying, “we do not intend on adopting the white man’s racism” by reverse-redlining other races away from municipal boundaries.  
McKissick’s team had spent time in planned cities like Columbia, Maryland, studying their playbooks. Now with funding in place, they were able to break ground on basic infrastructure: a wastewater treatment plant, a municipal utility company, a health center, and an “industrial incubator facility” called “Soul Tech I,” which was designed to train the laborers, manufacturers, and entrepreneurs that McKissick envisioned as the leaders of Soul City.
Roger Biles, “The Rise and Fall of Soul City,” Journal of Planning History
The city was also to include three residential villages and a recreational center, but what McKissick wasn’t able to swiftly produce were the people who would live and work there. Soul City had a population of less than 150 in 1979, despite plans that projected a population of 2,000 by then. Huge companies like General Motors and Perdue, which at one point considered building operations centers in Soul City, began pulling out. McKissick was unable to land a major corporation to anchor the city’s economic development.
There were a number of reasons why Soul City’s recruitment efforts fell short. The U.S. economy tanked in the 1970s amid the oil and energy crises. Soul City also had to weather a fusillade of scrutiny from new hard-right conservatives like Jesse Helms, whom North Carolina elected to the U.S. Senate in 1972. Helms immediately made Soul City his whipping post, casting it as a symbol of wasteful government spending.
A series of articles from the Raleigh News & Observer surfaced accusations of financial malfeasance inside Soul City’s development team, which stymied investor confidence even more. McKissick’s crew was eventually cleared through an audit from the U.S. Government Accountability Office, but the damage was done. By the early 1980s, even HUD had begun winding its financial support down, citing Soul City’s lack of progress.
Jet, June 11, 1981
McKissick’s rejoinder to HUD was that the department was “taking a baby nine-months-old and asking why he is not a lawyer,” which failed to convince his benefactors. At this point, a new Republican Party had taken over, one that wanted nothing to do with any program that Nixon’s Watergate-scandal-plagued hands had touched. Not only that, but Ronald Reagan had fully embraced the “Southern Strategy” adopted by Republicans, which focused on disaffected white Southern voters—especially those angry about federal programs that benefited African Americans.
Not that most black Americans reaped much benefit from these programs, but the New Deal/Great Society eras were officially over, and Soul City would be drained away with the grantsmanship bathwater.
Unable to attract adequate private capital and having lost federal funding, McKissick had to sell Soul City off to private interests. As Illinois State University professor Roger Biles wrote of the city’s short lifespan in the Journal of Planning History:  
Facing a hostile political environment and hampered by a foreboding economic climate, Floyd McKissick’s bold attempt to sustain a free-standing new town based on African American activism seemed doomed from the start. The uneasy marriage between black capitalism and the federal bureaucracy sundered at Soul City, a part of the larger failure of the new towns movement to solve the urban crisis of the late twentieth century.
McKissick’s team was never able to lure major industry to Soul City, but two industries did end up landing there after McKissick left. One was hazardous waste: Warren County notoriously became the place where North Carolina decided to dump tons of untreated, PCB-tainted soil in a landfill near Soul City. Black residents protested this conversion of their homeland into dumping grounds, many of them laying in front of the trucks delivering the soil. Their resistance marked the birth of the civil-rights-based environmental justice movement, though these groups weren’t able to stop toxic soil waste from infiltrating the earth around Soul City.
The other industry that ended up there: prison. The “Soul Tech I” business incubator McKissick built was sold off and is now the Warren Correctional Institution, an 809-bed facility housing far more people than McKissick was ever able to recruit to Soul City.  
As Devin Fergus writes in “Black Power, Soft Power”:
Like the nation at large, Soul City would not employ young black workers as much as warehouse them, as did rural communities and cities across America in the 1980s and 1990s, when prison corporations worked with local and state officials seeking to revitalize economically depressed areas by building prisons that incarcerated unprecedented numbers of nonviolent drug offenders, mostly men between the ages of eighteen and forty. The same able-bodied, young African American men whom McKissick had optimistically envisioned returning south to be gainfully employed in Soul City did return, but they did so, in the words of the state prisons director, as “out-of-state inmates back home, back home so they can be with families; back home where our employees can work with them in our facilities.”
In that respect, it now seems like the $14 million Nixon granted McKissick for Soul City turned out less a security deposit on reparations, and more a down payment on mass incarceration.

进口：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options