如何使用Selenium + Python加载动态内容

Question

On a Quora user profile page, I want to show the entire text for each user's answer, by clicking on "more", but I couldn't get the 'more' elements list, instead I am getting an empty list, how to do?

Here is my code:

more_buttons = browser.find_elements_by_name('more')
for more_button in more_buttons:
        more_button.click()

Here is an example of a user profile: https://www.quora.com/profile/Ashley-Isabella-3

Answer 1

That's because you're using the wrong CSS Selector. Try with:

more_buttons = browser.find_elements_by_class_name('ui_qtext_more_link')

You can find the right properties by right-clicking your element in a normal browser and clicking "Inspect".

Answer 2

In effect you are after the following the links (when you click) so you could just gather the links and then use a loop over with .get. Use return [...document.querySelectorAll ] syntax

from selenium import webdriver

d = webdriver.Chrome()
d.get("https://www.quora.com/profile/Ashley-Isabella-3")
links = d.execute_script("return [...document.querySelectorAll('.ui_qtext_more_link')].map(link => link['href'])")
info = []
for link in links:
    d.get(link)
    pageInfo = d.execute_script("return [...document.querySelectorAll('.ui_qtext_para')].map(link => link.innerText)")
    info.append(pageInfo)
print(info)

Answer 3

On a Quora user profile page to click on the link with text as (more) and retrieve the user answers you can use the following solution:

• Code Block:

   from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC driver=webdriver.Firefox(executable_path=r'C:\\Utility\\BrowserDrivers\\geckodriver.exe') driver.get("https://www.quora.com/profile/Ashley-Isabella-3") more_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ui_qtext_more_link"))) for more_button in more_buttons: more_button.click() my_text_elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.ui_qtext_expanded>span.ui_qtext_rendered_qtext>p>b"))) for my_text_element in my_text_elements: print(my_text_element.text) 

• Console Output:

   11 mistakes you make when trying to lose belly fat You could be doing the completely wrong exercise. 1. You're not getting enough sleep 2. You're doing the wrong type of workout 3. You're consuming too much sugar 4. You're not eating enough protein 5. You're feeling stressed or anxious 6. You're expecting a quick fix 7. You're not tracking your progress 8. You're crash-dieting 9. You're doing too much 10. You haven't got the right exercise balance 11. You've lost motivation 9 Ways To Burn Fat Fast Implement these 9 fat-burning tips that use exercise and diet and watch the body fat melt like the butter you're no longer using. 1. Stay Off The Scale 2. Reduce Your Calories Gradually 3. Vary Your Caloric Intake 4. Train With Weights 5. Do High-Intensity Intervals (HIIT) 6. Eat More Fat 7. Cut Carbs 8. Increase Your Protein 9. Eat 6 Smaller Meals Per Day, Not 2-3 Feasts Bonus: Researchers Explain 6 Reasons Why A Man Falls In Love 1. HE'S CAPTIVATED BY THE WHOLE PACKAGE 2. HE FEELS HE CAN MAKE HER HAPPY 3. WHEN SHE'S OPEN TO LOVE 4. THE WAY HE FEELS AROUND HER 5. SHE SAYS “YES” TO LIFE 6. HE FEELS PURPOSE AND PASSION Must Watch: Secret For You in Under 20 Minutes... These are the 7 weirdest things men find attractive in women Who knew anyone would ever find stubbornness attractive in a woman? 1. Weakness 2. Applying makeup 3. Stubbornness 4. Stretch marks 5. Wearing glasses 6. No make up 7. Clumsiness Secret For You in Under 20 Minutes... 5 surprising ways to get great skin There are so many bad ideas out there about how to achieve great skin. Here are five ways to get on the right track, regardless of 'the rules.' 1. You don't need to wash your face with hot water to get it clean. 2. Oil is good for your skin, not bad. 3. You don't need to scrub to exfoliate. 4. What matters most for healthy skin is not what you put on it, but what you eat. 5. Chocolate doesn't cause acne, but bread and pasta might Make Your Skin Naturally Glow 1. Skip the long, steamy showers and opt for shorter, cooler sprays. 2. Check the dryness of your skin by scratching a small area on your arm or leg with your fingernail. 3. Treat your neck and chest like an extension of your face. 4. Run a humidifier every night in the winter to moisturize the air in your bedroom. 12 easy tips for glowing skin 1: Discover these glowing skin home remedies: 2: Sweat it out for healthy skin 3: Practice Yoga for Face Glow 4: Know who you are! 5: Follow a natural diet for glowing skin 6: Indulge yourself…weekly 7: Make Sudarshan Kriya your beauty mantra 8: Meditate everyday to get Glowing Skin 9: Silence is truly golden 10: Save your mind at any cost 11: 18 till I …why not? 12: Flex those facial muscles 15 Reasons Yoga Is Better Than The Gym 1. Yoga benefits the mind, body and spirit. 2. Yoga benefits your full body—externally and internally. 3. Yoga teaches acceptance. 4. Yoga is about focusing on yourself. 5. Yoga will make you lean. 6. Yoga is more efficient. 7. You can do yoga anywhere. 8. Yoga is kinder to the body. 9. Yoga eases your aches and pains. 10. Yoga helps you breathe easier. 11. Yoga is calming. 12. Yoga reduces stress. 13. Everyone can practice yoga. 14. Yoga improves concentration. 15. Yogis rock. 9 Ways to Tell If You're Exercising Enough The sign: You never feel sore or tired the day after a workout The sign: You're not losing any weight The sign: You barely break a sweat The sign: The last rep feels hard, but you could probably do another The sign: You can chat with your friend throughout your workout The sign: You're not getting any more flexible in your yoga classes The sign: You run several days a week but aren't getting faster The sign: You're making strides in boot camp but are always stiff and un-flexible The sign: You take the same class regularly but aren't getting better at the exercises How To Build Muscle Mass After 50 Exercise Plan Strength-Training Exercises Weight, Reps and Sets Aerobics Workouts Savannah Guthrie Says The Keto Diet Is Making Her Mind Sharp—And Her Body Sluggish She has zero desire to work out. How to Lose Weight in 10 Days at Home Flight of Stairs / Treadmill Lots of Water Fresh – Fruits and Vegetables – Low-Fat Foods Fish / Meat – Avoid Junk Food – Moderate Exercise - Drink lots of water - Day 1 Day 2 and 3 Day 4 and 5 Day 6 and 7 Day 8 and 9 Day 10 What are natural ways to make my face look younger? Smoothing Exfoliator Tightening Toner Plumping Moisturizing Facial Mask 10 Ways to Lose 25 Pounds Follow these steps and weight loss won't seem impossible. 1. Do some detective work. 2. Get moving. 3. Plan ahead. 4. Fill up on fruits and veggies. 5. Don't drink your calories. 6. Don't do anything drastic. 7. Set smart calorie targets. 8. Keep track. 9. Do not make weight the only measure of success. 10. Just practice; don't try to be perfect.