使用SED進行網頁抓取

Question

首先，我要說的是，我知道當有這么多更好的工具時，sed中的Web抓取是一個愚蠢的主意。 我試圖告訴我的老師，但是他堅持要使用sed和awk。

我的目標是刮擦Twitter頁面並返回所有推文。 使用此代碼：

sed -n '/<p class=\"TweetTextSize  js-tweet-text tweet-text\" lang=\"en\" data-aria-label-part=\"0\">/,/<\/p>/p' $targ | sed 's/<[^>]\+>/ /g;s/^[ \t]*//;/^$/d'

到目前為止， 在此URL上，我無法准確獲得我想要的東西。

這是圍繞一條推文的代碼：

<p class="TweetTextSize js-tweet-text tweet-text" lang="en" data-aria-label-part="0">Where is our universe headed? Answer to it&#39;s death, not to worry <a href="/PaulMattSutter" class="twitter-atreply pretty-link js-nav" dir="ltr" data-mentioned-user-id="556263627" ><s>@</s><b>PaulMattSutter</b></a> explains. <a href="https://t.co/ig8KtQzwOI" rel="nofollow" dir="ltr" data-expanded-url="http://oak.ctx.ly/r/42bnj" class="twitter-timeline-link" target="_blank" title="http://oak.ctx.ly/r/42bnj" ><span class="tco-ellipsis"></span><span class="invisible">http://</span><span class="js-display-url">oak.ctx.ly/r/42bnj</span><span class="invisible"></span><span class="tco-ellipsis"><span class="invisible">&nbsp;</span></span></a> <a href="https://t.co/kOy4jLCogk" class="twitter-timeline-link u-hidden" data-pre-embedded="true" dir="ltr" >pic.twitter.com/kOy4jLCogk</a></p>

現在，我只想從<p class="TweetTextSize js-tweet-text tweet-text" lang="en" data-aria-label-part="0">及其對應的</p> ，然后我要對其進行過濾，以除去中間的所有標簽（打開和關閉），使其看起來像這樣：

Where is our universe headed? Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj pic.twitter.com/kOy4jLCogk

Answer 1

嘗試這個：

 sed -n '/<p class="TweetTextSize  js-tweet-text tweet-text" lang="en" data-aria-label-part="0">.*<\/p>/{s/<[^>]*>//g;p}' file

輸出：

You’ve never seen Pluto like this! @NASANewHorizons returns sharpest views yet! Take a look: http://go.nasa.gov/1NxLViz  pic.twitter.com/iK4OJKeLUE
        The world is filled with those who do nothing yet criticize. I love this @rickygervais motto: Go create! pic.twitter.com/m7zB8ucclT
        Can you see me waving? How to spot #Mars in the night sky: https://youtu.be/hv8hVvJlcJQ  https://amp.twimg.com/v/9b945202-5a89-4efa-80ab-514a0c17a965 …
        We knew the first high-res images of Pluto would be good, but we didn't expect THIS good: http://ow.ly/Vvrvm  pic.twitter.com/r8ZX4Z5tVt
        Toddler wears the most precious 'Monsters, Inc.' costume this world has ever seen http://on.mash.to/1HLrcMk  pic.twitter.com/2PyuDW9yms
        Are you on your way to your @TMobile store, @ATT customers?! GO!! #UncarrierUnwrapped http://t-mo.co/1SzhOfb  pic.twitter.com/0vNJbD2J4k
        1 year ago today, @USNavy @ussanchorage recovered Orion in Pacific ocean after successful flight. pic.twitter.com/DKW00LzlC3
        How Fallout 4 mastermind Todd Howard builds his epic dream worlds http://wrd.cm/1IILf8N  pic.twitter.com/WUV98e6U62
        23-year-old’s design collaboration tool Figma launches with $14M to fight Adobe http://tcrn.ch/1YPW8Pw  pic.twitter.com/wVrZbfgBOc
        Here’s a list of questions investors will ask themselves after you pitch a company to them. http://kpcb.cc/b466b5b  pic.twitter.com/zOJDOrq0na
        Blow your mind with this--ENJOY! (And check the SWEET video too)! http://www.nasa.gov/feature/new-horizons-returns-first-best-images-of-pluto … #PlutoFlyby pic.t
        Human gene editing is OK, but no designer babies—for now, summit concludes: http://ow.ly/VscSg  #GeneEditSummit pic.twitter.com/UW6DKtQX4h
        NASA releases the highest resolution images ever taken of Pluto http://www.theverge.com/2015/12/4/9851162/nasa-new-horizons-pluto-photo-new-high-resolution?utm_campai
        A drone that finds you a parking space and then hovers in it to stop anyone else taking it first.
        The ubiquity of guns poses a much greater threat to Americans than terrorism http://bit.ly/1lbvPoQ  By @Horganism pic.twitter.com/pdxvVqGS59
        Scientist disinvited from speaking at conference because of her pregnancy http://ow.ly/Vvyam  pic.twitter.com/FUAjaycIom
        Samsung agrees to pay Apple $548 million in settlement http://engt.co/1jEN28t  pic.twitter.com/3DmmccJUgW
        World's first in-human gene-editing treatment will tackle hemophilia http://www.engadget.com/2015/12/03/worlds-first-in-human-CRISPR-hemophilia/ … pic.twitte
        Where is our universe headed?  Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj  pic.twitter.com/kOy4jLCogk
        New Horizons just sent back the sharpest images possible of Pluto's surface. Drink it in http://www.wired.com/2015/12/new-horizons-just-sent-its-highest-res-image
        Get people to do what you want with these 11 clever psychological tricks: http://lifehac.kr/GZaSWhN  pic.twitter.com/5rhWlVL9t0