Python3 XML get text between tags

Question

I have the following code in Python 3. I am using the import xml.etree.ElementTree as ET for XML parsing. the webScraper grab the text from an webside but on that website there is text between the <link></link> tag, but the program returns None. I can se that the program finds all tags but where the tag result should be printed it only says None.

    result = webScrapper.scrappPart("http://www.dn.se/rss/senaste-nytt/", "body")
    root = ET.fromstring(result)
    for items in root.findall('.//item'):
        link = items.find('link')
        print(link.text)

Does anyone know how to fix this?

Answer 1

Since your URL is actually an RSS feed, you'd be much better off using an RSS feed parser on it, instead of trying to roll your own. Fortunately, this is why feedparser exists. Check this out:

import feedparser as fp

feed = fp.parse("http://www.dn.se/rss/senaste-nytt/")
for entry in feed["entries"]:
    print(entry["link"])

This returns

http://www.dn.se/sport/fotboll/cavani-het-i-svalt-psg/
http://www.dn.se/sport/fotbolls-em/kompany-missar-em/
http://www.dn.se/nyheter/sverige/livvaktens-slakting-fick-praktik-hos-sahlin-trots-myndighetens-avslag/
http://www.dn.se/sport/st-louis-andraperiod-avgjorde/
http://www.dn.se/nyheter/varlden/syrien-spanska-journalister-fria/
http://www.dn.se/sport/dansk-dynamit-ska-stoppa-tre-kronor/
http://www.dn.se/nyheter/sverige/mordmisstankt-slappt-ur-haktet-1/
http://www.dn.se/nyheter/varlden/ekonomiprofessor-loste-ekvation-togs-for-terrorist/
http://www.dn.se/sport/fotboll/leicester-firade-med-storseger/
http://www.dn.se/ekonomi/protester-mot-ny-granskontroll-urartade/
http://www.dn.se/sport/ishockey-vm/jimmie-ericsson-jag-ar-beredd-gora-allt-for-att-vinna/
http://www.dn.se/sport/ishockey-vm/schweiz-straffat-av-kazakstan/
http://www.dn.se/nyheter/varlden/natosoldater-dodade-i-afghanistan-2/
http://www.dn.se/sport/forsta-matchen-till-eslov/
http://www.dn.se/nyheter/sverige/drunknad-man-hittad-av-dykare/
http://www.dn.se/ekonomi/tagstopp-efter-olycka/
http://www.dn.se/sport/kristianstad-till-sm-final/
http://www.dn.se/sthlm/en-person-attackerad-med-kniv-i-centrala-stockholm/
http://www.dn.se/nyheter/sverige/inga-spar-efter-forsvunnen-22-arig-student/
http://www.dn.se/sport/fotboll/forlust-for-rydstrom-i-tranardebuten/
http://www.dn.se/nyheter/sverige/manga-grasbrander-runt-om-i-landet/
http://www.dn.se/nyheter/sverige/tre-gripna-efter-skottlossning-i-malmo/
http://www.dn.se/sport/fotboll/elfsborg-ar-med-i-toppen-igen/
http://www.dn.se/sport/em-silver-till-rissveds/

which I assume is what you're looking for.

Answer 2

You can use ElementTree just fine, you just need to pass the source and use the xpath:

from xml.etree import ElementTree as et
import requests

tree = et.fromstring(requests.get("http://www.dn.se/rss/senaste-nytt/").content)

print([x.text for x in tree.findall(".//item//link")])

Output:

['http://www.dn.se/nyheter/varlden/andlig-ledare-ihjalhackad-i-bangladesh/', 'http://www.dn.se/nyheter/sverige/tillstandet-battre-for-pakord-ettaring/', 'http://www.dn.se/ekonomi/maria-crofts-dags-att-gora-nagot-at-orattvisa-pensioner/', 'http://www.dn.se/nyheter/varlden/turkisk-militar-dodade-55-is-krigare/', 'http://www.dn.se/nyheter/varlden/massiv-fiskdod-i-sjo/', 'http://www.dn.se/nyheter/varlden/kanadabranden-i-bilder/', 'http://www.dn.se/nyheter/sverige/manga-saknas-efter-jordskred-i-kina/', 'http://www.dn.se/nyheter/sverige/fortsatt-sokande-efter-student/', 'http://www.dn.se/nyheter/sverige/en-dod-i-villabrand-8/', 'http://www.dn.se/nyheter/politik/v-vill-ta-bort-terrorstampel-pa-pkk/', 'http://www.dn.se/ekonomi/raknehjalp-pa-natet-ger-ratt-underhall/', 'http://www.dn.se/nyheter/varlden/kanadabranden-fullstandigt-okontrollerad/', 'http://www.dn.se/nyheter/varlden/attentat-mot-journalister-besvarande-for-erdogan/', 'http://www.dn.se/nyheter/varlden/superlobbyist-ska-gora-trump-serios/', 'http://www.dn.se/nyheter/vetenskap/karin-bojs-en-typisk-foralder-ar-28-ar-gammal/', 'http://www.dn.se/sport/nervos-vantan-pa-em-biljetter/', 'http://www.dn.se/ekonomi/ovantat-stort-exportfall-i-kina/', 'http://www.dn.se/ekonomi/lott-gav-35-miljarder-i-vinst-i-usa/', 'http://www.dn.se/nyheter/vetenskap/fabels-kansliga-nos-ska-ge-svar-om-massmordet/', 'http://www.dn.se/sport/johan-esk-nu-borde-idrotten-lara-ledarskap-av-naringslivet-1/', 'http://www.dn.se/sport/melker-karlsson-malskytt-for-san-jose/', 'http://www.dn.se/sport/backstrom-visade-vagen-till-washingtons-viktiga-vinst/', 'http://www.dn.se/nyheter/varlden/15-miljoner-signaturer-bekraftade/', 'http://www.dn.se/nyheter/varlden/medan-du-sov-varlden-i-korthet-8-maj-1/', 'http://www.dn.se/nyheter/varlden/karnvapen-bara-om-landet-hotas/', 'http://www.dn.se/nyheter/varlden/protester-mot-att-avskaffa-senat/', 'http://www.dn.se/nyheter/sverige/industri-brann-i-uppsala/', 'http://www.dn.se/nyheter/varlden/atta-poliser-dodade-i-attack/', 'http://www.dn.se/nyheter/varlden/tva-fast-vid-myr-utanfor-kiruna/', 'http://www.dn.se/sport/hockeyhasten-nyquist-vann-kentucky-derby/', 'http://www.dn.se/nyheter/varlden/15-miljoner-signaturer-bekraftade-i-venezuela/']

Or using lxml which can also get the source for you:

from lxml import etree

result = etree.parse("http://www.dn.se/rss/senaste-nytt/")
print(result.xpath("//item//link//text()"))

Which gives you the exact same output.