Get HTML inside div tag Python Scrapy

Question

I have this spider in scrapy

import scrapy


class ProductDetailSpider(scrapy.Spider):
    name = 'productdetail'
    allowed_domains = ['bisabeli.id']
    start_urls = [
        "https://bisabeli.id/index.php?route=product/product&product_id=7874"
    ]

    def parse(self, response):
        description = response.css('div.producttab div#tab-description div#collapse-description').get().strip()

        yield {
            'description': description,
        }

And when I run this spider in console with

scrapy crawl productdetail

The result is:

{
    "description": "<div id=\"co
llapse-description\" class=\"desc-collapse showup\">\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
 Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 &amp; 3x2.42 GHz Kryo 485 &amp; 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
 B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t</div>"}

What I want to get is all tags inside div tag ( '<div id="collapse-description" class="desc-collapse showup">'....'</div>' ),

How I write the code for it?

Update

The result I want is like this:

{
    "description": "\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
 Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 &amp; 3x2.42 GHz Kryo 485 &amp; 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
 B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t"}

Update v2

Trying some code:

description = response.xpath("//div[@class='producttab']//div[@id='tab-description']//div[@id='collapse-description']").get()

Result:

{'description': '<div id="co
llapse-description" class="desc-collapse showup">\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
 Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 &amp; 3x2.42 GHz Kryo 485 &amp; 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
 B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t</div>'}

description = response.xpath("//div[@class='producttab']//div[@id='tab-description']//div[@id='collapse-description']/text()").get()

result:

{'description': '\n\t\t\t\t\
t\t\t\t\t\t'}

description = response.xpath("//div[@class='producttab']//div[@id='tab-description']//div[@id='collapse-description']//text()").get()

result:

{'description': '\n\t\t\t\t\
t\t\t\t\t\t'}

Still... Not the result I desired...

Answer 1

OLD VERSION:

At this moment I found only .re(".+") to get all (more or less) as list

[opening_tag, item, item, ..., closing_tag] 

If I skip first and last element and use "".join() then I can get inner HTML.

# without `get()`
description = response.xpath("//div[@class='producttab']//div[@id='tab-description']//div[@id='collapse-description']")

description = description.re('.+')
description = "".join(description[1:-1]).strip()

---

EDIT:

Old version works only because there are \n in HTML - and it splits HTML into list on \n

["<opening tag>text", "children", "<closing_tag>"]

When there is no \n then old version create one string

[ "<opening tag> text children <closing_tag>" ]

similar to get() .

Besides every item has structure

 <tag> text children_tags </tag> tail 

and when \n is between text and children_tags then old version skips text` from final result.

Code which works for me for different HTML is

import lxml.html

html = '<body>hello<span>good</span>world</body>'

tree = lxml.html.fromstring(html)

text = tree.text or ''
children = [lxml.html.tostring(x).decode() for x in item.getchildren()]

inner_html = text + "".join(children).strip()

print(inner_html)

Result:

hello<span>good</span>world

It seems scrapy already uses lxml in Selectors so lxml should be already installed.

---

Minimal working code:

import scrapy
import lxml.html

def get_inner_html(html):

    tree = lxml.html.fromstring(html)

    text = tree.text or ''  # to skip `None`
    children = [lxml.html.tostring(x).decode() for x in tree.getchildren()]

    inner_html = text + "".join(children).strip()

    return inner_html 

class ProductDetailSpider(scrapy.Spider):

    name = 'productdetail'
    allowed_domains = ['bisabeli.id']
    
    start_urls = [
        "https://bisabeli.id/index.php?route=product/product&product_id=7874"
    ]

        
    def parse(self, response):
        print('--- example 1 ---')
        
        html = response.xpath("//div[@class='producttab']//div[@id='tab-description']//div[@id='collapse-description']").get()
        results = get_inner_html(html)
        
        print(results.strip())

        print('--- example 2 ---')
        
        html = '<body>hello<span>good</span>world</body>'
        results = get_inner_html(html)
        
        print(results.strip())