在 Python 中使用 URL 从 txt 中提取数据

Question

I have a URL which contains txt data From this URL I want to extract a particular section of data:

data here:

I have added the screenshot of the txt file. In the screenshot you can see "Table of Contents". From these table of contents I want to extract the textual data of a particular Item number for example I want to extract data from a Part 2 Item 5 which have data on page number 12. Can anyone help me in extracting this particular data using python

Answer 1

There are a few ways you could go about this. The first, and probably the most simplistic, would be the string.find() method. Of course, that assumes that you know what you are looking for, you just want to use a program to fetch it instead of manually.

However, after looking at the way the document is formatted, there's the possibility that you could pass it to something like BeautifulSoup , although I really wouldn't recommend feeding that type of document into BS4, you'd likely get a large number of errors.

The third option is possibly the easiest method (as far as I can tell) which would be to construct a regex to search the document for a string matching what you want.

In your given example of Part 2, Item 5, you could write a simple program that would look like the following:

import re
import requests

r = requests.get("https://www.sec.gov/Archives/edgar/data/3662/0000950170-98-000413.txt")

str_to_check = r.text

what_i_want = re.findall(r"(?i)(Part 2,? Item 5)", str_to_check)

print(what_i_want)

A site like regex101 could be very helpful in learning how to construct the regex you would need.

Answer 2

You can split your text into pages, and print the content of the page that you want:

import requests

r = requests.get("https://www.sec.gov/Archives/edgar/data/3662/0000950170-98-000413.txt")

r_text = r.text

text_pages = r_text.split("<PAGE>")

# get content of page 12
page = 12
text_page_12 = text_pages[page+1]
print(text_page_12)

Answer 3

Get Table data with Requests and BeautifulSoup then you can do other operations like save it to txt file or anything else.

# https://stackoverflow.com/questions/64618978/extracting-data-from-txt-using-url-in-python
import requests
from bs4 import BeautifulSoup


def get_data(url):
    r =  requests.get(url)
    if r.status_code == 200:
        return r.content


url = "https://www.sec.gov/Archives/edgar/data/3662/0000950170-98-000413.txt"
data = get_data(url)

soup =  BeautifulSoup(data, "lxml")

table = soup.find("table")
print(table)

Answer 4

This code outputs the desired table in the terminal and also puts that table into a csv file (table_output.csv). You can select the PART, ITEM and the TABLE (in case there is more than one table in the section, like for example, in PART I, Item 2.) Or you can select text with PART and ITEM.

Note: For the CSV file: you have to do a bit a work yourself (see the code) because how you group elements is not necessarily unique and I don't see how to do that completely mechanically for any table.

The code is not heavily commented but I hope that's enough. Don't hesitate if you have questions.

Best regards,

Stéphane

The code:

import requests
import re

def Encode_digit(digit, one, five, nine):
# from https://codereview.stackexchange.com/questions/141402/converting-from-roman-numerals-to-arabic-numbers

    return (
        nine                     if digit == 9 else
        five + one * (digit - 5) if digit >= 5 else
        one + five               if digit == 4 else
        one * digit              
    )

def Encode_roman_numeral(num): 
# from same source as above
    num = int(num)
    return (
        'M' * (num // 1000) +
        Encode_digit((num // 100) % 10, 'C', 'D', 'CM') +
        Encode_digit((num //  10) % 10, 'X', 'L', 'XC') +
        Encode_digit( num         % 10, 'I', 'V', 'IX') 
    )

def LookForPart(r_text, Part):
    Part=Encode_roman_numeral(Part)
    l_tags=[]
    s='PART '+Part
    for i in range(0,len(r_text)):#len(r_text)):
        if re.search('PART '+Part+'$',r_text[i]) and r_text[i].find('    ')>-1:
            l_tags.append(i)
    return l_tags

def LookForItem(r_text, Item):
    l_tags=[]
    s='ITEM '+str(Item)+'.'
    for i in range(0,len(r_text)):#len(r_text)):
        if re.search(s,r_text[i]):# and r_text[i].find('    ')>-1:
            l_tags.append(i)
    return l_tags

def SelectSection(r_text,Part,Item):
    start=LookForPart(r_text,Part)[0]
    try:
        end=LookForPart(r_text,Part+1)[0]
    except:
        end=len(r_text)
    r_text=r_text[start:end]
    start=LookForItem(r_text,Item)[0]
    try:
        end=LookForItem(r_text,Item+1)[0]
    except:
        end=len(r_text)
    r_text=r_text[start:end]
    return r_text

def RawTable(Section):
    tables=[]
    nb_tables=0
    for i in range(0,len(Section)):
        if Section[i].find('<TABLE>')>-1:
            Start=i
        if Section[i].find('</TABLE>')>-1:
            End=i
            nb_tables+=1
            tables.append(Section[Start:End+1])

    return tables

def get_table(r_text, Part, Item, Table_Nb):
    Section=SelectSection(r_text,Part,Item)
    table=[]
    ThisTable=RawTable(Section)[Table_Nb]
    for i in range(len(ThisTable)):
        line=ThisTable[i].replace('<S>','   ').replace('<C>','   ')              .replace('<TABLE>','')              .replace('</TABLE>','')              .replace('<CAPTION>','')
        line=re.sub('-','  ',line)
        delta=len(line)-len(re.sub('(\.+){3,}','',line))
        line=re.sub('(\.+){3,}',' '*delta,line)
        if len(line)>0:
            #print(line,file=f)
            if len(line.strip(' '))>0:
                table.append(line)    
    return table

##################################################################################
#
# Main:
# 
##################################################################################

# Retrieve data:

url='https://www.sec.gov/Archives/edgar/data/3662/0000950170-98-000413.txt'
r = requests.get(url)
r_text=r.text
r_text=r_text.splitlines()


# Get table in Part II, Item 5, first table (first table in case there are more than one table in the PART II Item 5. section) 

Result=get_table(r_text,2,5,0) # 2,5,0 = Part II, Item 5, first table
print("<Extracted table>")
for i in range(0,len(Result)):
    print(Result[i])
print('<End of table>')


# The rest really depends on the intrinsic variablity of the data. If we assume that this format is strictly
# valid in every case, then we can extract the data as follows:

# Let's output the result to a comma separatred csv file:

import csv

Result=Result[2:]; # We skip the first two rows


fields=['DATE','$ HIGH','FRAC_HIGH','$ LOW','FRAC_LOW'] 
dump=[]
dump.append(fields)
for i in range(0,len(Result)):
    dump.append([Result[i][8:22],Result[i][30:38].strip(' '),Result[i][38:44].strip(' '),                 Result[i][45:51].strip(' '),Result[i][51:].strip(' ')])

f=open('table_output.csv', 'w')
write = csv.writer(f) 
for row in dump:   
    write.writerow(row) 
f.close()

# Another example: Select PART II, Item 5.
#
# Example to select PART 2 Item 5.
print()
print('<Selection: PART 2 Item 5.>')
sel=SelectSection(r_text, 2,5)
for i in range(0, len(sel)):
    print(sel[i])

The output from terminal:

<Extracted table>
                                         MARKET PRICE
                                       HIGH          LOW
        1997:
        Fourth Quarter             $50 7/16     $37 
        Third Quarter              $45 3/4      $35 3/8
        Second Quarter             $40 3/4      $29 3/4
        First Quarter              $34 1/2      $24 5/8
        1996:
        Fourth Quarter             $29 1/2      $22 3/4
        Third Quarter              $24 3/4      $12 1/4
        Second Quarter             $17 1/8      $13 1/2
        First Quarter              $19 3/4      $15 1/8
<End of table>

<Selection: PART 2 Item 5.>
ITEM 5. MARKET FOR REGISTRANT'S COMMON EQUITY AND RELATED STOCKHOLDER MATTERS


     The Common Stock is listed and traded on the New York Stock Exchange under
the symbol "SOC". The Company has paid quarterly cash dividends of $.01 per
share since December 15, 1992. The Company presently intends to continue to pay
cash dividends at a quarterly rate of $.01 per share; however, future payments
of cash dividends will be at the discretion of the Company's Board of Directors
and dependent upon the Company's results of operations, financial condition and
other relevant factors.


     The following table sets forth the high and low sale prices for the Common
Stock for the calendar quarters indicated as reported by the New York Stock
Exchange Composite Tape:



<TABLE>
<CAPTION>
                                         MARKET PRICE
                                   ------------------------
                                       HIGH          LOW
                                   -----------   ----------
<S>                                <C>           <C>
        1997:
        Fourth Quarter .........   $50 7/16     $37 
        Third Quarter ..........   $45 3/4      $35 3/8
        Second Quarter .........   $40 3/4      $29 3/4
        First Quarter ..........   $34 1/2      $24 5/8
        1996:
        Fourth Quarter .........   $29 1/2      $22 3/4
        Third Quarter ..........   $24 3/4      $12 1/4
        Second Quarter .........   $17 1/8      $13 1/2
        First Quarter ..........   $19 3/4      $15 1/8
</TABLE>

     On February 27, 1998 there were approximately 1,311 record holders of the
Company's Common Stock.


                                       12
<PAGE>