Searching for language specific words in python
Question
I have a program that I need to loop through files and find a danish word. I then need it to extract the text it finds (between html tags) and input that to a csv file..
My code so far:
from bs4 import BeautifulSoup
import re
import csv
import glob
#def get_danish(text):
# return re.compile(r'\b({0})\b'.format(text), flags=re.IGNORECASE).search
with open('dk_snip.csv', 'w', newline='') as f_out:
csv_out = csv.writer(f_out)
# csv_out.writerow(["Nessus", "ID", "Descrip"])
for filename in glob.glob('/home/rj/Documents/snip/snips/*'):
print("Processing:", filename)
with open(filename) as f_in:
soup = BeautifulSoup(f_in, 'html5lib')
var1 = soup.find('li', text = re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
var2 = soup.find('li', text = re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})
vendor = var1.find_next('li').get_text(strip=True)
vend_id = var2.find_next('li').get_text(strip=True)
# rows = [[vendor, vend_id, dk_desc.get_text(strip=True)] for dk_desc in soup.find_all("textarea")[:3]]
for textarea in soup.find_all("textarea"):
print(len(textarea))
# if !not textarea[7]:
# desc = textarea[7].get_text(strip=True)
# elif not textarea[7]:
# desc = textarea[6].get_text(strip=True)
# elif not textarea[6]:
# desc = "unknown"
# csv_out.writerow([vendor, vend_id])
# for elem in soup.select("textarea"):
# if "disse" in elem:
# second_text = elem.text.get_text(Strip=True)
# print(second_text)
# textarea = soup.find_all(re.compile("textarea"))
# second_text = textarea[6].text.rstrip(' ')
# wr.writerow([fin, fin2, second_text])
The files have a varying number of "textarea" tags, sometimes 5, sometimes 7, the danish text is NOT present i all of them, but if it is, its allways in the 6th or 7th.. Now I dont know if its just easier to search for a given word and extract the text from the tag where its found, or if finding it by counting tags is best..
My code:
<!DOCTYPE html> <html lang="en"> <head> <li class="property_name"> <label for="id_194-description"> Description: </label> </li> <li class="property_value"> <textarea class="mceNoEditor" cols="40" id="id_194-description" name="194-description" rows="10" style="width:100%">According to its version, the installation of Oracle Database on the remote host is no longer supported. Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities.</textarea> </li> <li class="property_name"> <label for="id_194-consequence"> Consequence: </label> </li> <li class="property_value"> <textarea class="mceNoEditor" cols="40" id="id_194-consequence" name="194-consequence" rows="10" style="width:100%">The remote host is running an unsupported version of a database server.</textarea> </li> <li class="property_name"> <label for="id_194-solution"> Solution: </label> </li> <li class="property_value"> <textarea class="mceNoEditor" cols="40" id="id_194-solution" name="194-solution" rows="10" style="width:100%">Upgrade to a version of Oracle Database that is currently supported.</textarea> </li> <li class="property_name"> <label for="id_194-cve_id"> Cve id: </label> </li> <li class="property_value"> <textarea class="mceNoEditor" cols="40" id="id_194-cve_id" maxlength="8192" name="194-cve_id" rows="10" style="width:100%; height:80px"></textarea> </li> <input id="id_194-override" name="194-override" type="hidden" value="11953"/> <input id="id_194-priority" name="194-priority" type="hidden"/> <li class="property_name"> Vulnerability priority </li> <li class="property_value"> <select name="prio_194"> <option selected="selected" value="0"> 0 </option> </select> : Oracle Database Unsupported (Nessus) <br/> </li> <li class="property_name"> Save </li> <li class="property_value"> <input type="submit" value="Save vulnerability changes"/> </li> </ul> </form> <br style="clear:both"/> </div> <div class="box"> <h4> Related vulnerabilities </h4> <hr/> <h5> Oracle Database Unsupported (Nessus) </h5> <ul> <li class="property_name"> Description </li> <li class="property_value"> According to its version, the installation of Oracle Database on the remote host is no longer supported. <br/> <br/> Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities. </li> <li class="property_name"> Consequence </li> <li class="property_value"> The remote host is running an unsupported version of a database server. </li> <li class="property_name"> Solution </li> <li class="property_value"> Upgrade to a version of Oracle Database that is currently supported. </li> </ul> <br style="clear:both"/> </div> <div class="box"> <h4> Create new snippet </h4> <form action="/report/vulnerabilityEditor/? action=edit& id=194& sid=& model=snippet" method="POST"> <ul> <li class="property_name"> <label for="id_language"> Language: </label> </li> <li class="property_value"> <select id="id_language" name="language" style="width:100%"> <option selected="" value="1"> Danish (DK) </option> <option value="2"> English (EN) </option> <option value="3"> Icelandic (IS) </option> </select> </li> <input id="id_vulnerability" name="vulnerability" type="hidden" value="194"/> <li class="property_name"> <label for="id_title"> Title: </label> </li> <li class="property_value"> <input id="id_title" maxlength="100" name="title" style="width:100%" type="text"/> </li> <li class="property_name"> <label for="id_recommendation"> Recommendation: </label> </li> <li class="property_value"> <input id="id_recommendation" maxlength="255" name="recommendation" style="width:100%" type="text"/> </li> <li class="property_name"> <label for="id_snippet"> Snippet: </label> </li> <li class="property_value"> <textarea cols="40" id="id_snippet" name="snippet" rows="10" style="width:100%"></textarea> </li> <li class="property_name"> Scan type </li> <li class="property_value"> <select multiple="multiple" name="scan_type" size="6" style="width:100%"> <option selected="selected" value="5"> COMPANY PCI </option> <option selected="selected" value="7"> Other </option> <option selected="selected" value="8"> Firewall Audit </option> <option selected="selected" value="6"> Penetration Test </option> <option selected="selected" value="9"> WIFI Test </option> <option selected="selected" value="10"> APP Test </option> <option selected="selected" value="1"> External Security Analysis </option> <option selected="selected" value="2"> Internal Security Analysis </option> <option selected="selected" value="3"> Web Application Test </option> <option selected="selected" value="4"> Host Discovery Analysis </option> </select> -- Use ctrl to mark multiple types </li> <li class="property_name"> Save </li> <li class="property_value"> <input type="submit" value="Save new snippet"/> </li> </ul> <br style="clear:both;"/> </form> </div> <div class="box"> <h4> Edit snippets </h4> <input id="property_vulnerability_id" type="hidden" value="194"/> <input id="property_url_filter_snippets" type="hidden" value="/report/filterSnippets/"/> <ul> <li class="property_name"> Language </li> <li class="property_value"> <select id="language" name="language"> <option value="0"> All </option> <option value="1"> Danish </option> <option value="2"> English </option> <option value="3"> Icelandic </option> </select> </li> <li class="property_name"> Scan Type </li> <li class="property_value"> <select id="scantype" name="scantype"> <option value="0"> All </option> <option value="5"> COMPANY PCI </option> <option value="7"> Other </option> <option value="8"> Firewall Audit </option> <option value="6"> Penetration Test </option> <option value="9"> WIFI Test </option> <option value="10"> APP Test </option> <option value="1"> External Security Analysis </option> <option value="2"> Internal Security Analysis </option> <option value="3"> Web Application Test </option> <option value="4"> Host Discovery Analysis </option> </select> </li> </ul> <br style="clear:both;"/> <div class="snippet"> <form action="/report/vulnerabilityEditor/?action=edit&id=194&sid=1290&model=snippet" method="POST"> <input id="id_1290-vulnerability" name="1290-vulnerability" type="hidden" value="194"/> <hr/> <ul> <li class="property_name"> <label for="id_1290-language"> Language: </label> </li> <li class="property_value"> <select id="id_1290-language" name="1290-language" style="width:100%"> <option value="1"> Danish (DK) </option> <option selected="" value="2"> English (EN) </option> <option value="3"> Icelandic (IS) </option> </select> </li> <li class="property_name"> <label for="id_1290-title"> Title: </label> </li> <li class="property_value"> <input id="id_1290-title" maxlength="100" name="1290-title" style="width:100%" type="text" value="Oracle Database Unsupported"/> </li> <li class="property_name"> <label for="id_1290-recommendation"> Recommendation: </label> </li> <li class="property_value"> <input id="id_1290-recommendation" maxlength="255" name="1290-recommendation" style="width:100%" type="text" value="Upgrade to a version of Oracle Database that is currently supported."/> </li> <li class="property_name"> <label for="id_1290-snippet"> Snippet: </label> </li> <li class="property_value"> <a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank"> How to: Add figure number in snippet. </a> </li> <li class="property_value"> <textarea cols="40" id="id_1290-snippet" name="1290-snippet" rows="10" style="width:100%"><p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">It has been detected, that the installed version of Oracle Application Server is&nbsp;<strong>XXXX.&nbsp;</strong>This version is known to be vulnerable to a number of unspecified vulnerabilities, categorized as 'urgent'.</span></p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">As this version is no longer supported for this platform, updates or patches may no longer be released, which have the consequence that vulnerabilities can not be patched, leaving the system vulnerable.</span></p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">In version 10.1.2.0.2 there are, according to http://www.cvedetails.com more than 54 vulnerabilities which affects the installed version.</span></p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: center; line-height: normal;" align="center"><strong><em><span lang="EN-US" style="font-size: 8pt;">Figure 1: </span></em></strong><em><span lang="EN-US" style="font-size: 8pt;">Oracle Application Server version.</span></em></p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">More information on these vulnerabilities can be found at:&nbsp;</span><span style="font-size: 10pt;"><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"><span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;">http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html</span></a><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"><span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;">&nbsp;</span></a></span><span lang="EN-US" style="font-size: 10pt;">.</span></p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;">&nbsp;</p> <p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">It is recommended that the installed version is updated as soon as possible to the latest version.</span></p></textarea> </li> <li class="property_name"> Scan type </li> <li class="property_value"> <select multiple="multiple" name="scan_type" size="6" style="width:100%"> <option selected="selected" value="5"> COMPANY PCI </option> <option selected="selected" value="7"> Other </option> <option selected="selected" value="8"> Firewall Audit </option> <option selected="selected" value="6"> Penetration Test </option> <option selected="selected" value="9"> WIFI Test </option> <option selected="selected" value="10"> APP Test </option> <option selected="selected" value="1"> External Security Analysis </option> <option selected="selected" value="2"> Internal Security Analysis </option> <option selected="selected" value="3"> Web Application Test </option> <option selected="selected" value="4"> Host Discovery Analysis </option> </select> -- Use ctrl to mark multiple types </li> <li class="property_name"> Update </li> <li class="property_value"> <input type="submit" value="Update snippet"/> </li> </ul> </form> <br style="clear:both;"/> </div> <div class="snippet"> <form action="/report/vulnerabilityEditor/?action=edit&id=194&sid=172&model=snippet" method="POST"> <input id="id_172-vulnerability" name="172-vulnerability" type="hidden" value="194"/> <hr/> <ul> <li class="property_name"> <label for="id_172-language"> Language: </label> </li> <li class="property_value"> <select id="id_172-language" name="172-language" style="width:100%"> <option selected="" value="1"> Danish (DK) </option> <option value="2"> English (EN) </option> <option value="3"> Icelandic (IS) </option> </select> </li> <li class="property_name"> <label for="id_172-title"> Title: </label> </li> <li class="property_value"> <input id="id_172-title" maxlength="100" name="172-title" style="width:100%" type="text" value="Forældet Oracle Application Server 10g"/> </li> <li class="property_name"> <label for="id_172-recommendation"> Recommendation: </label> </li> <li class="property_value"> <input id="id_172-recommendation" maxlength="255" name="172-recommendation" style="width:100%" type="text"/> </li> <li class="property_name"> <label for="id_172-snippet"> Snippet: </label> </li> <li class="property_value"> <a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank"> How to: Add figure number in snippet. </a> </li> <li class="property_value"> <textarea cols="40" id="id_172-snippet" name="172-snippet" rows="10" style="width:100%"><p style="font-size: 13px;">Det konstateret, at den installerede version af Oracle Application Server er&nbsp;<strong>XXXX.&nbsp;</strong>Denne version indeholder flere kendte samt uspecificeret s&aring;rbarheder, der kategoriseres som v&aelig;rende 'yderst kritiske' og 'kritiske'.</p> <p style="font-size: 13px;">Da der ikke l&aelig;ngere komme opdateringer til denne platform, vil disse s&aring;rbarheder ikke blive udbedret, hvorfor systemet er meget udsat.</p> <p style="font-size: 13px;">I version 10.1.2.0.2 findes der if&oslash;lge http://www.cvedetails.com ikke mindre end 54 s&aring;rbarheder, der ber&oslash;rer denne version. Mere information om disse findes p&aring; adressen&nbsp;<a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html">http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html</a><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html">&nbsp;</a>.</p> <p style="font-size: 13px;">Det anbefales leverand&oslash;ren af software l&oslash;sningen kontakts, s&aring; der hurtigst muligt kan opgraderes til en nyere, supporteret version.&nbsp;</p></textarea> </li>
1 answers
solution1
1 ACCPTED 2019-03-06 19:53:11
Question : Searching for language specific words
I dont know if its just easier to search for a given word and extract the text from the tag where its found, or if finding it by counting tags is best.
Neither of them,
- "search for a given word" will fail if not present and needs preview.
- "by counting tags" will fail, if the place in the list or the page layout changes.
The main order of the given html are <form>...</form> Elements.
Within any <form there is a <select Element with <option selected='' Elements. The <form with <option selected='' and <option ...>Danish (DK) are the one you are searching.
Find all
<form>...</form>Elementsforms = soup.find_all('form')Loop the form Elements
for form in forms:Verify, if the
<option>tag has aselected=attribut and the.textincludes'Danish (DK)':danish = [True for option in form.find_all("option", selected=True) if 'Danish (DK)' in option.text]Condition: If
danishis not empty, then option with text'Danish (DK)'are selected.if danish: print('MATCH:{}'.format(danish)) print('{}'.format(form.textarea.text))
Output :
MATCH:[True] <p style="font-size: 13px;">Det konstateret, at den installerede version... (omitted for brevity) <p style="font-size: 13px;">Da der ikke længere komme opdateringer... (omitted for brevity) <p style="font-size: 13px;">I version 10.1.2.0.2 findes der ifølg... (omitted for brevity) <p style="font-size: 13px;">Det anbefales leverandøren af softwar... (omitted for brevity)
Tested with Python: 3.4.2
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
searching for specific words in a text python
Searching for specific words in text file Python
Searching for key words in python
searching words in file in python
python: searching words in different files
Searching an array for key words -python
Python - searching for letters in a list of words
Trouble searching a string for words python
Python searching for two words regex
add language in python searching google
Related Tags