Web在python中抓取多个站点

Question

I signed up to this website just to ask this question as I have been searching for hours over multiple days and haven't found anything. I am trying to, within 10 seconds, scrape the 2-3 characters from 5 websites, combine them, and paste them into a box. I have a rough idea of what I would need, but no idea how to go about this. I believe I want to assign variables the scraped contents from a certain website, and then get it to print the combination of these variables for me to copy and paste. I'm not an expert by any means in Python, so if possible, a copy/pasteable script would be great. The websites are: https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=BY%2F8lhw%2BtbBgvOMDiHeB5A%3D%3D https://assess.joincyberdiscovery.com/challenge-files/clock-pt2?verify=BY%2F8lhw%2BtbBgvOMDiHeB5A%3D%3D https://assess.joincyberdiscovery.com/challenge-files/clock-pt3?verify=BY%2F8lhw%2BtbBgvOMDiHeB5A%3D%3D https://assess.joincyberdiscovery.com/challenge-files/clock-pt4?verify=BY%2F8lhw%2BtbBgvOMDiHeB5A%3D%3D https://assess.joincyberdiscovery.com/challenge-files/clock-pt5?verify=BY%2F8lhw%2BtbBgvOMDiHeB5A%3D%3D Keeping this up now only because I cannot take it down. Thank you to those who have helped, I hope this helps someone else. Sorry for being dumb

Answer 1

Thing is, I've done the code and tried it. It works, but that isn't the answer to the question. Getting the characters from the links and putting them together doesn't work. I've tried many things and I am still working it out myself. My advice, work it out yourself. It's a lot more rewarding and will probably help for future parts of the competition. Also, if you ever think about removing all of the 'a's from the code, that doesn't work either. I tried.

To answer your stack overflow question, here is the code (you need to install the 'requests' python modeule first):

import requests
page1 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
page1_content = requests.get(page1)
page1text = page1_content.text

page2 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt2?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
page2_content = requests.get(page2)
page2text = page2_content.text

page3 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt3?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
page3_content = requests.get(page3)
page3text = page3_content.text

page4 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt4?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
page4_content = requests.get(page4)
page4text = page4_content.text

page5 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt5?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
page5_content = requests.get(page5)
page5text = page5_content.text

print(page1text + page2text + page3text + page4text + page5text)

But this method doesn't answer challenge 14.

Answer 2

I have done something very similar with just as poor results at the end. I did, however, leave this running for a while and notice that the clock follow a pattern. Some time ago the clock read all as "aaaaaaaaaaaaaaa" then "aBaa1aafaa2aa3a" and "aDaafaaHaajaala". I'm going to wait for a full list and try suggesting the next clock sequence in the final URL. I'll get back to you if this works, just something to think about.

Also for help importing moduals I suggest : https://programminghistorian.org/lessons/installing-python-modules-pip & https://docs.python.org/3/installing/index.html

import requests
abc = ""
while 1 == 1 :
    page1 = requests.get('your first link')
    page2 = requests.get('your second link')
    page3 = requests.get('your thrid link')
    page4 = requests.get('your fourth link')
    page5 = requests.get('your fith link')
    text = page1.text+page2.text+page3.text+page4.text+page5.text

    # abc1 = "the verify link except clock pts is replaced with "+"text>" so the end looks like this :string=<"+text+">"
    abc1 = text
    if abc1 != abc:
       print (abc1)
       abc = abc1

Edit The clock runs in 15-minute cycles with 90 codes altogether Im not sure how this helps as of yet but just posting ideas. I had to make some changes to get the codes to output cleanly and here is my improved version (this is very messy sorry):

import requests
abc = ""
page1 = requests.get('your first link')
page2 = requests.get('your second link')
page3 = requests.get('your thrid link')
page4 = requests.get('your fourth link')
page5 = requests.get('your fith link')
while 1 == 1 :
    page12 = requests.get('your first link')
    page22 = requests.get('your second link')
    page32 = requests.get('your thrid link')
    page42 = requests.get('your fourth link')
    page52 = requests.get('your fith link')
    if page1.text != page12.text and page2.text != page22.text and page3.text != page32.text and page4.text != page42.text and page5.text != page52.text:


        text = page12.text+page22.text+page32.text+page42.text+page52.text
        abc1 = text
        # abc1 = * your url for verification   with * string=<"+text+">"
        if abc1 != abc:
            print (abc1)
            abc = abc1
            page1 = page12
            page2 = page22
            page3 = page32
            page4 = page42
            page5 = page52

Final edit I had sepnt so long going down the path of figuring out how that made the tak and doing way too much work. When Submitting the final url dont incluede your solutin as a repalcement for the section and NOT inside the <> so yours should like https://assess.joincyberdiscovery.com/challenge-files/get-flag?verify=*this is an identifiere*&string=*The string you get*

Answer 3

I know the answer to the question, but instead of giving the code to complete it, I'll tell you one of the ways you might find it, as I completed that question myself.

When you asked this question, you completely forgot to mention that there was a sixth link: https://assess.joincyberdiscovery.com/challenge-files/get-flag?verify=j7fPvtmWLDY5qeYFuJtmKw%3D%3D&string=%3Cclock%20pts%3E

Notice at the end of that hyperlink it says 'clock pts', whereas all the other links have had something like clock-pt1 or clock-pt4. What if the clock pts refers to all of the different links at once such as you have to create a string out of all the previous links you've been given, replace the 'clock pts' in the string section of the hyperlink WITH the string you made from the separate links, which would then give you the code to complete the level?

Below is the code I used to get the answer. It requires the requests module, in case you want to use it. (Also, I'm not 100% certain it will work all the time, since the challenge is based on a timer, the program may not get all the strings in time before the clock change, so make sure to run the program after the timer has reset)

    import requests
    page1 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
    page1_content = requests.get(page1)
    page1text = page1_content.text

    page2 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt2?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
    page2_content = requests.get(page2)
    page2text = page2_content.text

    page3 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt3?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
    page3_content = requests.get(page3)
    page3text = page3_content.text

    page4 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt4?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
    page4_content = requests.get(page4)
    page4text = page4_content.text

    page5 = "https://assess.joincyberdiscovery.com/challenge-files/clock-pt5?verify=4VjvSgWQQ8yhhiYD9cePtg%3D%3D"
    page5_content = requests.get(page5)
    page5text = page5_content.text

    code=(page1text + page2text + page3text + page4text + page5text)

    page6= "https://assess.joincyberdiscovery.com/challenge-files/get-flag?verify=j7fPvtmWLDY5qeYFuJtmKw%3D%3D&string="+code
    page6_content = requests.get(page6)
    print(page6_content.text)

Answer 4

I completed the challenge, I used an excel spreadsheet with functions to get all the little code things from every clock cycle and put them together to make one code every 10 seconds. Sorry if that doesn't make sense I'm not sure how to explain it. Then I pasted this into the end of the "validation link" to replace the < clock pts > at the end of the URL. I had to do this very fast before the clock reset. Very stressful haha. Then eventually I did this in time and it gave me the code. I hope this helps. But you'll have to figure out how to get all the codes together in under 10 seconds by yourself, otherwise this is basically cheating, right?