使用标准格式搜索网址

Question

I am trying to figure out how to search through AWS .xml metadata files to check whether or not a particular imagery tile from Landsat or Sentinel meets my requirements.

the files for these data products follow a standard url format:

http://sentinel-s2-l1c.s3.amazonaws.com/tiles/10/S/DG/2015/12/7/0/metadata.xml

the format includes references to the Military grid reference system and the date the that the image was captured, what i'd like to do is to search through available URLs for a given tile so any available .xml url

http://sentinel-s2-l1c.s3.amazonaws.com/tiles/10/S/DG/2015/../../0/metadata.xml

so in the above example 10 is the utm zone, S is the latitude, and DG is the specific tile, so I would like a way to find and read all the metadata.xml files for a given tile in a given year.

I really have no idea how to go about this, but I have some experience with python and Java any help or resources to look at would be greatly appreciated

Answer 1

Consider a two-fold procedure. One that checks URLs and if valid, downloads each XML and then deletes any request error XMLS. Use python's built-in os module for the filesystem work.

Note : below script saves files in an existing subfolder called AWS relative to running .py script. Deletion loop only removes folders in this subfolder:

import os
import requests as rq

baseurl = 'http://sentinel-s2-l1c.s3.amazonaws.com/tiles/10/S/DG/2015/{}/{}/0/metadata.xml'

# ITERATE THROUGH ALL MONTH / DAY COMBINATIONS
for i in [(m, d) for m in range(1,13) for d in range(1,31)]:    

    if request.status_code == 200:
        rqpage = rq.get(baseurl.format(i[0], i[1]))
        rqcontent = rqpage.content

        with open('AWS/{}-{}-{}_metadata.xml'.format('2015', i[0], i[1]), 'wb') as f:
            f.write(rqcontent)        

# REMOVE BY SIZE ERROR RETURNED XML (WHICH ARE STILL VALID URLS)
for d, subdir, files in os.walk('AWS'):
    for f in files:
        if os.stat(os.path.join(d, f)).st_size < 400:
            os.remove(os.path.join(d, f))

Output