Python Api 请求关键字研究工具

Question

I am currently trying to build a script which allows to do keyword research at scale (for SEO) by using the Grepwords API.

I have come across a few issues along the way (also still very new to Python so any help is really appreciated :-))

I would like the following script to be able to handle an input file containing ~600k keywords and return the search volume for these.

One of the first issues I had a was a TimeoutError which I tried to solve by using the sleep module, however now i am getting a HTTPError: HTTP Error 502: Bad Gateway

How would I also update the code so it creates a new file after every 10,000th row? Thanks

import csv
import urllib.request
import urllib.parse
import json
import time


# Construct the URL template.
root_url = 'http://api.grepwords.com/lookup?apikey='
api_key = 'xxx'
country = 'united_kingdom'
url_template = root_url + api_key + '&loc=' + country + '&q='

# Read from the source file.
keywords = []
with open("example.csv", 'r') as input_file:
   fileReader = csv.reader(input_file, delimiter=',')
   for row in fileReader:
       keywords.append(row)

# Get and format the output.
output = [['Country: ' + country], [], ['Keyword', 'ams']]
for keyword in keywords:
   # Construct the final URL.
   parsed_keyword = urllib.parse.quote(keyword[0])
   url = url_template + parsed_keyword

   # Query the API.
   all_keyword_data = json.loads(urllib.request.urlopen(url).read())
   time.sleep(2) 

   try:
       estimated_impressions = all_keyword_data["keywords"][keyword[0]]['ams']
   except TypeError:
      ams = "NA"

   # Prepare the output data.
   keyword_data = [keyword[0], ams]
   output.append(keyword_data)

# Write to the output file.
with open("examplefile2.csv", 'w', newline='') as output_file:
   fileWriter = csv.writer(output_file, delimiter=',')
   for row in output:
       fileWriter.writerow(row)

Answer 1

You may want to contact the grepwords admin team to ask why their service is giving you 502 HTTP errors, this indicates something is broken on their side.

That said, sending 600k separate requests is certainly excessive, and likely to lead to issues. Only the top-tier Enterprise pricing plan lets you query for that many keywords in a month. You may be running into a rate limit where you have exceeded your plan quota.

Next, if you do have such a generous plan, then the API allows for multiple keywords to be sent, per query. This can be used to cut back greatly on the number of API calls you have to make, but not on the number of keywords you query (so it won't let you get more keywords queried against your paid plan). The number you can send is limited by the limits of what URL size the server accepts; generally speaking you can count on 2k characters as a reasonable limit.

You also want to avoid reading all of your keywords into memory at once, just loop over the file directly to read keywords from it as needed (up to what will fit in a single API query). Your input file doesn't appear to be a CSV format since every line is just a single keyword, no need to use the csv module in that case.

Next, you want to use a better HTTP library, one that can reuse open connections . This reduces the strain both on your network side and on the API service. I recommend you use the requests library , it offers the most user-friendly API for this. Use a single session object to ensure the connection is reused where possible.

Writing out to separate files for every 10k results is trivial enough, just keep a counter. I've created a new class below that encapsulates that behaviour, and output files are closed when you reach the maximum, and a new file is opened as more rows are written, automatically. The files produced use a sequential numbering scheme, so examplefile-1.csv , examplefile-2.csv , etc.

I've also encapsulated the keyword grouping and API calls into separate functions, to aid readability:

import requests

url = 'http://api.grepwords.com/lookup'
api_key = 'xxx'
keywords_filename = "example.csv"
output_filename = 'examplefile-{}.csv'  # a template, will get an order number

base_query = {
    'api_key': api_key,
    'country': 'united_kingdom',
}

# determine a maximum length for the keywords query
base_length = len(requests.Request('GET', url, params=base_query).prepare().url) + 3  # url + the &q= characters
max_length = 2000 - base_length  # max length of the keywords plus | symbols


class RotatingCSVOutput:
    """CSV writer object that rotates files after every max_rows rows have been written

    A new file is only opened if there are rows to write

    """
    def __init__(self, base_filename, *args, max_rows=10000, **kwargs):
        self._base = base_filename
        self._max = max_rows
        self._open_file = None
        self._writer = None
        self._rows_written = 0
        self._file_counter = 1
        self._args, self._kwargs = args, kwargs

    def _open(self):
        self._open_file = open(self._base.format(self._file_counter), 'w', newline='')
        self._writer = csv.writer(self._open_file, *self._args, **self._kwargs)
        self._file_counter += 1

    def _close(self):
        if self._open_file is not None:
            self._open_file.close()
            self._open_file = None
            self._writer = None

    def __enter__(self):
        return self

    def __exit__(self, *args):
        self._close()

    def writerow(self, row):
        if self._open_file is None:
            self._open()
        self._writer.writerow(row)
        self._rows_written += 1
        if self._rows_written % self._max == 0:
            # close after max row count is reached; a new file is opened on demand.
            self._close()

    def writerows(self, rows):
        for row in rows:
            self.writerow(row)


def max_keywords(keywords, max):
    """Produce a list of keywords from the input sequence fitting within max characters

    The length of each keyword is padded with 1 character to allow for the separator

    """
    group = []
    length = 0

    for keyword in keywords:
        if length + len(keyword) > max:
            # our group is filled up, yield it and set up a new group
            yield group

            group, length = [], 0

        group.append(keyword)
        length += len(keyword) + 1  # count the | separator too

    if group:
        # last of the keywords at the end of the file
        yield group


def fetch_impressions(session, keywords):
    """Fetch the impressions value for each of the keywords from the API"""
    parameters = {'q': '|'.join(keywords), **base_query}
    response = session.get(url, params=parameters)
    response.raise_for_status()  # abort if there is an error code
    data = response.json().get('keywords', {})
    for keyword in keywords:
        yield keyword, data.get(kw, {}).get('ams', 'NA')


with open(keywords_filename, 'r') as input_file,\
        requests.Session() as session,\
        RotatingCSVOutput(output_filename) as output:

    stripped_lines = (line.strip() for line in input_file)
    for keywords in max_keywords(stripped_lines, max_length)
        result_rows = fetch_impressions(session, keywords)
        output.writerows(result_rows)