RuntimeValueProviderError when Running Dataflow Template Job

Question

Trying to figure out why I'm getting these errors. A quick search just resulted in answers that referred to a broken version, but it doesn't seem to be the case here. Creating the template works fine, but when I run it (and as I pass the limit arg) I get the error below. The idea is to build up the query based off of arguments provided in the template. If there's a better alternative to doing this, I'm open for it.

Code:

class Options(PipelineOptions):
    @classmethod
    def _add_argparse_args(cls, parser):
        parser.add_value_provider_argument(
            '--limit',
            default=0,
            type=int,
            help='Limit the amount of rows retrieved'
        )

...
    
def from_bq(options):
    with beam.Pipeline(options=options) as p:
        (p 
            | 'Read From BQ' >> beam.io.ReadFromBigQuery(query=NestedValueProvider(options.limit, create_query), use_standard_sql=True)
        ) 
    
def create_query(limit):
    query = """
        SELECT * FROM ...
    """

    if limit > 0:
        query = query + " LIMIT {limit}".format(limit=limit)

    return query

Error:

raise error.RuntimeValueProviderError('%s not accessible' % obj)
apache_beam.error.RuntimeValueProviderError: NestedValueProvider(value: RuntimeValueProvider(option: limit, type: int, default_value: 0), translator: create_query) not accessible [while running 'Read From BQ/Read/Split-ptransform-324']

Running apache-beam version 2.27.0.

Answer 1

I don't think this is possible with standard templates. You should look into using Flex Templates which have the full flexibility of non-template pipelines.