无法从插件文件夹 airflow 导入自定义运算符

Question

I am new to airflow and trying to import custom operator from plugins folder in my dag. Below is the file structure:

├── dags
│   ├── my_dag.py
├── myrequirements.txt
├── plugins
│   ├── __init__.py
│   ├── my_airflow_plugin.py
│   └── operators
│       ├── __int__.py
│       └── my_airflow_operator.py

my_dag.py

from operators.my_airflow_operator import AwsLambdaInvokeFunctionOperator

my_airflow_plugin.py

from airflow.plugins_manager import AirflowPlugin
from operators.my_airflow_operator import AwsLambdaInvokeFunctionOperator

class lambda_operator(LambdaOperator):
    pass
                    
class my_plugin(AirflowPlugin):
                    
    name = 'my_airflow_plugin'
    operators = [lambda_operator]

my_airflow_operator.py

from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
from airflow.contrib.hooks.aws_lambda_hook import AwsLambdaHook


class AwsLambdaExecutionError(Exception):
    """
    Raised when there is an error executing the function.
    """


class AwsLambdaPayloadError(Exception):
    """
    Raised when there is an error with the Payload object in the response.
    """


class AwsLambdaInvokeFunctionOperator(BaseOperator):
    """
    Invoke AWS Lambda functions with a JSON payload.
    The check_success_function signature should be a single param which will receive a dict.
    The dict will be the "Response Structure" described in
    """
    
    def succeeded(response):
        payload = json.loads(response['Payload'].read())
        # do something with payload
    
    @apply_defaults
    def __init__(
        self,
        function_name,
        region_name,
        payload,
        check_success_function,
        log_type="None",
        qualifier="$LATEST",
        aws_conn_id=None,
        *args,
        **kwargs,
    ):
        super().__init__(*args, **kwargs)
        self.function_name = function_name
        self.region_name = region_name
        self.payload = payload
        self.log_type = log_type
        self.qualifier = qualifier
        self.check_success_function = check_success_function
        self.aws_conn_id = aws_conn_id

    def get_hook(self):
        """
        Initialises an AWS Lambda hook
        :return: airflow.contrib.hooks.AwsLambdaHook
        """
        return AwsLambdaHook(
            self.function_name,
            self.region_name,
            self.log_type,
            self.qualifier,
            aws_conn_id=self.aws_conn_id,
        )

    def execute(self, context):
        self.log.info("AWS Lambda: invoking %s", self.function_name)

        response = self.get_hook().invoke_lambda(self.payload)

        try:
            self._validate_lambda_api_response(response)
            self._validate_lambda_response_payload(response)
        except (AwsLambdaExecutionError, AwsLambdaPayloadError) as e:
            self.log.error(response)
            raise e

        self.log.info("AWS Lambda: %s succeeded!", self.function_name)



    def _validate_lambda_response_payload(self, response):
        """
        Call a user provided function to validate the Payload object for errors.
        :param response: HTTP Response from AWS Lambda.
        :type response: dict
        :return: None
        """
        if not self.check_success_function(response):
            raise AwsLambdaPayloadError(
                "AWS Lambda: error validating response payload!"
            )

But i get this error: No module named 'operators'

I tried changing the import statement in my_dag.py to:

from airflow.operators.my_airflow_plugin import AwsLambdaInvokeFunctionOperator

I get this error No module named 'airflow.operators.my_airflow_plugin'

Could someone please suggest what is not right here?( Airflow version is 1.10.12)

init .py files are empty

Answer 1

What do you need the plugin for? Is it just to expose your custom operators in the DAGs? That's not really the purpose for plugins within the Airflow architecture, starting from the version 2 this is not longer supported.

Instead of using a plugin you can create a module with your operator and use that module to import it in the DAG. You can follow this instructions to do that.

Answer 2

My airflow version is 1.10.12 but still accessing custom operators from plugins folder was showing "No module found error". Finally it worked when i moved the custom operator file to dags folder.