Implementing decorators in terms of closures with pyo3

Question

As a learning exercise I'm trying to implement a parameterised decorator function in pyo3 using closures. The pyo3 documentation contains an example of a (non-parameterised) decorator implemented as a class with a __call__ method, and I've built on this and created a parameterised decorator using an outer class with a __call__ method that returns an inner class with a __call__ that invokes the target function, and it works. But as a learning exercise (I could do with improving my understanding of lifetimes especially) I wanted to try to implement the same thing in terms of closures. (NB I've previously done this with lambdas in C++)

So my non-parameterised decorator, after some experimenting and fighting with the compiler, looks like this:

#[pyfunction]
pub fn exectime(py: Python, wraps: PyObject) -> PyResult<&PyCFunction> {
    PyCFunction::new_closure(
        py, None, None,
        move |args: &PyTuple, kwargs: Option<&PyDict>| -> PyResult<PyObject> {
            Python::with_gil(|py| {
                let now = Instant::now();
                let ret = wraps.call(py, args, kwargs);
                println!("elapsed (ms): {}", now.elapsed().as_millis());
                ret
            })
        }
    )
}

Note I needed to wrap the captured py in a Python::with_gil to make it work. Trying to extend this to a nested decorator I came up with:

#[pyfunction]
pub fn average_exectime(py: Python, n: usize) -> PyResult<&PyCFunction> {
    let f = move |args: &PyTuple, _kwargs: Option<&PyDict>| -> PyResult<&PyCFunction> {
        Python::with_gil(|py| {
            let wraps: PyObject = args.get_item(0)?.into();
            let g = move |args: &PyTuple, kwargs: Option<&PyDict>| -> PyResult<PyObject> {
                Python::with_gil(|py| {
                    let now = Instant::now();
                    for _ in 0..n-1 {
                        wraps.call(py, args, kwargs);
                    }
                    let ret = wraps.call(py, args, kwargs);
                    println!("elapsed (ms): {}", now.elapsed().as_millis());
                    ret
                })
            };
            PyCFunction::new_closure(py, None, None, g)
        })
    };
    PyCFunction::new_closure(py, None, None, f)
}

for which the compiler tells me:

error: lifetime may not live long enough ] 44/45: poetry-rust-integration
  --> src/decorator.rs:48:13
   |
35 |         Python::with_gil(|py| {
   |                           --- return type of closure is Result<&'2 pyo3::types::PyCFunction, pyo3::PyErr>
   |                           |
   |                           has type `pyo3::Python<'1>`
...
48 |             PyCFunction::new_closure(py, None, None, g)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`


    Building [=========================> ] 44/45: poetry-rust-integration                                                                                                                              error: aborting due to previous error

I've tried all sorts of lifetime parameters including enclosing lifetimes to no avail, I just end up with more errors. I guess I don't understand why the compiler thinks the inner lifetime must outlive the other? Isn't it sufficient to tell the complier they both have the same lifetime? And if so, how to achieve this?

Answer 1

pub fn new_closure<'a, F, R>(
    py: Python<'a>,
    name: Option<&'static str>,
    doc: Option<&'static str>,
    closure: F
) -> PyResult<&'a PyCFunction>

So the &PyCFunction you get is only valid for the extent of the with_gil block.

To make the function outlive that GIL block you need to convert it to a GIL-independent reference .

Answer 2

Many thanks to Masklinn for explaining the error and pointing me in the right direction. I now have this working solution:

#[pyfunction]
pub fn average_exectime(py: Python, n: usize) -> PyResult<&PyCFunction> {
    let f = move |args: &PyTuple, _kwargs: Option<&PyDict>| -> PyResult<Py<PyCFunction>> {
        Python::with_gil(|py| {
            let wraps: PyObject = args.get_item(0)?.into();
            let g = move |args: &PyTuple, kwargs: Option<&PyDict>| -> PyResult<PyObject> {
                Python::with_gil(|py| {
                    let now = Instant::now();
                    let mut result: PyObject = py.None();
                    for _ in 0..n {
                        result = wraps.call(py, args, kwargs)?;
                    }
                    println!("elapsed (ms): {}", now.elapsed().as_millis());
                    Ok(result)
                })
            };
            match PyCFunction::new_closure(py, None, None, g) {
                Ok(r) => Ok(r.into()),
                Err(e) => Err(e)
            }
        })
    };
    PyCFunction::new_closure(py, None, None, f)
}