Rust Polars - expected fn pointer, found opaque type

Question

I am building a raku module to work with Rust Polars - via Rust ffi.

Generally it is working by passing opaque containers back and forth (SeriesC, ExprC and so on).

My first pass for the Expr.apply function looks like this (an extract):

use libc::c_char;
use libc::size_t;
use std::slice;
use std::ptr;
use std::ffi::*; //{CStr, CString,}
use polars::prelude::*;//{CsvReader, DataType, DataFrame, Series};
use polars::prelude::{Result as PolarResult};
use polars::lazy::dsl;
use polars::lazy::dsl::Expr;

pub struct SeriesC {
    se: Series,
}

impl SeriesC {
    fn new<T>(name: String, data: Vec::<T>) -> SeriesC
        where Series: NamedFrom<Vec<T>, [T]>
    {
        SeriesC {
            se: Series::new(&name, data),
        }
    }
// ...
}

type AppMap = extern fn(appmap: *mut SeriesC) -> SeriesC;

pub struct ExprC {
    pub inner: dsl::Expr,
}

impl ExprC {
    
    fn sum(&self) -> ExprC {
        self.clone().inner.clone().sum().into()
    }

// ... some more Exprs

    fn apply(&self, appmap: AMWrap) -> ExprC { 
        let o = GetOutput::from_type(DataType::UInt32);
        self.clone().inner.clone().apply(appmap, o).into()
    }
}

// ffi helper functions

#[no_mangle]
pub extern "C" fn ex_sum(ptr: *mut ExprC) -> *mut ExprC {
    let ex_c = unsafe {
        assert!(!ptr.is_null());
        &mut *ptr
    };

    Box::into_raw(Box::new(ex_c.sum()))
}

type AppMap = fn(*mut SeriesC) -> SeriesC;
type AMWrap = fn(Series) -> Result<Series>;

#[no_mangle]
pub extern "C" fn ex_apply(ptr: *mut ExprC, appmap: AppMap) -> *mut ExprC {
    let ex_c = check_ptr(ptr);


    fn c2s(se_c: SeriesC) -> Result<Series> {
        Ok(se_c.se.into_series())
    }

    fn s2c(series: Series) -> *mut SeriesC {
        let mut se_c = SeriesC::new::<u32>("dummy".to_owned(), [].to_vec());
        se_c.se = series;
        Box::into_raw(Box::new(se_c))
    }

    fn compose<A, B, C, F, G>(f: F, g: G) -> impl Fn(A) -> C
    where
        F: Fn(B) -> C,
        G: Fn(A) -> B,
    {
        move |x| f(g(x))
    }
    let am_wrap_a = compose(appmap, s2c);
    let am_wrap_b = compose(c2s, am_wrap_a);

    Box::into_raw(Box::new(ex_c.apply(am_wrap_b)))
}

This technique works fine for Expr's such as .sum()

For .apply() , as you can see I get a callback from the raku side (not shown), with type AppMap that takes and returns SeriesC containers and I am composing that into type AMWrap that takes and returns Series as required of the function pointer passed into the Expr.apply method.

I have managed to wrangle the Rust errors down to just one:

root@83471ed9aab4:~/raku-Dan-Polars/dan# cargo build
   Compiling dan v0.1.0 (/root/raku-Dan-Polars/dan)
error[E0308]: mismatched types
   --> src/lib.rs:881:39
    |
871 |     fn compose<A, B, C, F, G>(f: F, g: G) -> impl Fn(A) -> C
    |                                              --------------- the found opaque type
...
881 |     Box::into_raw(Box::new(ex_c.apply(am_wrap_b)))
    |                                       ^^^^^^^^^ expected fn pointer, found opaque type
    |
    = note: expected fn pointer `fn(polars::prelude::Series) -> std::result::Result<polars::prelude::Series, PolarsError>`
              found opaque type `impl Fn(polars::prelude::Series)-> std::result::Result<polars::prelude::Series, PolarsError>`

For more information about this error, try `rustc --explain E0308`.
error: could not compile `dan` due to previous error

But I can't get my head around how to bridge the return type of compose to the AMWrap type. Please can you help me?

Here is the Cargo.toml for completeness:

  1 [package]
  2 name = "dan"
  3 version = "0.1.0"
  4 edition = "2021"
  5 
  6 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
  7 
  8 [dependencies]
  9 libc = "0.2.126"
 10 polars = {version = "0.22.1", features = ["lazy"]}
 11 ffi-convert = "0.5.0"
 12 
 13 [lib]
 14 name = "dan"
 15 path = "src/lib.rs"
 16 crate-type = ["cdylib"]

Answer 1

Guess I made that question a bit impenetrable... after a few days I am close to an answer, like this:

 fn apply(&self, appmap: impl Fn(Series) -> Result<Series> + std::marker::Send + std::marker::Sync + 'static ) -> ExprC {
        let o = GetOutput::from_type(DataType::UInt32);
        self.clone().inner.clone().apply(appmap, o).into()
    }
}

type AppMap = fn(*mut SeriesC) -> SeriesC;
//type AMWrap = fn(Series) -> Result<Series>;

#[no_mangle]
//iamerejh ... do the impl trick here too to address not FFI-safe warning??
pub extern "C" fn ex_apply(ptr: *mut ExprC, appmap: AppMap) -> *mut ExprC {
    let ex_c = check_ptr(ptr);


    fn c2s(se_c: SeriesC) -> Result<Series> {
        Ok(se_c.se.into_series())
    }

    fn s2c(series: Series) -> *mut SeriesC {
        let mut se_c = SeriesC::new::<u32>("dummy".to_owned(), [].to_vec());
        se_c.se = series;
        Box::into_raw(Box::new(se_c))
    }

    //https://stackoverflow.com/questions/45786955/how-to-compose-functions-in-rust
    fn compose<A, B, C, F, G>(f: F, g: G) -> impl Fn(A) -> C
    where
        F: Fn(B) -> C,
        G: Fn(A) -> B,
    {
        move |x| f(g(x))
    }
    let am_wrap_a = compose(appmap, s2c);
    let am_wrap_b = compose(c2s, am_wrap_a);

    Box::into_raw(Box::new(ex_c.apply(am_wrap_b)))
}

Biggest concerns are that this forces a thread restriction... and there are some higher design aspects that I need to change, so will not be able to fine tune.

Hopefully that will help anyone who stumbles over here!