Is it possible to share Rcpp object/pointer between R and Python?

[gregorgorjanc]

I am developing R access to the tskit C API (a library for working with massive genomic datasets by mapping how we relate to each other through our ancestors in the past - https://tskit.dev/). tskit has a very good C API that is further enhanced via a Python API. Both are very complete, well tested, and documented, but the Python API is more popular. I am well aware of reticulate to call Python API from R, but that doesn't suit all our needs(calling the tskit C API within loops in R/Rcpp, hence my effort to bring tskit C API to R/Rcpp).

I have developed an R package tskitr at https://github.com/HighlanderLab/tskitr which includes the tskit's C code and have 1) wrote some Rcpp functions that call the C API and 2) developed library build with instructions for other packages to use tskit C API via Rcpp in their work (the reason why I have built the package in the first place).

While this all works very well thanks to Rcpp, I don't want to duplicate the work done on the Python API. Most users will still be encouraged to analyse the objects using Python API, possibly via reticulate.

While we can build tskit objects on the R side (using Rcpp), export them to disk, load into Python and analyse in standalone or reticulate Python, I wonder if we could share Rcpp object/pointer between R and reticulate Python sessions? Doing a bit of LLMing, I surmise that the answer is no due to ABI incompatibility, but I thought I would ask real human experts! In the meanwhile I have written ts_r_to_py() and ts_py_to_r() that use disk-write-read to transfer between R and reticulate Python. tskit objects are complex so not sure I can figure out r_to_py()/py_to_r() wrappers and I only have a pointer in R session anyway - looking at the C++ code of the reticulate package I get lost very quickly.

Thanks!

[eddelbuettel]

From the top of my head, we should be able to send a pointer a struct around. That is what Arrow does after. But I do not have a working example nor can I think of one -- so what you do by serializing to disk seems like a fair approach. Alternatives would be to write to (in memory?) data bases like duckdb, or (if you remain on Linux) rely on shared memory to access a common object.

All these are great questions and would be really good to have a studied reference example.

[kevinushey]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

A somewhat bare version of what I'm imagining is the "advanced" Armadillo constructors, whereby you can create an Armadillo matrix from a pointer to auxiliary memory / data. https://arma.sourceforge.net/docs.html#adv_constructors_mat

[eddelbuettel]

Yup. It is a somewhat simpler example as Armadillo 'merely' reuses length * sizeof(x) for either an int or double vector. But the idea of the abstraction is the same: if we share the same C/C++ headers between Python and R so that sizeof(struct foo) is the same than we can instantiate a new object off a pointer. The other thought I had was that instead of serializing to disk as you do you could also look into one of the usual serialization-on-wire examples such as Protocol Buffers, MsgPack, ...

[gregorgorjanc]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

It's my understanding that Python API is wrapped around the C API, though I don't understand how this is done (I am not a Python developer). Will talk to tskit devs and come back.

[Enchufa2]

As Dirk said, this is exactly what Arrow does, so the answer is yes. You only need to ensure that both sides know the data structure, and then you pass the pointer around. r_to_py and py_to_r are S3 generics, so you just need to write your own methods for your data type. You can see what arrow does e.g. for arrow:::r_to_py.Array and arrow:::py_to_r.pyarrow.lib.Array.

And now that we are at it, I was taking a look and I see this. You are doing too much work here. You can just define a new XPtr type that knows the C function it needs to call to free the object it wraps. This is e.g. what units does here.

[gregorgorjanc]

Thanks @Enchufa2, this gives me plenty of study “pointers”!

[eddelbuettel]

Shared pointer is definitely one way to do this, and it is done by quite a few packages. Maybe search github.com/cran/ for XPtr or EXTPTRSXP or R_ExternalPtr. I really would like to construct a 'simple example' around this. Anybody else game for that?

[gregorgorjanc]

Here I am compiling pointers to what arrow does:

• r_to_py.Array() is defined here and looks like this:

r_to_py.Array <- function(x, convert = FALSE) {
  schema_ptr <- allocate_arrow_schema()
  array_ptr <- allocate_arrow_array()
  on.exit({
    delete_arrow_schema(schema_ptr)
    delete_arrow_array(array_ptr)
  })

  # Import with convert = FALSE so that `_import_from_c` returns a Python object
  pa <- reticulate::import("pyarrow", convert = FALSE)
  x$export_to_c(array_ptr, schema_ptr)

  out <- pa$Array$`_import_from_c`(
    pyarrow_compatible_pointer(array_ptr),
    pyarrow_compatible_pointer(schema_ptr)
  )

  # But set the convert attribute on the return object to the requested value
  assign("convert", convert, out)
  out
}

• allocate_arrow_array() (and delete_arrow_array()) look like:

R
# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/R/arrowExports.R#L259
allocate_arrow_array <- function() {
  .Call(`_arrow_allocate_arrow_array`)
}

# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/R/arrowExports.R#L263
delete_arrow_array <- function(ptr) {
  invisible(.Call(`_arrow_delete_arrow_array`, ptr))
}

C++

# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/src/bridge.cpp#L61
// [[arrow::export]]
arrow::r::Pointer<struct ArrowArray> allocate_arrow_array() { return {}; }

# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/src/bridge.cpp#L63C1-L64C86
// [[arrow::export]]
void delete_arrow_array(arrow::r::Pointer<struct ArrowArray> ptr) { ptr.finalize(); }

• x$export_to_c(array_ptr, schema_ptr) from here is a method on Array defined as an R6 class and it looks like:

...
    export_to_c = function(array_ptr, schema_ptr) ExportArray(self, array_ptr, schema_ptr)
...

• ExportArray() from here looks like:

// [[arrow::export]]
void ExportArray(const std::shared_ptr<arrow::Array>& array,
                 arrow::r::Pointer<struct ArrowArray> array_ptr,
                 arrow::r::Pointer<struct ArrowSchema> schema_ptr) {
  StopIfNotOk(arrow::ExportArray(*array, array_ptr, schema_ptr));
}

• pa$Array$_import_from_c from here looks like:

@staticmethod
    def _import_from_c(in_ptr, type):
        """
        Import Array from a C ArrowArray struct, given its pointer
        and the imported array type.

        Parameters
        ----------
        in_ptr: int
            The raw pointer to a C ArrowArray struct.
        type: DataType or int
            Either a DataType object, or the raw pointer to a C ArrowSchema
            struct.

        This is a low-level function intended for expert users.
        """
        cdef:
            void* c_ptr = _as_c_pointer(in_ptr)
            void* c_type_ptr
            shared_ptr[CArray] c_array

        c_type = pyarrow_unwrap_data_type(type)
        if c_type == nullptr:
            # Not a DataType object, perhaps a raw ArrowSchema pointer
            c_type_ptr = _as_c_pointer(type)
            with nogil:
                c_array = GetResultValue(ImportArray(
                    <ArrowArray*> c_ptr, <ArrowSchema*> c_type_ptr))
        else:
            with nogil:
                c_array = GetResultValue(ImportArray(<ArrowArray*> c_ptr,
                                                     c_type))
        return pyarrow_wrap_array(c_array)

• pyarrow_compatible_pointer(array_ptr) from here looks like:

pyarrow_compatible_pointer <- function(ptr) {
  # GH-39933: Workaround because there is no built-in way to send a
  # 64-bit integer to Python from an R object
  py <- reticulate::import_builtins(convert = FALSE)
  addr <- external_pointer_addr_character(ptr)
  py$int(addr)
}

• external_pointer_addr_character() from here looks like:

// [[arrow::export]]
std::string external_pointer_addr_character(SEXP external_pointer) {
  void* ptr_void = R_ExternalPtrAddr(external_pointer);
  uint64_t ptr_int64 = reinterpret_cast<uintptr_t>(ptr_void);
  return std::to_string(ptr_int64);
}

• py_to_r.pyarrow.lib.Array() from here looks like:

py_to_r.pyarrow.lib.Array <- function(x, ...) {
  schema_ptr <- allocate_arrow_schema()
  array_ptr <- allocate_arrow_array()
  on.exit({
    delete_arrow_schema(schema_ptr)
    delete_arrow_array(array_ptr)
  })

  x$`_export_to_c`(
    pyarrow_compatible_pointer(array_ptr),
    pyarrow_compatible_pointer(schema_ptr)
  )

  Array$import_from_c(array_ptr, schema_ptr)
}

• _export_to_c() from here looks like:

    def _export_to_c(self, out_ptr, out_schema_ptr=0):
        """
        Export to a C ArrowArray struct, given its pointer.

        If a C ArrowSchema struct pointer is also given, the array type
        is exported to it at the same time.

        Parameters
        ----------
        out_ptr: int
            The raw pointer to a C ArrowArray struct.
        out_schema_ptr: int (optional)
            The raw pointer to a C ArrowSchema struct.

        Be careful: if you don't pass the ArrowArray struct to a consumer,
        array memory will leak.  This is a low-level function intended for
        expert users.
        """
        cdef:
            void* c_ptr = _as_c_pointer(out_ptr)
            void* c_schema_ptr = _as_c_pointer(out_schema_ptr,
                                               allow_null=True)
        with nogil:
            check_status(ExportArray(deref(self.sp_array),
                                     <ArrowArray*> c_ptr,
                                     <ArrowSchema*> c_schema_ptr))

• And registering S3 methods

.onLoad <- function(...) {
  for (cl in c("Array", "ChunkedArray")) {
    s3_register("reticulate::py_to_r", paste0("pyarrow.lib.", cl))
    s3_register("reticulate::r_to_py", cl)
  }
}

[eddelbuettel]

There may be simpler ways. We are essentially discussing the (really old and recurrent) problem of portable serialization. I have done some simpler case with writing to / reading from Redis. Same could be done with other database. And/or serialization to (portable) binary streams via something like Cereal, or Protocol Buffers, or Thrift.

On the other an R external pointer is just that: a portable (!!) way to pass a pointer to a object (a struct, say) around. I am just not a Python programmer so I cannot help you with how to pick the pointer up there. That (to me) is a missing piece.

[Enchufa2]

As you can see they're not doing much, despite the amount of boilerplate seems to suggest. They just pass a pointer around and read it into the proper struct at the other side. The problem is to pass a 64-bit pointer when we have 32-bit integers in R. So they just convert it to a string, then convert that string back to int with Python.

[gregorgorjanc]

For completness adding here link to onether tab I had openned on "Integrating PyArrow with R", but the above code "summary" is the gist of how arrow goes about sending pointers between R and Python.

@Enchufa2 I agree that the code does not look complicated, but the sheer amount of boilerplate is a bit off-putting;)

[eddelbuettel]

@gregorgorjanc This topic, as mentioned, is important, and I would like to if possible illustrate it.

You started by saying that you wanted to with data structures from tskit. I presume those would 'classic' C or C++ structs, and not necessarily Arrow data structures (which generally vectors, or groups of vectors as in a data.frame)?

If so, then working with an external pointer (XPtr to Rcpp) and a PyCapsule on the Python side may work. There is for example [this PR to reticulate](rstudio/reticulate#674 supporting just that. Now, I personally do too little Python to be able to help on other side but maybe someone else can fill in that gap?

[eddelbuettel]

@gregorgorjanc And for completeness, if a standard data.frame is good then here is an example where I use nanoarrow to wrap an existing data.frame I have in R -- I use mtcars; I think penguins stumbles over NA values using penguins also works -- and pass it to rpolars (as a check) and to the Python Polars package via reticulate.

#!/usr/bin/env Rscript

suppressMessages({
    ## reticulate uses ~/.cache/R/reticulate/uv
    ## with the virtualenv in ~/.virtualenv/r-reticulate
    Sys.unsetenv("RETICULATE_PYTHON")
    library(reticulate)
    library(polars)
})

## we add the rownames as an explicit column here
mtcars_df <- data.frame(model=rownames(mtcars), mtcars)
## we use a nanoarrow 'RecordBatch'
narb <- nanoarrow::as_nanoarrow_array(mtcars_df)

viaR <- function(narb) {
    pdf <- polars::as_polars_df(narb)
    print(pdf)

    invisible(NULL)
}

## local helper to unpack pyCapsule
capsule_prep <- function(na) {
    ## pointer addresses for array and schema in 'long long int' format
    arrchr <- nanoarrow::nanoarrow_pointer_addr_chr(na)
    schchr <- nanoarrow::nanoarrow_pointer_addr_chr(nanoarrow::infer_nanoarrow_schema(na))

    ## Import with convert = FALSE so that `_import_from_c` returns a Python object
    py <- reticulate::import_builtins(convert = FALSE)
    pa <- reticulate::import("pyarrow", convert = FALSE)
    out <- pa$Array$`_import_from_c`(py$int(reticulate::r_to_py(arrchr)),
                                     py$int(reticulate::r_to_py(schchr)))
    out
}

viaPython <- function(narb) {
    py_require("polars")

    ## now import Polars
    po <- import("polars")

    ## local helper function until `nanoarrow` catches up
    na2 <- capsule_prep(narb)

    ## use it to import the Table into a Polars DataFrame
    df <- po$from_dataframe(na2)
    print(df)

    invisible(NULL)
}


viaR(narb)      # works
viaPython(narb) # also works

This still needs one 'hand shake' via the capsule_prep() helper. IIRC the corresponding functionality should be baked into the next nanoarrow release too. I have a few examples like this as I prefer to work on this on the C++ side via nanoarrow, without having to compile or link with the (heavier) Arrow C++ libraries, and can still pass Arrow objects around.

For completeness the output is

$ Rscript mtcars_polars.R
shape: (32, 12)
┌───────────────────┬──────┬─────┬───────┬───┬─────┬─────┬──────┬──────┐
│ model             ┆ mpg  ┆ cyl ┆ disp  ┆ … ┆ vs  ┆ am  ┆ gear ┆ carb │
│ ---               ┆ ---  ┆ --- ┆ ---   ┆   ┆ --- ┆ --- ┆ ---  ┆ ---  │
│ str               ┆ f64  ┆ f64 ┆ f64   ┆   ┆ f64 ┆ f64 ┆ f64  ┆ f64  │
╞═══════════════════╪══════╪═════╪═══════╪═══╪═════╪═════╪══════╪══════╡
│ Mazda RX4         ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ Mazda RX4 Wag     ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ Datsun 710        ┆ 22.8 ┆ 4.0 ┆ 108.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 1.0  │
│ Hornet 4 Drive    ┆ 21.4 ┆ 6.0 ┆ 258.0 ┆ … ┆ 1.0 ┆ 0.0 ┆ 3.0  ┆ 1.0  │
│ Hornet Sportabout ┆ 18.7 ┆ 8.0 ┆ 360.0 ┆ … ┆ 0.0 ┆ 0.0 ┆ 3.0  ┆ 2.0  │
│ …                 ┆ …    ┆ …   ┆ …     ┆ … ┆ …   ┆ …   ┆ …    ┆ …    │
│ Lotus Europa      ┆ 30.4 ┆ 4.0 ┆ 95.1  ┆ … ┆ 1.0 ┆ 1.0 ┆ 5.0  ┆ 2.0  │
│ Ford Pantera L    ┆ 15.8 ┆ 8.0 ┆ 351.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 4.0  │
│ Ferrari Dino      ┆ 19.7 ┆ 6.0 ┆ 145.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 6.0  │
│ Maserati Bora     ┆ 15.0 ┆ 8.0 ┆ 301.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 8.0  │
│ Volvo 142E        ┆ 21.4 ┆ 4.0 ┆ 121.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 2.0  │
└───────────────────┴──────┴─────┴───────┴───┴─────┴─────┴──────┴──────┘
shape: (32, 12)
┌───────────────────┬──────┬─────┬───────┬───┬─────┬─────┬──────┬──────┐
│ model             ┆ mpg  ┆ cyl ┆ disp  ┆ … ┆ vs  ┆ am  ┆ gear ┆ carb │
│ ---               ┆ ---  ┆ --- ┆ ---   ┆   ┆ --- ┆ --- ┆ ---  ┆ ---  │
│ str               ┆ f64  ┆ f64 ┆ f64   ┆   ┆ f64 ┆ f64 ┆ f64  ┆ f64  │
╞═══════════════════╪══════╪═════╪═══════╪═══╪═════╪═════╪══════╪══════╡
│ Mazda RX4         ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ Mazda RX4 Wag     ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ Datsun 710        ┆ 22.8 ┆ 4.0 ┆ 108.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 1.0  │
│ Hornet 4 Drive    ┆ 21.4 ┆ 6.0 ┆ 258.0 ┆ … ┆ 1.0 ┆ 0.0 ┆ 3.0  ┆ 1.0  │
│ Hornet Sportabout ┆ 18.7 ┆ 8.0 ┆ 360.0 ┆ … ┆ 0.0 ┆ 0.0 ┆ 3.0  ┆ 2.0  │
│ …                 ┆ …    ┆ …   ┆ …     ┆ … ┆ …   ┆ …   ┆ …    ┆ …    │
│ Lotus Europa      ┆ 30.4 ┆ 4.0 ┆ 95.1  ┆ … ┆ 1.0 ┆ 1.0 ┆ 5.0  ┆ 2.0  │
│ Ford Pantera L    ┆ 15.8 ┆ 8.0 ┆ 351.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 4.0  │
│ Ferrari Dino      ┆ 19.7 ┆ 6.0 ┆ 145.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 6.0  │
│ Maserati Bora     ┆ 15.0 ┆ 8.0 ┆ 301.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 8.0  │
│ Volvo 142E        ┆ 21.4 ┆ 4.0 ┆ 121.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 2.0  │
└───────────────────┴──────┴─────┴───────┴───┴─────┴─────┴──────┴──────┘
$ 

[eddelbuettel]

Ah. And I just learned in the nanoarrow thread that helped me with all this (see here) that if the Python variant of nanoarrow is installed we no longer need the capsule_prep(). One does however need the pre-release version of R's nanoarrow but this should get updated at CRAN shortly.

[eddelbuettel]

@gregorgorjanc I have a complete, working and (at least I think so) quite interesting demo using external pointers. I wrote a minimal Python package, and I think it mirrors your situation.

I was looking around for a small and simple enough struct or class in C++ that I already wrapped on the R side. And I found a gem I had hidden away in package RcppSpdlog -- upstream has a very neat and very minimal stopwatch class. This is similar to the tic() and toc() genre ... but even simpler, and now shareable with Python. For reference, the original class is here, my derived more minimal one in RcppSpdlog is here.

Back to our use case. A nice feature is that all it took in Python for this particular example was to add a) one new constructor taking a string encoding the memory address of the object from this shared C++ class and b) exposing the class (and of course this new constructor) via pybind11. So this may well fit your case as you have access to the Python bindings of your tskit. The string takes the address of the external pointer we already have in R.

I will clean this up a little and make the Python package a repo, and will then blog or write a bit more. That may take me a day or two. In the meantime, a demo (from R) follows.

> library(reticulate)
> use_virtualenv("/opt/venv/stopwatch")
> cc <- import("stopwatch")                      # access the Python object from R
> 
> ## quick demo just from the R side
> sw <- RcppSpdlog::get_stopwatch()              # we use a simple C++ struct as example
> Sys.sleep(1.234)                               # imagine doing some code here
> print(sw)                                      # stopwatch shows elapsed time
1.399661 
>
> xptr::is_xptr(sw)                             # from R, a stopwatch is an external pointer (plus two S3 methods)
[1] TRUE
> xptr::xptr_address(sw)                        # access where the external pointer points to, format is "0x...."
[1] "0x5bde23be23b0"
>
> sw2 <- xptr::new_xptr(xptr::xptr_address(sw))      # cloned (!!) but that is unclassed, so an external pointer
> attr(sw2, "class") <- c("stopwatch", "externalptr")# class it .. and use it!
> print(sw2)                                         # so `xptr` allows us close and use
19.477163 
>
> sw3 <- cc$Stopwatch(  xptr::xptr_address(sw) )     # and so does the Python object _with an added string ctor_
> print(sw3$elapsed())
datetime.timedelta(seconds=24, microseconds=69720)   # different output _format_ as we hit the Python formatter
>
> print(sw)                                          # and it still works on the R side
36.453815 
> 

We can also run this as one script (and I reduced the sleep to 0.5s):

$ Rscript example.R
0.500946 
[1] TRUE
[1] "0x5711cd7e48f0"
0.502682 
datetime.timedelta(microseconds=503092)
0.503710 
$ 

which shows the marginal 'cost' accruing from running the R (and Python) interpreter and crossing back and forth. Total cost including the standard output printing (and of course ignoring the demo 'sleep' of half a second) is 3.7 milliseconds.