SLinC Update Dec 6, 2021

Over the past month I've been developing SLinC, a library to generate bindings to C functions from pure Scala code using Java 17's foreign incubator.

I'm doing my best to design SLinC to be easy to use to generate C bindings, while still having them be performant, and I think this second update reaches that goal.

Binding C functions

C functions are fairly simple to bind in SLinC. You write a method definition with the name of the C function, and the params with corresponding scala types, and use the bind macro and SLinC will generate a binding for you.

def atof(string: String)(using SegmentAllocator): Double = bind

A using SegmentAllocator clause is needed in the function definition if one or more of the parameter inputs, or the result, requires allocation of native memory. In this case, Strings require allocation because the String must be copied into native memory as a C-String before it can be passed on. In general, Strings and Struct types cause the need of a SegmentAllocator.

When using a method binding that requires an allocator, you can wrap it in a scope.

val result = scope {
  atof("5.0")
} //result == 5.0

Scopes control memory allocation and deallocation. Once a scope has ended, any memory that was allocated using that scope will be immediately freed.

Defining Structs

Structs in SLinC are represented by standard case classes which derive the Struct typeclass.

case class div_t(quot: Int, rem: Int) derives Struct

Structs can be serialized (pushed into native memory) and passed to/returned from functions.

val ptr: Ptr[div_t] = div_t(5,2).serialize
def div(numer: Int, denom: Int)(using SegmentAllocator): div_t = bind

scope{
  div(5,2)
} //result == div_t(2,1)

Structs are pass by value. When used as a method parameter, the structs contents will be copied into native memory for usage by the C world. When used as a method return, the struct data will be copied out of native memory into the JVM heap.

Pointers

Pointers in SLinC are datatypes that have information about accessing a region of native memory. They have a dereference operator and an update operation.

val ptr = div_t(5,2).serialize //push the struct into native memory

!ptr //result == div_t(5,2)
!ptr = div_t(2,1)
!ptr //result == div_t(2,1)

Pointer dereferencing/updating involves copying the entire data type being pointed to into/out of native memory. This can be fairly expensive, especially if you only want to update a small portion of native memory, so partial dereferencing has been provided.

!ptr.partial.quot //result == 2
!ptr.partial.quot = 0
!ptr //result == div_t(0,1)

Partial dereferencing enriches the pointer type (which uses Scala 3's programmatic structural types) with a set of members that are Ptr versions of the original members of the struct. In short, ptr.partial gives you access to the following type:

Ptr[div_t] {
  val quot: Ptr[Int]
  val rem: Ptr[Int]
}

This enriched type is calculated on a per-struct basis at compile-time and is fully compatible with nested structs.

Libraries

New types have been added to allow users to write bindings for non-standard lib libraries: Library and Location.

Library is used to define bindings for a library as well as how to load the library.

object Testlib extends Library(Location.Local("slinc/test/native/libtest.so")):
   case class a_t(a: Int, b: Int) derives Struct
   case class b_t(c: Int, d: a_t) derives Struct

   def slinc_test_modify(b_t: b_t)(using SegmentAllocator): b_t = bind

In the above example, we bind to a library that is at slinc/test/native/libtest.so, relative to the current working directory of the program. System is a location type used to load libraries that are on the system path, and Absolute is for binding to libraries using an absolute filesystem path. These values and bindings are not done at compile-time, but rather when the object is first accessed, so you can have your program search for a library and then use that path as the basis for your library binding. Likewise, it's not necessary to have your library binding be an object. It can be a class or trait as well.

Performance

At present, I've been mainly comparing SLinC performance to JNA. I am too lazy to learn and write JNI code, and JNR has proven to be difficult to get the bindings right for. At present, SLinC is 50x more performant on benchmarks like calling div from the standard library as opposed to JNA. When it comes to simpler methods like getpid SLinC is still more performant, but the difference is way less stark.

I plan on adding features that will help SLinC's performance in the near future, like something like a young generation native memory segment for copying structs into for copy by value parameters and such.

Behind the scenes

Most of what SLinC does at the moment is cut down on boilerplate via macros. A quick example of this is a binding to localtime.

def localtime(timer: Ptr[Long]): Ptr[tm] = { 
  val address: jdk.incubator.foreign.MemoryAddress = io.gitlab.mhammons.polymorphics.MethodHandleHandler.call1(io.gitlab.mhammons.slinc.components.UniversalNativeCache.addMethodHandle(8, io.gitlab.mhammons.slinc.components.Linker.linker.downcallHandle(io.gitlab.mhammons.slinc.components.SymbolLookup.given_SymbolLookup.lookup("localtime"), java.lang.invoke.MethodType.methodType(io.gitlab.mhammons.slinc.components.LayoutOf.given_LayoutOf_Ptr[StdlibSuite.this.tm].carrierType, io.gitlab.mhammons.slinc.components.LayoutOf.given_LayoutOf_Ptr[scala.Long].carrierType, ), jdk.incubator.foreign.FunctionDescriptor.of(io.gitlab.mhammons.slinc.components.LayoutOf.given_LayoutOf_Ptr[StdlibSuite.this.tm].layout, io.gitlab.mhammons.slinc.components.LayoutOf.given_LayoutOf_Ptr[scala.Long].layout))), timer.asMemoryAddress).asInstanceOf[jdk.incubator.foreign.MemoryAddress]
  new io.gitlab.mhammons.slinc.Ptr[StdlibSuite.this.tm](address.asSegment(StdlibSuite.this.tm.derived$Struct.layout.byteSize(), address.scope()), 0L, scala.Predef.Map.empty[java.lang.String, scala.Any])
}

Todo

There is still a lot left to do on SLinC. I must test it with all sorts of functions and libraries, and I must implement support for function pointers, padding, union types, varargs, and more. I will hopefully tackle these one-by-one in the coming months. I believe once I have a more solid subset of C features implemented I will release a v0.0.1 version of SLinC for others to make use of.

You can find the current version of SLinC at Gitlab and Github