Best way to pass large binary data to C from Lua

Question

I have an app where I get a large binary data from network (500KB) via LuaSockets. (Didn't implement this part yet). Based on what I know this data will be stored in some sort of String or a table or an array.

I need to complex processing on this data and I have an existing, proven ANSI-C Lib for this, I need to pass the data to the C. The approaches I have looked at usually pass the data in the call stack which would not work for me due to the data size.

The existing ANSI-C lib expects a pointer to the data and the size. I can probably wrap this API with a new API where it excepts the data byte by byte and finally once the whole size is established call the real API however this means huge number of calls to some C function from Lua and not preferred since I want performance.

What is the best way to pass this large data to C? (I have considered writing to a file and passing file name but this is not preferred, I really want a direct memory pass)

Answer 1

Assuming that LuaSocket provides data as a string, then it's really simple.

Your C application will be passed the string directly from Lua. So it's on the Lua stack. Use lua_tolstring to get the string and it's size. This will return a pointer to Lua-owned memory (so you need to ensure that it remains on the Lua stack or in the registry or somewhere while you're using it, to prevent Lua from garbage collecting it). Then pass that string+size to your API of choice.

Answer 2

In my opinion you dont have to care for the Lua stack size. Most if not every element on the Lua stack are actually heap allocated. If you are usinf luasocket, then the binary data you received is already available as Lua string (the return value of socket:receive i guess), so you are not extra-allocating anything! The Lua stack actually is more likely to pass around sort of references, so copying does not take place or is very cheap. To interface your C lib you may: a) wirte a C binding b) use alien c) luajit in 99% of all cases i'd prefer a). There are several ways to do this, but it seems to be a rather slim C API, so i'd do it without any extra binding libs (like luabind).

You may have a look on lua-cjson source http://www.kyne.com.au/~mark/software/lua-cjson-manual.html . That's a good example of a C binding.

Answer 3

I would skip luasocket and just using a C/C++ socket library if I were in your case.

Since the processing is going to be done in C/C++ anyway, using luasocket simplifies (a little) the socket handling at the expense of complicating (significantly) the interfacing between the received data and the C/C++ binary data treatment lib.

Using a C/C++ socket lib will make the socket handling a bit harder (but not much) while the data transmission to the treatment lib will be trivial.