Need an efficient binary networking example for server & clients.

[BorhilIan]

I'm looking for an example of a server and multiple-clients which can send/receive binary data very efficiently. Currently I understand socket for the most part but am not a fan of its string based data. I have already considered casting binary onto the string with each byte being a character but that will likely cause issues. I intend to create a wrapper with functions as listed below.

EDIT: I replaced my usage of byte with bit as that is actually what i'm looking for.

First Creating and sending a packet as such.

Code: Select all

net.Begin( "UpdatePlayer" ) - Converted to a numeric header for the packet and starts my packet assembler. (This method will allow custom net packets for modders to use)
	net.Write( 127 , 8 ) -- First is the value to send, Second is the number of bits. (Will be unsigned regardless no negative numbers needed)
net.Send() - Send the packet to the specified targets on server or to server on client and clears the packet assembler.
net.Broadcast() - Send the packet to all clients on the server except for specified. (would be called in place of 'net.Send()')

Second receive the packet.

Code: Select all

net.Receive( "UpdatePlayer" , function()
	local value = net.Read( 8 ) -- Read the first eight bits which would return the 127 value.
end )

Thirdly this basic networking system will enable me to do the following.

Code: Select all

net.WriteString = function( txt )
	net.Write( string.len( txt ) , 8 ) -- Send the length of the string up to 255 characters.
	for k , c in pairs( txt ) do
		net.write( string.byte( c ) , 8 ) -- Here would be a converter to binary from string.
	end
end
net.ReadString = function()
	local len = net.Read( 8 ) -- Read the length of the string from the first byte.
	local txt = {}
	for i = 1 , len do
		txt[ i ] = string.char( net.Read( 8 ) )
	end
	return table.concat( txt )
end

This is the sort of thing I am looking for though I currently have no working net code, just examples as above of how I will use it. If there are no good methods of binary networking in lua I will likely cast the data onto strings.

[bartbes]

strings are the natural way to represent binary data in lua. They are pretty much "a bag of bytes" everywhere that doesn't involve display or specific encodings.

[zorg]

As bartbes said above, strings are sent as arrays of chars (bytes), so there's no better way to represent binary data. That said, if you want to send more than one variable's contents all at once, or those not being strings already, then i'd suggest you serialize them with bitser, maybe even putting that through löve's own compression function.

Also, if you use enet instead of socket, you can have reliable UDP channels too, meaning you don't necessarily have to send the data length before the actual payload, and you don't have to worry about the message not reaching the destination... i think.

[BorhilIan]

Yeah I found a 'socket.lua' module this morning which depending on its packet overhead may be the solution. At first I mistook it for something related to LuaSocket which I did not care for. Thanks for the replies.

[zorg]

Yeah I found a 'socket.lua' module this morning which depending on its packet overhead may be the solution. At first I mistook it for something related to LuaSocket which I did not care for. Thanks for the replies.

I still don't understand why you'd use a separate lua implementation that will most likely have overhead, instead of just the löve inbuilt socket or enet libs.

Currently I understand socket for the most part but am not a fan of its string based data. I have already considered casting binary onto the string with each byte being a character but that will likely cause issues.

So you're not a fan of strings but you want to cast the data to a string anyway? It's unlikely that it'd cause any issues either.
What i'm ultimately interested in is in what form you think you could send binary data (that's not a string, i.e. a sequence of bytes)?
Also, your 'reference implementation' seems to send data through one byte at a time; that's insanely wasteful in terms of clogging up your and the recipient's tubes (each packet has a header; sending a bazilion small packets is worse than sending a few bigger ones)

[BorhilIan]

Sorry i'm not very good at explaining things. The issues with networking data in cast strings is each character in that string is 1 byte. So lets say I just wanted to send a value which ranges from 0 - 16 (4 bits) I would effectively cast that to 1 byte (8 bits) if using strings to network data. Which would not be ideal for my desired application which involves lots of varidable range values being transferred.

So if I can find a framework that lets me read/write numbers as a specified number of bits I can then build functions for value types I will send.

So lets say I have a Entity which has a variable change, what I would do is.

Code: Select all

local ent = 137 -- The identifier for the Entity which has had its variable changed.
local key = "points" -- This is the actual string representing the variable that changed
local keys = {
	doggos = 0, -- This is the id for the first variable.
	points = 15, -- This is the id for the variable that is being changed, and here I would write (4 bits) instead of the actual string. (saving on lots of data.)
}
entities = { 137 = {} } -- A dummy table for how the entities will be stored.
--	So now here is the example of writing this value change.
net.Write( ent , 8 ) -- (8 bits) to represent a maximum of 256 entities. (0 - 255)
net.Write( keys[ key ] , 4 ) -- only for BITS is needed to represent the number 15 which is the highest variable key in this Entity.
net.Write( value , 32 ) -- The actual value that will be sent which is (32 bits) or an integer.
--	Here is receiving the value change.
local ent = entities[ net.Read( 8 ) ] -- Get the index of the entity to change from the first 8 bits or byte.
local key = net.Read( 4 ) -- Get the variable key (as an index).
ent[ key ] = net.Read( 32 ) -- Read and set the variable of 'key' to 'value'.

Now if each of these networked numbers had to be strings then it would go from (44 bits) -> (48 bits) which isn't a big difference here. However do to the sheer volume of the data that will be sent in my project it will add up fast.

Now I will use the string based networking if I must, but if I can use a more bit oriented form of networking then that is what I'm looking for.

Regardless I'll re-read the documentation on those in-built libraries and see if I mistook something.

[BorhilIan]

It seems as though what I'm looking for currently does not exist in lua/love. So I will have to use the string casting method for the time being, hopefully it will exist in the future and I can implement it in my project. Thanks for the replies it's time for me to start coding my network code.

[zorg]

...

Okay, so most of my confusion came from not knowing whether you wanted to go below byte-alignment, which you seem to want to do.
Two things:

Both (TCP/)IP and UDP datagrams define their payload lengths in bytes; you can't send an amount that's not exactly a multiple of 8 bits.

On the other hand, locally constructing a string that can hold variables with non-byte-aligned bit lengths is perfectly doable; the reason why i suggested bitser above was precisely this, although it does use its own logic in terms of how it transforms data into a more compact binary form.

If you want to define your own, then you could use the FFI that löve includes (because of luaJIT), to define a C struct, since that can hold any type of data, perfectly packed/aligned, and also an union on the same struct, so that you could easily handle it both as many different variables and one string. (though you do need to make sure the total is a multiple of 8 bits; just put a few padding bits to the end of the struct)

If you don't want to mess with the FFI, you can kinda do the same things with pure lua as well, but that will probably be less efficient both in terms of memory and speed. (Unless JIT is off, which it is on handhelds) Can't say i have benchmarks ready on what would be the fastest, but even then, bitser should be fine enough.

[ivan]

Sure, you can pack 2x4 bits in a byte, but with networking - it probably won't make a big difference, except that your code will become more confusing and harder to maintain.

[Rucikir]

You could consider other libraries for various forms of serialization (there are a few on this forum) or other efficient binary serialization format like MessagePack or Protobuf.
I like TLVs.