Need an efficient binary networking example for server & clients.

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

Just need two functions for converting a number to/from binary. Preferably with the first bit being on the right -> side of the bit array. I have found a few different to binary functions but none to reverse the process. An example would be the number 3 becoming a binary 11 as a string or table of numbers. Ideally I will strip all but x (bits) from the outputted array starting from the left.

The idea is to take a series of numbers and convert them into a series of characters (string) based on assigned bits for each number into one coherent string.

number (bits) = binary = string/character
49 (6 bits) = 110001 = "1"
122 (8 bits) = 01111010 = "z"
122 (16 bits) = 0000000001111010 = "z"
1 (1 bits) = 1 = " :awesome:"
total (31 bits rounded up to 32 bits) = 'output string' = 0:110001|01111010|0000000001111010|1

One important distinction is I don't want to round the bits of individual items up to an 8 bit factor. Rather I want to round up the WHOLE combined array of bits to neatly fit into a string. As you can see in my above example I am trying to solve 'output string'.

Everything before ':' would be the inserted bits to round out the networked string, the '|' are just for this example.

EDIT: I guess I could just count up from the right for x bits to get the number from binary. However if you have any efficient ideas please let me know.

EDIT-2: Found what I was looking for, here is the basic code for anyone curious.

Code: Select all

function fromBits( s )
	return tonumber( s , 2 )
end
function toBits( n )
	local s = ""
	local b
	while ( n > 0 ) do
		b = math.fmod( n , 2 )
		s = b .. s
		n = ( n - b ) / 2
	end
	return s
end
EDIT-3: GOt everything figured out now, I simply covert each value to binary then append it all to a string and round it off to 8-bits. Then I iterate every 8 bits and convert that to a byte, then a char and append that to a string which is then networked. I will post the full conversion script in 'Libraries and Tools' forum when I finish it.

NOTE: It occurs to me I can cheapen the cost of conversion by storing the converted values for future lookup in the format of [ number] -> 'binary'. Could probably even take it a step further to maximize performance to ridiculous degrees at the cost of memory. (Most computers these days have sufficient memory to allow for such a trade-off)

EDIT-4: Here is the final tobits function that I will likely use.

Code: Select all

function tobits( n , bits )
	local n = math.abs( n )
	local s = ""
	local b
	while ( n > 0 ) do
		b = math.fmod( n , 2 )
		s = b .. s
		n = ( n - b ) / 2
	end
	if ( bits ) then
		local l = string.len( s )
		if ( l > bits ) then
			print( l , bits , l - bits )
			return string.sub( s , ( l + 1 ) - bits )
		elseif ( l < bits ) then
			return string.rep( '0' , bits - l ) .. s
		end
	end
	return s
end
User avatar
airstruck
Party member
Posts: 650
Joined: Thu Jun 04, 2015 7:11 pm
Location: Not being time thief.

Re: Need an efficient binary networking example for server & clients.

Post by airstruck »

Converting a number to a string representation of a binary representation of a number seems "questionable," to put it mildly.

Anyway, a less verbose solution is to format the number as octal and then replace the digits with binary triplets:

Code: Select all

local bits = { [0] = '000', '001', '010', '011', '100', '101', '110', '111' }
local function digits (d) return bits[tonumber(d)] end
local function tobits (n) return (('%o'):format(n):gsub('.', digits)) end
This might be useful for debugging or something, but if you're doing this for anything critical, something's off.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

The issue is networking in any lua library I've seen is string based (as opposed to binary). So the idea is to pack binary data into a string which will allow me to squeeze as much networking performance out of it as possible. The cpu/ram cost is negligible in my specific usage case.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

airstruck wrote: Mon Mar 27, 2017 4:49 am Converting a number to a string representation of a binary representation of a number seems "questionable," to put it mildly.

Anyway, a less verbose solution is to format the number as octal and then replace the digits with binary triplets:

Code: Select all

local bits = { [0] = '000', '001', '010', '011', '100', '101', '110', '111' }
local function digits (d) return bits[tonumber(d)] end
local function tobits (n) return (('%o'):format(n):gsub('.', digits)) end
This might be useful for debugging or something, but if you're doing this for anything critical, something's off.
As for your triplets suggestion it seems less efficient then just converting numbers to raw binary and appending them all to the same string. If I used the triplet method I would still be rounded to the nearest 8-bit as networking in lua is done with strings. Each character of which represent 8 bits or 1 byte.

In my usage case I will be appending all numbers that need to be networked in a single packet together without any padding bits. The only padding will be that inherited by the string based nature of networking.

I really do wish there was a pure binary networking library for lua however I have not found such a thing.
User avatar
zorg
Party member
Posts: 3436
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: Need an efficient binary networking example for server & clients.

Post by zorg »

BorhilIan wrote: Mon Mar 27, 2017 5:09 am The issue is networking in any lua library I've seen is string based (as opposed to binary). So the idea is to pack binary data into a string which will allow me to squeeze as much networking performance out of it as possible. The cpu/ram cost is negligible in my specific usage case.
His issue is still that he can't get a binary representation of the doubles lua (and löve) uses for numbers; strings are already binary, as it has been said quite a few times before, and solutions have been posted before, including serialization libs like bitser, that will even try to store some small integers in fewer bits (still byte aligned, because char array, otherwise known as string).

It has also been stated that both TCP and UDP datagrams and even IP packets have a granularity of 8 bits, so you can't realistically optimize further than that... nor should you even mess around with all this until you actually have something workable.

If your only issue is how to put numbers and bools into strings, and you find the most common approach too wasteful (or inaccurate, though floats may have issues between platforms... maybe) then there's always the FFI route; C union containing a char[8] and a double; you put the lua number in the double part, and you get its binary representation from the char array part. It doesn't get simpler/faster than that.

But hey, don't listen to me, clearly i'm mistaken and want to cause you suffering by not letting you overoptimize your nonexistent program. :awesome:
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

zorg I've been reading your posts and am still reading through your suggestions. However we seem to have very different ideas on this subject, as you said I should work with the most basic in-built library such as socket or enet. However I still need to format what I will network in a manageable way. In my specific case, in my specific project that is by going as bare bones as possible. I don't need to write a boolean to a string when I can write a single 'bit' to a packet as well as lots of other important information. I don't even need to write complex types in any regard, I only need binary. Zeroes and ones from which I can represent ANY data type that I need to.

EDIT: Also in regards to having something workable, I'm in no hurry to 'finish' this project. I am merely laying a foundation down and want it to be as robust as possible because in the long run it will be that important. (TL;DR This is a leisurely project and I can take the time to read and learn)
User avatar
airstruck
Party member
Posts: 650
Joined: Thu Jun 04, 2015 7:11 pm
Location: Not being time thief.

Re: Need an efficient binary networking example for server & clients.

Post by airstruck »

BorhilIan wrote:As for your triplets suggestion it seems less efficient then just converting numbers to raw binary and appending them all to the same string.
Sorry, but I honestly have no idea what you mean. What is "converting numbers to raw binary" supposed to entail? You posted a function that returns a string representation of a number, formatted as binary (a string of ones and zeros). I posted another function to do the same thing. Not sure what you're after, really. A string of ones and zeros isn't going to get you anywhere unless you're just looking to debug something. Shave some zeros off the front if you don't like them there, or add some more if you want padding.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

airstruck wrote: Mon Mar 27, 2017 5:51 am
BorhilIan wrote:As for your triplets suggestion it seems less efficient then just converting numbers to raw binary and appending them all to the same string.
Sorry, but I honestly have no idea what you mean. What is "converting numbers to raw binary" supposed to entail? You posted a function that returns a string representation of a number, formatted as binary (a string of ones and zeros). I posted another function to do the same thing. Not sure what you're after, really. A string of ones and zeros isn't going to get you anywhere unless you're just looking to debug something. Shave some zeros off the front if you don't like them there, or add some more if you want padding.
Ah I see where the confusion is, converting numbers to raw binary is really just a formality. What I am using it for is to pack raw binary numbers into a separate non binary string. Ill try to explain it as clearly as I can.

Code: Select all

local number = 2 -- 10 (2-bits)
local char = string.char( number ) -- '@' (in this example)
local packet = char .. char .. char -- '@@@' Here I lost 18 bits since the individual data I want to network is only 2 bits long. (The number of bits would be specified when writing values to my net packet)
--	So now what I do to optimize networking bit usage would be.
local bits = tobits( 2 ) -- "10" (2-bits as a string)
local data = bits .. bits .. bits -- '?' Now instead of sending three bytes (chars) I will instead send one single char, I also have room for another small value before needing to append another char. (networking is string oriented)
local packet = ""
--	Now I assemble the actual string that will be networked from the binary data. (I would add extra zeroes to the start of data to pad it out so this operation works as intended)
for i = 1 , string.len( data ) , 8 do
	packet = packet .. string.char( tonumber( string.sub( data , i , i + 8 ) , 2 ) ) -- Convert 8 bits at a time back into a number.
end
--	packet now equals x chars (bytes) from the binary data.
Now what you might be thinking is I just networking the individual numbers one at a time. The issue there is then I have even more overhead. Not only that but trying to pass the numbers into a 'send' is because it will get converted to a string. (since all lua-based networking libraries are string oriented)

Now I can take this a step further with compression but I will leave that as a configurable option for my projects server.

It's my understanding that bitser does a similar thing however it has complexities I simply have no use for. (currenty)
Last edited by BorhilIan on Mon Mar 27, 2017 8:23 am, edited 1 time in total.
User avatar
airstruck
Party member
Posts: 650
Joined: Thu Jun 04, 2015 7:11 pm
Location: Not being time thief.

Re: Need an efficient binary networking example for server & clients.

Post by airstruck »

There are several problems with that example. `string.char(2)` isn't "@", there's no character to represent it; "@" is 64. If bits is "10" then bits..bits..bits is "101010" of course, not sure where "?" came from. Passing a string to string.char throws an error.

Zorg's suggestion to use a union of structs seems perfectly reasonable, possibly with bit fields (colon followed by a number of bits at the end of each field). Or you can left-shift things and add them together and convert to characters. At no point should you need a string of ones and zeros. You might want to read back over the other comments in this thread more carefully, I feel like you've missed something.
BorhilIan
Prole
Posts: 38
Joined: Wed Mar 15, 2017 6:46 am

Re: Need an efficient binary networking example for server & clients.

Post by BorhilIan »

Yeah 'string.sub( data , i , i + 8 )' should have been 'tonumber( string.sub( data , i , i + 8 ) , 2 )'. It's quite late and I forgot it, my bad. Regardless this topic seems pointless now since I've figured out how to do what I wanted. If you don't see the point of it I'll leave explaining for a later date where I got a working example to show you.

EDIT: Yeah I don't understand what you mean by union of bits and it seems pointless to me. Since adding anything more then just what I have now would be unnecessary.

EDIT-2: Cant sleep, going to code the working example now. Will provide a '.love' and the raw 'main.lua' in my next post.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 54 guests