Sending Sounds over the network

[EliterScripts]

Hello,
I am interested in having players be able to talk to each other online with their microphones. I would need to be able to get the microphone input into a string, push the string over the network, then have the guy at the other turn the string into SoundData/Source then play it on his end.

However, I can't seem to figure out how to do it. Could anyone give me some sample code? I have tried love.data.encode, and love.data.decode and all that, but it never seems to work getting *back* to sounds that I can play. I have no intention of saving any of this into a file, since it will all be over network and maintained on the RAM.

[zorg]

Most of what you want is not too hard, namely:

You can use a RecordingDevice to record sound, which gets returned in a SoundData object, which you can convert (the meaningful data inside it) into a string with a method it has, send that string over to the other client(s), create a similar SoundData object on their end (or better yet, just keep re-using one), copy the string into that SoundData by getting its pointer, and using some ffi magic (or the slower way, setting each samplepoint with the provided setSample method, in a loop), and passing that SoundData to a QueueableSource, which you can feed data to realtime.

All that said, the biggest issue for sound transmission online is that sound information has rates that make it consume gigantic amounts of data to be sent over the net constantly, if you don't encode it into something smaller beforehand; the above example would just send "raw" samplepoints; for mono, 44.1kHz sampling rate and 8bit bit depth, that's 44100 Bytes/second, or 88200 B/s if you want 16bit.

Still, it's possible to do, and löve gives you almost everything to accomplish this, except the encoding of sound data into a more compressed form... then again, a-law and mu-law algorithm implementations exist on the net that one could reference and implement; those are the most generally used codecs in the telephony world.

Still, to give you an example for your specific question,

Code: Select all

-- assuming you have your received data completely in the variable "data"
-- assuming the transmitted sound's properities are mono, 8bit and 8000 Hz sampling rate.
-- also assuming that data holds unsigned values, i.e. you'll need to "transpose" the numbers down by 256/2.
local length = #data
local SD = love.sound.newSoundData(length, 8000, 8, 1)
for i=1, length do
	local smp = data:sub(i,i):byte() -- turns one character from the string into a number
	smp = smp - 128 -- [0,255] -> [-128,127]
	smp = smp / 128 -- [-128,127] -> [-1,~1]
	smp = math.min(math.max(smp, -1.0), 1.0) -- probably unnecessary
	SD:setSample(i-1, smp)
end
QS:queue(SD); QS:play()



-- Do this once
local QS = love.audio.newQueueableSource(8000, 8, 1)

Something like that could work.

[grump]

SoundData is missing a sane method to fill the buffer IMHO. The setSample loop seems verbose and inefficient.

[zorg]

If you're generating the data, it's fine enough; luajit should treat the loop efficiently enough; the issue is when you would want to copy a whole memory region into it, let's say, where it becomes too slow, especially if you'd want to do that many times. (as i said, it's a Data object, so it has a getpointer method, meaning you can set it via ffi, but still)

[EliterScripts]

@zorg, I really don't understand what you were doing there, but I tried using this code:

Code: Select all

function love.load()
	devices = love.audio.getRecordingDevices( )
	for k,v in pairs(devices)do
		print("device: " .. v:getName() )
		print("is recording: " .. tostring( v:isRecording()) )
	end
	devices[1]:start()
end

function love.update(dt)
	local SoundData = devices[1]:getData()
	if(SoundData ~= nil)then
		local Source = love.audio.newSource(SoundData)
		local DataStringed = love.data.encode("string", "base64", SoundData:clone() )
		print(DataStringed)
		local DataUnstringedd = love.data.decode("data", "base64", DataStringed)
		local DataReturned = love.audio.newSource( DataUnstringedd )
		love.audio.play( DataReturned )
	end
end

and I get this error:
"Error: main.lua:17: bad argument #2 to 'newSource' (string expected, got no value)"

I'm really having issues getting it from a SoundData object to a string (which I think I was able to do, not sure though), and from a string back to SoundData, then to a Source, then to being able to play it.

Once I can get the whole string conversion figured out, then I'm gonna move onto compressing (either lossy or lossless) data between peers.

[zorg]

My example code was for the clients receiving the data as a string, and converting that into values a SoundData can store, and queueing that up to a QueuableSource, which is different than a regular Source.

The error in your code example is that you want SoundData:toString() and not SoundData:clone(), since the latter returns a löve object (another SoundData).

Another thing is that it's completely pointless to encode it into base64, since it'll be even larger than if you just sent it as a "binary string", meaning it could contain all 0-255 values.

Here's an edited version of your last code block:

Code: Select all

function love.load()
	devices = love.audio.getRecordingDevices( )
	for k,v in pairs(devices)do
		print("device: " .. v:getName() )
		print("is recording: " .. tostring( v:isRecording()) )
	end
	devices[1]:start()

	qsource = love.audio.newQueueableSource(
		devices[1]:getSampleRate(),
		devices[1]:getBitDepth(),
		devices[1]:getChannelCount()
	)
end

function love.update(dt)
	local sounddata = devices[1]:getData()
	if(sounddata ~= nil)then
		local s = sounddata:getString()
		print(s) -- yes it might be garbled, you should not care about that, perfectly reasonable that a binary string is not printable easily. Also, printing data this fast will lag your project; DON'T DO THIS.
		-- Also, as was said before, there's no real easy way to turn a string back into a SoundData; you either do what i suggested above, or use the FFI to copy the whole string at once into the area you can get by the getPointer method of the SoundData.

		local anothersd = love.sound.newSoundData(
			#s,
			devices[1]:getSampleRate(),
			devices[1]:getBitDepth(),
			devices[1]:getChannelCount()
		)
		
		-- Further complexities ahead, since you need to make it work differently whether it's 8bit or 16bit, and 1 or 2 channels(interleaved in the latter case). Then again, with the FFI, it'd be easier in the sense that what you got from :getString is what you'd need to copy into the memory area anyway, without needing to convert it.
		local i = 1
		while i <= #s do
			local smp = 0
			if devices[1]:getChannelCount() > 1 then error("Implement this yourself.") end
			if devices[1]:getBitDepth() == 8 then
				smp = s:sub(i,i):byte() -- turns one character from the string into a number
				-- Since i don't recall how the samplepoints are stored internally, this might not work 	without some modifications.
				smp = smp - 128 -- [0,255] -> [-128,127]
				smp = smp / 128 -- [-128,127] -> [-1,~1]
				smp = math.min(math.max(smp, -1.0), 1.0) -- probably unnecessary
				anothersd:setSample(i-1, smp)
				i=i+1
			else
				smp = s:sub(i,i):byte() * 256 + s:sub(i+1,i+1):byte()
				smp = smp - (65536/2)
				smp = smp / (65536/2)
				smp = math.min(math.max(smp, -1.0), 1.0) -- probably unnecessary
				anothersd:setSample(i/2-1, smp)
				i=i+2
			end
			
		end
		qsource:queue(anothersd)
		qsource:play()
	end
end

[grump]

Code: Select all

local sptr, ssize = soundData:getPointer(), soundData:getSize()
require('ffi').copy(sptr, streamDataString, math.min(#streamDataString, ssize)

Something like this should work too. Consider this to be pseudo code, some details might be wrong. Make sure there is enough data to fill the entire SoundData. Maybe fill it with zeroes(?), see ffi.fill.