Bit16 - Draw Engine and Media Player

Showcase your libraries, tools and other projects that help your fellow love users.
MachineCode
Citizen
Posts: 70
Joined: Fri Jun 20, 2014 1:33 pm

Bit16 - Draw Engine and Media Player

Post by MachineCode »

I am about 80% through coding a pixel based draw engine in love2d. The idea is to make an interpreter that looks like a 16 bit CPU, but the instructions are aimed at drawing primitive graphics objects on a pixel map. Each instruction is a byte pair, so a "program" is just a string. This means that you can make a drawing or animation and encode it just as a string. The engine is just a sandbox, so it is safe to execute a string of commands from any 3rd party source. The worst that can happen is it draws junk on the screen.

This draw engine can do 24 bit colour, but it also can use compressed colour spaces like 12bit, 10bit, or 8bit mono tints. The idea is that images you draw are not photo realistic, but have a retro feel to them. After the video, I would like to include a sound generator with parametric controls and compressed audio including 8 bit audio for retro sound.

Anyway, this is the basic spec of the drawing virtual machine. I still have a few spare opcodes to fill, so maybe someone can think of useful extra ones. I might use the extra op codes to extend drawing to the canvas directly.

Code: Select all

Bit16 - Draw Engine and Media Player
====================================
The Bit16 engine digests a string of byte pairs and executes each one. All instructions are 
16 bit long and control a Virtual Machine with registers and instructions that impliment basic
drawing primitives in a 1024x1024 pixel map in CPU space. There is a 1024x1024 Canvas which 
matches the VM pixel map and is updated by draw instructions.
A string of byte pairs is presented to Bit16 in an update (~60fps) and the Bit16.draw is done 
in the draw callback (Love2d), which draws part of the canvas to the display.
Each string is a "draw fragment" and executes atomically. Most instructions will draw shapes, 
set pixels, or change the color of the draw operation. A draw fragment may also execute 
commands like "publish" that flushes changes up to VRAM and the final 1024x1024 image 
that the ui program uses. Since a draw fragment is locked to a frame, it is possible to
stagger the VRAM flushes over several frames and only publish every (say) 6th frame 
(for 10fps video).
A "clip" is a data structure (table or json) that has an array of draw fragments plus 
metadata describing how to draw them, including loops and pans, zooms. The array of draw 
fragments is 1 .. n.  
The Bit16 engine also has a "snapshot" command that copies the final 1024x1024 canvas to a 
second canvas. This image persists for viewing separate to the normal output, or it could 
be used for generating dynamic textures or tile maps.


16 bit Drawing Op-Codes
-----------------------

|| 4 bit op code || 2 bits op/data ||    10 bits    ||

In general, the 16 bit instruction is divided up into a 4bit op code and 12bit data.
For many instructions that only need 10 bits data, the opcode is divided into 4
Also, op code 0000 subdivides into smaller instructions with less or no data
To simplify decode, we have the top 4 bits as an instruction group, then the 
following bits decoded as required - 12 bit data, 2 op bits + 10 bit data, or 
4 op bits + 8 bit data. The 0 opcode is divided into 16 groups. 1-15 are 8 bit 
parameters. Group 0 (0x000X) is a block of up to 256 commands that have no data.
the opcode 0x0000 is the NOP - it always executes nothing. A NOP instruction (2 bytes) 
can be presented to the engine and it will just consume a draw tick.

X,Y position can be set as 2 10b values. In addition there is a second XX,YY pair 
for destinations of block copy, line draw etc

Colour can be 24 bit, 12 bit, 10 bit or 8 bit mono/tint. 24 bit requires three instructions
to set RGB independently.

Once a colour is set, runs can be done with a draw/skip command that draw 1-32 pixels
then skips 0-31, or block, rect, circle, ellipse, line.


ImageData and canvas
--------------------
Note:-   the pixel map workspace is 1024x1024 matching 10bit data co-ordinates
         the published area is set by commands in the data stream, but may be drawn 
         at any size in the user application. The user view port is
         set by the user program and is not controlled by the instruction stream.

Publish and refresh are now a lot more complex.  Partial updates of VRAM images from 
a rect of the bitmap are not supported in love from 0.11, and not at all in other engines
like godot or unity. The safe way to do this is to divide the 1024x1024 bitmap into 8x8 tiles
of 128x128 pixels and selectively refresh these. There is no longer any point in having
a 1024x1024 Image, so instead we have a 1024x1024 canvas and let the gpu blit from the
image tiles to the canvas. 

So - we get the rect of tiles that includes the publish/refresh area, paste from the 
main bitmap into the tile bitmaps, refresh the selected tile images, then blit the 
tile images to the destination canvas. It is assumed that this is done as separate
operations as the GPU commands will be sent in batches which may be more efficient than 
interleaving a replacePixels() with a draw for each tile. 

The engine may be extended to include some draw commands that operate on the canvas in VRAM.
Useful would be rect fills and clears to save overhead of useless RAM->VRAM copies.

ViewPort
--------
Commands that change the ViewPort don't need any VRAM update. For example, to simulate video
noise (snow) you randomise a section of the draw bitmap, then every frame just issues a
command to change the ViewPort position. That will look like a new random noise every frame,
but actually use zero RAM->VRAM bandwidth. Same with pan and scroll effects. Load the bitmap
with an image (which might take hundreds of frames), then scroll, pan, and zoom around the
image with no draw overhead. Zooming is for free as well, since the draw fragments set the
published ViewPort size (nominally 360 x 240) and the external user program sets the display
ViewPort independently.  The UI might be a window 720x480 and the published VP might be
128x85 which will be automatically scale up in the draw command. That gives a lot of
opportunity for playing with image resolution. In 1024x1024 you can fit 96 128x85 images
which gives a 10sec clip if played at 10fps. Once loaded, playing sequences of these frames
only requires a ViewPort change per frame - which is 3 16bit instructions (6 bytes).

Image Compression
-----------------
This Draw Engine knows nothing about .png or .jpeg. Any image has to be composed 
from primitive draw commands. For simple logos and shapes it should do a lot better than
png  and jpeg.  For random images it will depend on the image and the amount of compute 
time encoding.  

The basic method for a general image is first to reduce the color space 
to 12 or 10 bits - in games you often don't want photo realistic.  Reducing the color 
space means more blocks and runs of the same color.  Next, you divide your image into 
blocks and look for the most common color in that block. Issue a block fill command to 
set each of the blocks (2 bytes per block).  As you generate primitive drawing commands, 
you keep a bitmap of wrong pixel colors. Then, on a block by block basis, you search for 
bad pixels that are adjacent and matching and fill them with run instructions which are
quite efficient. Finally, to clear the remaining bad pixels in each block, you find sets 
of same color pixels in the block and then set them with a mov_relative_xy_setpixel 
instruction. This only uses 1 instruction (2 bytes) per pixel. Assuming that typically 
about half the pixels can be covered by efficient block fills and runs, then that would 
put the encoded image size at about 70k-80k for a 360x240 general image, which is similar 
to a low loss jpeg and about 50% of a png. For lossy complex images, jpeg is better, 
however for any cartoon style images for games, this encoding systems is very good.

Effects
--------
Random noise patterns are built in. You can specify a rectangle, block fill it with 
a background color, then set a random threshold and change pixels above that threshold
to a random color, bound by color ranges. Or, you can set a block size and make a mosaic
of random color tiles in the rectangle - all in one instruction (2 bytes). You can build 
small images somewhere on your 1024x1024 bitmap and then block copy them to build an image
in your main ViewPort. This draw engine is like a cut'n'paste worksheet of bits of images.



OP CODE definition
------------------

[0] 0000.0000.[0]            -- nop, special case  (256 no param commands)
             .[1]            -- reset - clear the bitmap and the canvas
             .[2]            -- reset tick - set back to zero
             .[3]            -- tick - put this at the end every draw fragment
             .[4]            -- reset frame
             .[5]            -- frame - increment with a publish or VP change
             .[6]            -- 
             
             .[10]          -- change viewport only (image is assumed correct)
             .[11]          -- refresh (force update of image from VP2)
             .[12]          -- publish (force update of image from publish area and vp) 
             .[13]          -- snapshot - grab the canvas and the VP2 window as well
           
             .[50]          -- noise mono rectangle on XY rectangle (speckle)
             .[51]          -- noise color rectangle on XY rectangle 
             .[52]          -- noise mono block draw a rnd blk color or not 
             .[53]          -- noise color block draw a rnd blk color or not  
             .[54]          -- 
             .[55]          -- rect copy with mono noise 
             .[56]          -- rect copy with color noise  
             .[57]          -- rect copy stencil black
             .[58]          -- rect copy color squish (noise ranges)
             .[59]          -- rect copy mono squish (noise range)
             .[60]          -- rect mono (convert to mono)
             .[61]          -- rect mono negative (convert to mono and invert)
             .[62]          -- rect color invert (color negative)
             .[63]          -- rect color squish (use noise range params)
             .[64]          -- rect XY, XXYY fill with random blocks mono
             .[65]          -- rect XY, XXYY fill with random blocks color
          
             .[100]          -- set dwg mode pixel (default)
             .[101]          -- set dwg mode block
             .[102]          -- set dwg mode rectangle
             .[103]          -- set dwg mode circle
             .[104]          -- set dwg mode ellipse 
             .[105]          -- set dwg mode line 
           
             .[200]          -- rect copy (straight copy src->dest)
          
    [1] .0001.[8b]          -- undefined
    [2] .0010.[8b]          -- 
    [3] .0011.[8b]          -- 
    [4] .0100.[8b]          -- 
    [5] .0101.[8b]          -- 
    [6] .0001.[8b]          -- 
    [7] .0010.[8b]          -- 
    [8] .0011.[8b]          -- 
    [9] .0100.[8b]          -- 
    [a] .0101.[8b]          -- 
    [b] .0001.[8b]          -- 
    [c] .0010.[8b]          -- 
    [d] .0011.[8b]          -- 
    [e] .0100.[8b]          -- 
    [f] .0101.[8b]          -- 

[1] 0001.xx.[10b value]      -- publish parameters commands
          0.X_pos              -- X TL corner of published area
          1.Y_pos              -- Y TL corner of published area
          2.X_size             -- width of published area
          3.Y_size             -- height of published area

[2] 0010.xx.[10b value]        -- VP2 parameters commands
          0.X_pos              -- X TL corner of 2nd viewport
          1.Y_pos              -- Y TL corner of 2nd viewport
          2.X_size             -- width of VP2 area
          3.Y_size             -- height of VP2 area

[3] 0011.xx.[10b value]      -- position commands
          0.X_pos              -- X move absolute
          1.Y_pos              -- Y move absolute
          2.XX_pos             -- secondary X (destination)
          3.YX_pos             -- secondary Y (destination)

[4] 0100.xx.[10b value]     -- block values
          0.X_size            -- X size of rect (also ellipse)
          1.Y_size            -- Y size of rect
          2.B_size            -- side of block to draw ind of rect
          3.C_size            -- dia of circle to draw ind as well

[5] 0101.xx.[10b value]      -- relative instructions
          0.XY_mov             -- move +/-16 X   +/-16 Y
          1.XY_movblk          -- move +/-16 X   +/-16 Y in blocks
          2.XY_pix             -- mov,set pixel at rel position
          3.XY_blk             -- mov,set block at rel position in blocks


[6] 0110.xx.[10b value]      -- runs of pixels
          0.X_run              -- draw hor run of pixels
          1.Y_run              -- draw vert run of pixels
          2.X_runskp           -- Hor fill 1-32 then skip 0-31
          3.Y_runskp           -- Vert fill 1-32 then skip 0-31
          

[7] 0111.xx.[10b value]      -- draw immediate with 10 bit color
          0.pix                -- set pixel 10b color and mov right
          1.blk                -- draw block 10b color and move 1px right
          2.rect               -- draw rect 10b color and mov 1 px right
          3.line               -- draw a line diag in immediate rect

[8] 1000.xx..[10b value]       -- draw immediate with 10 bit color
          0.circ               -- draw circle 10b color and mov right        
          1.ellipse            -- draw ellipse 10b color and mov right  
          2.circ band          -- circle band 
          3.ellipse band       -- ellipse band
          
[9] 1001.xx.[10b value]      -- draw immediate object with 10 bit parameter
          0.blk                -- draw a sq block - side 10b
          1.circ               -- draw a circle - dia 10b
          2.rect               -- draw a rect 1-32,1-32
          3.

[a] 1010.     -- not defined

[b] 1011.xxxx.[8b value]      -- noise and color parameters
        .0.[8b value]            -- set R lower limit  
        .1.[8b value]            -- set R upper limit
        .2.[8b value]            -- set G lower limit
        .3.[8b value]            -- set G upper limit
        .4.[8b value]            -- set B lower limit
        .5.[8b value]            -- set B upper limit
        .6.[8b value]            -- set Y lower limit (mono)
        .7.[8b value]            -- set Y upper limit
        .8.[8b value]            -- set prob of noise 0-255
        .9.[8b value]            -- set mono (tint) color
        .10.[8b value]           -- set R component (set for 24 bit color)
        .11.[8b value]           -- set G component
        .12.[8b value]           -- set B component
        .13.[8b value]           -- set tint R
        .14.[8b value]           -- set tint G
        .15.[8b value]           -- set tint B

[c] 1100.xxxx.[8b value]      -- more 8 bit parameters and mono draw commands

        .0.[8b value]            -- set circle band x/255
        .1.[8b value]           -- set rand X min (4block boundary)
        .2.[8b value]           -- set rand X max 
        .3.[8b value]           -- set rand Y min
        .4.[8b value]           -- set rand Y max
        .5.[8b value]           -- set rand X grain
        .6.[8b value]           -- set rand Y grain
        
        .8.[8b value]           -- draw pixel mono
        .9.[8b value]           -- draw block mono
        .10.[8b value]          -- draw circle mono
        .11.[8b value]          -- draw rect mono
        .12.[8b value]          -- draw ellipse mono
        .13.[8b value]          -- draw circle band mono
        .14.[8b value]          -- draw ellipse band mono
        .15.[8b value]          -- draw line mono


[d] 1101.                   -- not defined

[e] 1110.[12b value]        -- set 12bit colour

[f] 1111.[12b value]        -- draw object 12b colour

grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Bit16 - Draw Engine and Media Player

Post by grump »

Why? What is it good for?
User avatar
zorg
Party member
Posts: 3436
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: Bit16 - Draw Engine and Media Player

Post by zorg »

"Why not" i would guess. :P
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Bit16 - Draw Engine and Media Player

Post by grump »

zorg wrote: Sat Jan 27, 2018 4:48 pm "Why not" i would guess. :P
That's a perfectly fine reason, I just thought I might miss some obvious use case for this.
MachineCode
Citizen
Posts: 70
Joined: Fri Jun 20, 2014 1:33 pm

Re: Bit16 - Draw Engine and Media Player

Post by MachineCode »

This is a media player that lets you store/generate images and animations as compact strings. It is like a custom codec that compresses images into drawing commands. Since it accepts strings, you could for example, have a window in your game that is controlled by a program running on a remote server with a websocket connection. As for the audio part, a parametric sound generator programmed by string sequences is something I would have liked, except I can't find one.
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Bit16 - Draw Engine and Media Player

Post by grump »

MachineCode wrote: Sat Jan 27, 2018 8:16 pm It is like a custom codec that compresses images into drawing commands.
So you developed an encoder that decomposes images (or even animations?) into an efficient series of primitives, and this is the decoder specification? I'm stoked to see the encoder.

I'm still not entirely sure I understand what I'm looking at.
MachineCode
Citizen
Posts: 70
Joined: Fri Jun 20, 2014 1:33 pm

Re: Bit16 - Draw Engine and Media Player

Post by MachineCode »

The encoder can be as simple as a brute force pixel by pixel set, however even a simple encoding scheme can do much better -

The basic method for a general image is first to reduce the color space
to 12 or 10 bits - in games you often don't want photo realistic. Reducing the color
space means more blocks and runs of the same color. Next, you divide your image into
blocks and look for the most common color in that block. Issue a block fill command to
set each of the blocks (2 bytes per block). As you generate primitive drawing commands,
you keep a bitmap of wrong pixel colors. Then, on a block by block basis, you search for
bad pixels that are adjacent and matching and fill them with run instructions which are
quite efficient. Finally, to clear the remaining bad pixels in each block, you find sets
of same color pixels in the block and then set them with a mov_relative_xy_setpixel
instruction. This only uses 1 instruction (2 bytes) per pixel. Assuming that typically
about half the pixels can be covered by efficient block fills and runs, then that would
put the encoded image size at about 70k-80k for a 360x240 general image, which is similar
to a low loss jpeg and about 50% of a png. For lossy complex images, jpeg is better,
however for any cartoon style images for games, this encoding systems is very good.

Drawing basic images like lines, rectangles and circles is like writing assembler code. The reason for describing the draw primitives as 16bit words is that it is very compact. If you are sending images and draw commands over a network, you want compact data.

My main application for this is a simulation of a multi channel TV that can deliver all sorts of content. Images, text, audio. The best place for this content is on a server, so you need a way to encode that content and deliver it over a network socket. That really means a draw engine of some sort. This codec is very different to most codecs in that it works in code space. The encoder can be simple, or it could be a very complex optimising compiler. Because it is so different it means that it is much harder for random real world images to pollute the content space.

Conventional codecs are not really suitable for games. They try to make perfect images, where often what you want is a stylised effect. Codecs don't let you do remote drawing, they don't let you control the frame rate or finish an animation with a repeat of a command string. They don't let you do block colour inversions, or degrade sections of the image with random noise. If you think about this engine as a wrapper around the love graphics module, it is a bit like an interpreter above the lua language. You could write a game that draws by sending this engine strings of draw operations, rather than calling the graphics module. Why? because it lets you split the client and server across a network.
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Bit16 - Draw Engine and Media Player

Post by grump »

MachineCode wrote: Sun Jan 28, 2018 12:43 amThis codec is very different to most codecs in that it works in code space.
That's hilarious. You're one funny dude :awesome:
MachineCode
Citizen
Posts: 70
Joined: Fri Jun 20, 2014 1:33 pm

Re: Bit16 - Draw Engine and Media Player

Post by MachineCode »

Yes, yes, all codecs are written in code, however they mostly they work in image space. They use properties of the image to compress the data and reconstruct it at the other end. I know the latest ones are very sophisticated, but techniques like DCTs are signal processing methods to reduce redundant information in 2/3 dimensional colour spaces. Most codecs rely on very sophisticated signal processing techniques - with lots of heuristic programming techniques thrown in to exploit the inter frame redundancy.

When I say "code space" I mean the image is compressed by compiling it into an arbitrary computer program. Would you be happier if I called it "Turing space"? For movie codecs, this technique of compiling a drawing program to generate the images at the other end would be impractical, because it doesn't necessarily have a solution. For a simple drawing engine, it is practical, particularly if the data source is a program as opposed to a movie. A programmed source will know exactly what the engine can do and exploit that.

I can guarantee you that the engine I have specified here can draw things in a handful of instructions that no conventional codec can match. For example, I can draw a 1024x1024 pixel map of completely random 24bit colour pixels in a single frame with a 2 byte instruction. That is the worst case for conventional codecs. Another example would be a mosaic of coloured blocks that needs perhaps 100 bytes. No conventional codec can compress that to a similar degree, because it doesn't know the drawing rules of this draw engine. Codecs are general purpose and optimised for video content. They work in image space.

In the extreme case the best codec would be one that analyzed a movie, constructed 3d models of the actors and objects in the movie, downloaded models and textures, and then let the GPU construct the movie in real time. That would be a codec that operated in "Turing space". It is probably not possible at the moment, but if you were in the business of making movies (particularly VR movies) it would probably occur to you that shipping a 3d model of the movie might be a much better idea that filming it in 2D and using codecs to reconstruct it. The new VR headsets have good GPUs built in, so that is the place to do the transforms to the images.

@Grump I am sure you are aware of this https://en.wikipedia.org/wiki/Jan_Sloot, since I get the impression you are the type of guy who knows everything. There were other attempts at encoding techniques. The other thing here is that if you look at any youtubes about advanced image compression, the first thing they say is it is a patent minefield. Most of their time is dodging patents. Draw engines are a simple concept and I doubt they can be affected by patents.
User avatar
pgimeno
Party member
Posts: 3544
Joined: Sun Oct 18, 2015 2:58 pm

Re: Bit16 - Draw Engine and Media Player

Post by pgimeno »

This reminds me so much of WMF.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 43 guests