Author Topic: Call me crazy!  (Read 4958 times)

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Call me crazy!
« on: April 18, 2017, 04:30:37 PM »
This may be crazy, but I'm remaking Terrablox again!
OK, put down the pitchforks and torches. I have lots of reasons.
So read through and let me know your thoughts. :)

Current Version:
I will likely be coming back to the current version to get it into a game/playable state. But it's going to be difficult to ever evolve it too much. And the time I can spend on it is very limited as it doesn't, and probably won't, generate money. So I have to concentrate on things that do. It also won't port to GMS2, the shaders are over-complex, which is my mistake, yes they are cool and fast but then YYG took out DX9 support and screwed them all over, and their support to DX11 was broken for so long and there's still issues. And even if it did port, the new IDE does not lend itself well to it's current structure.

The Hate:
On top of those issues, I'm finding GM just toxic these days. I no longer have belief in it. The experience of using GM is a bumpy road, from something always being broken, to fixes breaking other things, to rubbish/incorrect documentation, to the only decent reference material (the old GMC) vanishing from the internet (can't google it). I just got fed up.

The Solution:
Anyway, enough of the rant. I have a new plan. So I entered a game jam recently, and for it I decided to use pure Javascript, using HTML5 canvas for rendering. Now I was already familiar with JS, but I hadn't actually made and released a full game with it before. So I did and it went really well and was enjoyable. So for the sort of games I usually make, 2D pixel art stuff, I'm more comfortable with that now. But what about my 3d stuff? I only really have Terrablox. So, can I make that with pure Javascript?

The Attempt:
OK, so I have already set up a project and have things actually working. I'm using the Three.js library for all the 3d stuff at the moment, it's quick to set up and do stuff with and will demonstrate viability sooner. It manages things like the renderer, scene, camera, geometry, etc. Voxels are now objects, so they can have all the properties they need, rather than just being a single value. It has multi-threading using web workers, currently I have this building the mesh. Web workers are tricky as you can only transfer object data as strings, so string>object conversion still happens in the main thread.

Despite Javascript not having any "include" command by default, it does lend itself well to being modular. So this project will also be open source. That way I finally keep that promise, and can easily keep any game (art, audio, game logic, etc) separate from the engine. Not being able to easily separate engine and game before was the reason it didn't happen.

I have not much to show atm, currently there is a chunk that can fill up with voxels, and then build a mesh in a separate thread, and then return it, and finally add it to the scene. My next goal is to have the world object load in a bitmap (heightmap), create chunks and generate voxels based on the world heightmap data. There's also currently no culling or any optimizations. I'm still in the viability stage. So don't get too exicted. You'll see it when I get it looking like something. :)

So to recap, I know these are things that have been requested before:
  • Open Source
  • Multi-threading

EDIT:
Spoiler (hover to show)
« Last Edit: April 18, 2017, 09:21:49 PM by JackOatley »

Matchesia

  • Full Member
  • ***
  • Posts: 219
  • A Person
    • View Profile
Re: Call me crazy!
« Reply #1 on: April 19, 2017, 12:41:50 AM »
Your crazy, but I really do like the idea of this though i'm not entirely sure how far you can go with JavaScript.

It's dead, but not as dead as the chat.
https://discord.gg/BA9FpQJ

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #2 on: April 19, 2017, 02:33:41 PM »
Your crazy, but I really do like the idea of this though i'm not entirely sure how far you can go with JavaScript.
It's going to take a bit of effort to get a 3d application running smoothly, but I will get it there.

Three.js is decent for rendering, but I've already been working around some of it's built in functions for dealing with geometry. I got it to send just the vertex data between the main thread and the worker thread, so no more string converting. But that wasn't completely the issue anyway, the issue was trying to run too many workers at once, they all ate into each other which had a knock-on effect in the main thread when all the callbacks came in at once. But I build the vertex buffers manually now which is both faster and more memory efficient and takes <30ms for basic chunks (remember I'm not culling atm, so this could be better). So here's a pic of the obvious improvement since yesterday. :)



EDIT:
Those with slower connections are going to hate me for all these gifs. :) I got normals data going as well now. All data is currently derived from a single cube model so this doesn't need to be calculated or anything, it just needs to be compiled into an array a handful-tens-of-thousand times (like I said, no optimizations yet!).



And here's a wire-frame view to show how unoptimized it currently is. I think it's doing things rather quickly considering the amount of unnecessary vertices in there.



EDIT:
Today I took three.js out of the web worker file. Web workers can load in other js files to access their functions, to do this however they don't get it from cache so have to recompile that file. This was to the tune of 300ms. All I was using it for in the end was the cube model, so I now construct this manually and lose that 300ms waste.

This is the box model:
Spoiler (hover to show)

This tiny model can be used to represent all voxel configurations. The idea being that you can selectively pick the faces to use per voxel when building a mesh.

There's still some lag/delay if I set it to run all the workers at once, but this may be because there's supposedly (I read) a 30ms-ish delay when setting up a web worker. And seeing as each chunk creates it own worker, and there's 8x8 chunks this comes to nearly 2 seconds, which seems to be the startup delay I get. I am planning on adding a proper manager/queue though, so there's just one worker to handle all chunks.

Anyway, enough tech stuff, here's a gif showing the normals working with three.js lighting with a shading material on the mesh. Forgive the jerkyness, that only happens when I'm capturing the gif. >.<



EDIT:
I just wanted to get this working to make sure the current mesh setup allows it. Shadow maps aren't very well documented or they don't work quite as they should. I only found out the bit I was missing from a bug report from 2012. All I was trying to do was move the shadow map camera, or where it looked at at least, turns out it has to look at another object and that object has to be put in the scene, confusing. But I got the result I wanted eventually:



Definitely need to get some height-map terrain in here!

EDIT:
OK, terrain is in:



I also got memory usage down A LOT by directly writing into the vertex/normal buffers rather than filling up an array and then converting it to a buffer. Using arrays, even temporarily, too much seems to send memory up quickly. It does go down after a while when the garbage collection catches up, but on the amount of data that gets shifted in this kind of game it's not really workable to use arrays in some places.
« Last Edit: April 21, 2017, 12:43:31 PM by JackOatley »

Matchesia

  • Full Member
  • ***
  • Posts: 219
  • A Person
    • View Profile
Re: Call me crazy!
« Reply #3 on: April 21, 2017, 03:43:57 PM »
Impressive.

It's dead, but not as dead as the chat.
https://discord.gg/BA9FpQJ

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #4 on: April 21, 2017, 11:52:24 PM »
I've changed the height-map system into a class. This means you can load a height-map and do loads of things to it before passing it as a single object to the world. So create a whole world from an image currently looks like this:
Code: [Select]
var world = new World();
world.generateFromHeightmap(
    new Heightmap( "img/hm2.png" )
);

I'm working on the camera and movement, so I can ditch that orbital camera and actually navigate around the world. This will help when I come to implementing the face culling so I can see it actually working. I can do culling in another thread so as not to distrupt the game, I reckon when this is all done it'll be a very fast little engine! :p

Here's a cool render:


I'm not sure yet how that phong shading will work with different textures/surfaces, seeing as a whole chunk shares a material. Though I think you can add secular maps. I'm not too familiar with all these lighting/shading things, so we'll see!

Terrabloxian

  • Hero Member
  • *****
  • Posts: 633
  • Drop it like it's hot
    • View Profile
Re: Call me crazy!
« Reply #5 on: April 22, 2017, 03:36:30 PM »
This is freaking crazy! I didn't know you were so adept at JavaScript! It makes me want to learn it.
Say nope to dope!

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #6 on: April 22, 2017, 07:10:22 PM »
This is freaking crazy! I didn't know you were so adept at JavaScript! It makes me want to learn it.
You should, JS is awesome and a very important language to know. :)

I started learning it ages ago when I was first messing around with websites. And I've made lots of little game engines for my own amusement since HTML5 came about. But it was only recently I decided to finish something with it. And so this is just a culmination of steps.

Anyway, most of the visual stufs right now is handled by Three.js, like the rendering, shading and shadows. But the voxel engine and the system around it, and now even the meshing, is all my code.

I got textures working now, well "A" texture. Though it should be simple enough to use multiple textures like before, just offset the uv by whatever the voxel needs. There's also mip-mapping here (again, three.js just has that as part of the material object). Also, got Input and Camera classes set up. So the world is navigate-able now:



EDIT:
OK, so multiple textures are working now. Although WebGL orders it's UV from the bottom-left as opposed to GMs upper-left, so that was confusing for a bit. But really all it means is that I have to index the tiles a different way round. For this image I also added in a sky-colored background and some fog, just makes the whole thing come together nicely:



EDIT:
I got delta time sorted out today. I wasn't going to bother for a while but it seems a weird thing happens due to v-sync. Chrome can switch from 60fps to 30fps arbitrarily. From what I've read this is a v-sync issue where Chrome would rather sacrifice frame count for frame quality. It's not a big issue but it was annoying moving around with the camera and full or half speed intermittently.

I also started working on culling. The actual culling doesn't seem to take any time at all, JS (Chrome at least) seems to handle arrays uber fast and that's all the culling requires atm, array lookups. I haven't actually timed it yet but it's not making a dent in FPS. However, with voxel faces culled and a new "visible voxels" object being passed to the meshing worker, the meshing itself is around 15-40 TIMES faster, now clocking in at about 0ms - 4ms a chunk. If you consider that a single frame at 60fps is 16ms, this is well within bounds to do in even the main thread. So it's should be OK when it comes to more complex geometry. :) I'm not handling the edges just yet but here's a picture of my work in progress:



EDIT:
Culling is now in. Handling the edge cases did slow it down a bit, although I do do it all at once, but I was planning on doing it in a thread anyway, to handle future complexity. Nor did I spend long trying to come up with a genius way of doing it, but it's still better than before. The GM Terrablox actually has the culling and meshing in the same place. Here it's 2 separate things. The meshing is the fast bit, it deals with everything it's given without caring what it is. The culling is the slow bit, but this only needs to be done when the chunk is first generated. Later culling operations only need to cover changed areas.

I added a water plane as well:

« Last Edit: April 24, 2017, 09:18:33 PM by JackOatley »

R34LD34L

  • Sr. Member
  • ****
  • Posts: 340
  • "TERRABLOX!!!!!!!!"
    • View Profile
Re: Call me crazy!
« Reply #7 on: April 25, 2017, 09:57:58 AM »
This looks so epic...good god even tho ive seen most of this stuff before hand still !!! That water plane scene just killed my soul.
Schematic Mod Thingy!!!!

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #8 on: April 25, 2017, 07:05:36 PM »
This looks so epic...good god even tho ive seen most of this stuff before hand still !!! That water plane scene just killed my soul.
To be fair I do use that MSAA hack I mentioned to ya when taking these nice shots. Softens up all those edges and reduces distance noise. :)

I haven't done too much today. We otherwise have some free time here yesterday and today and a bit of tomorrow so I've been decorating! I did do something cool though. Right, well, this engine has 2 UI/HUD layers, one of them is a drawn one inside the renderer (currently has the crosshair), the other is a HTML one (like the fps counter and now the console). So the console is the main thing here, JS can execute code from a string so I thought I'd put that in quickly as A DEV TOOL (you wouldn't release wit full JS execution), so if there's something I want to try and don't want to go into the source to mangle some test code in, I can run it from this console. Check it out:



EDIT:
I put this on Twitter last night because someone was curious as to how fast this generates chunks:



This is in real-time, you can move around smoothly while that generation is happening. But there is still a delay at the start where it reads the heightmap (fast) and places the voxels in the chunks (slow) and even a delay sending that data to the culling worker (slow, not sure why but probably because of large amounts of data to be copied). Each chunk is processed based on the order they are sent to the culling worker, hence the row-pattern. Meshing is almost instant, 15 times faster than culling, so it's culling thats making this pattern actually visible, because it won't process the next chunk until the current one is finished and has been sent back to the main thread.

I'm not sure if this is faster then the Windows TB yet, that initial startup certainly isn't. But I still have optimizations for the culling, as it currently iterates ALL voxel space... TB actually skips 95% of hidden voxels and thin air, so I'm going to use that same optimization method in this.

EDIT:
OK, compare this to the gif above and that's how much faster it is, about 2 - 3 times faster. It's difficult to get an accurate measurement in a browser and when using multi-threading.



The total time it used to take to finish creating the whole map (256 x 256 x 32) was 6.5 seconds. Now it clocks in at around 2 seconds. I spent a lot of time tweaking my process, which included optimizations that worked really well in GM, but it made no noticeable difference.
The actual problem turned out to be the data. When I was sending the voxel map of a chunk to the web worker it had to copy it, a voxel map has the structure [array][array][array]{object} which turns out to be pretty big. I now convert this before sending it to the worker, so now it's an ArrayBuffer with a linear list of entries in the pattern of [x, y, x, texture, cullingmask], which is allowed as a "transferable object", meaning when I post it to the web worker I can list it as a transfer and only the reference will be passed, it clears the data in the main thread and hands it to the web worker with no copying, instantly faster.

EDIT:
I'll be starting to write on my blog again soon with these updates. Because I want to explain so much and this doesn't really seem the best place to do it. Anyway, I'll still put some "exclusive" stuff up here occasionally. :)

In just the last day the engine has come on leaps and bounds. It can now generate even huge maps in a second. I even loaded in a 1024 x 1024 x 64 height map of Denmark. There's still some startup delay as it reads the heightmap and populates the chunks (though I will be fixing this soon), but the culling and meshing still take around a second in total, it's mindbogglingly fast, I'm still surprised how much faster than even GM YYC builds of TB.

Some other things I added are "side rendering", where if it's a finite map it'll draw the sides of the world. And I've improved the shadow mapping, it's now more focused and it follows the camera around to try and get the best shadows possible for the local area; it's not cascading shadow maps as three.js doesn't fully supported that (through lack of maintenance. ie, nobody is really using that feature). Anyway, here's a pic:

« Last Edit: April 28, 2017, 11:27:13 PM by JackOatley »

R34LD34L

  • Sr. Member
  • ****
  • Posts: 340
  • "TERRABLOX!!!!!!!!"
    • View Profile
Re: Call me crazy!
« Reply #9 on: April 29, 2017, 01:50:27 AM »
Good god
Schematic Mod Thingy!!!!

Terrabloxian

  • Hero Member
  • *****
  • Posts: 633
  • Drop it like it's hot
    • View Profile
Re: Call me crazy!
« Reply #10 on: April 30, 2017, 02:19:42 PM »
I dont understand how youre doing all this so fast but all i know is i want to do it too  ;D ;D ;D
Say nope to dope!

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #11 on: April 30, 2017, 06:37:13 PM »
I dont understand how youre doing all this so fast but all i know is i want to do it too  ;D ;D ;D
Spoiler (hover to show)

R34LD34L

  • Sr. Member
  • ****
  • Posts: 340
  • "TERRABLOX!!!!!!!!"
    • View Profile
Re: Call me crazy!
« Reply #12 on: May 02, 2017, 12:22:53 AM »
Lol i cant stop laughing at that response
Schematic Mod Thingy!!!!

JackOatley

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 732
  • Nobody
    • View Profile
Re: Call me crazy!
« Reply #13 on: May 13, 2017, 12:56:44 AM »
I slowed down a bit there. Just my regular depressive slump lol.

I've been mostly working on tidying up the engine and modularizing (I think I made that word up) it. To this end I also started working on a separate web app just for the heightmap thing as I suggested before on my blog, it directly uses the height map class and methods from the engine so when either project evolves, so does that source.

I'll talk more about the height map app soon as it's functionally complete and the UI is pretty much done. Just a few buttons need text and some non-core features. But for any of you working on projects involving height maps it'll be pretty useful for creating unique maps from generated or imported images.

As for the engine, I've set it up now to allow specular, bump and normal maps. You can't use bump and normal maps together, but they give different effects. I plan on setting it up so you no longer have to edit the textures as a whole sheet, they'll be individual textures in sorted folders and the engine will handle stitching them together. The goal of that is to make the different maps optional per texture, and possibly allow custom mipmap textures. Mipmapping is already automatic but currently suffers from texture bleeding if used, so the stitching will be able to fix that anyway without custom maps.

Today I added in most of the old Terrablox tree models, the only thing that needed to change was changing the string names like "leaves" to the actual index number. Anyway, here's a quick pic:



Also, I got a 1,000,000,000+ (2048*2048*256) (billion) voxel world running. It's very hard to get a good result when you hit that 256 height mark when using a height map, as that's it's max fidelity. So I'll have to come up with a way to increase detail using the Heightmap class, rather than effecting the images.

EDIT:
Got the texture loading from individual tiles today, which is handy. It's also able to build and display the world before any of the textures have finished loading, and it should be able to update the texture on loading of each individual tile... Which isn't massively helpful, but is interesting, it would mean even if you have large textures with normal maps, specular maps, etc that it wouldn't incur any loading time, it's all asynchronous.

Also added basic beach/shore detection and fixed the tree generation (it previously only spawned trees on slopes facing a certain way), this was due to me forgetting I have a 1 voxel border on all 6 sides of a chunk now, so I was forgetting the "up" offset:



And here's a sample screen of the Heightmap Creator app I mentioned, I wasn't able to actually do anything on it today, though. I've got a lot of cool ideas for this though, it'll be a cool tool. :)


« Last Edit: May 13, 2017, 10:03:44 PM by JackOatley »

R34LD34L

  • Sr. Member
  • ****
  • Posts: 340
  • "TERRABLOX!!!!!!!!"
    • View Profile
Re: Call me crazy!
« Reply #14 on: May 14, 2017, 09:03:30 AM »
Welp terrablox is going to be epic again !!!!
Schematic Mod Thingy!!!!