-I moved out of home into new apartment. It took me almost two months to find it, the rent is quite high, but the place it's really worth the money. Finally, I have some own space, privacy and equanimity, so I can work on my projects without being disturbed all the time. Hooray! :)

-I finally decided to visit my doctor for a medical check-up. I had last one like 10 years ago, so I was a bit afraid of the results. Well, apparently I should be perfectly healthy, as the results of all tests were okay.

-I found my lost uncle that I haven't seen for about 20 years. I was five when my father died, but at that time, I was living in a children's house and I never seen anyone from father's side of family again since I returned to home.  Unfortunately, I found out that he is the last living member of that family, as my grandparents passed away a long time ago, and the rest of the family as well.  Well, I also found out that my father and my grandparents weren't buried, thanks to economy situation at that time  (post-communism era). I think that they deserved better, so I'm going to start saving money for a joint/family grave, so they can be buried and be together once again. Anyway, my uncle gave me some new-found photos of my father and my grandparents, so I'm really glad that I found him afterall.

-I'm currently trying to finish my thesis and prepare myself for a final exams. I already have the practical part of the thesis finished, but I still have to write at least 40 pages for it, which will be the hardest part for me. Well, I still have a month left to manage that (the dealine is July 6, exams starts in August), so I hope that I will be able to finish it in time. It's my very last chance, and if I don't finish the thesis or pass the exams, then I'm not going to get my degree. No retakes! :(

That's it. I probably forgot to mention something un/important, so I'll update this post later :(

She is truly irreplaceable, and I'm not going to replace her, so this was my very last pet I ever had.

Rest in peace, my beloved friend... and hold on there, we will see each other again one day...

It's only a two screenshots of main menu, but as you can see, it's running in windowed mode, which is one of the most requested mods for fullscreen-only games, like Arcanum :) It's also possible (thru config file) to force any windowed game to run in fullscreen, and force display resolution, window geometry, etc.

Now I'm going to implement support for palettes, color keys and other misc stuff. Hopefully, I'll have it finished before end of this week.

Edit: Open those screenshots in new window, if you want to see them in full resolution.

Anyway, I just found out that there are a huge design flaws regarding the color palettes and DirectDraw usage.

First of all, Arcanum isn't using IDirectDrawPalette interface at all, and all the color index to pixel conversions are done on CPU. This slows down whole level loading process, but that's still okay, no big deal.

Secondly, the game is running in 16bit fullscreen mode. Yes, you read that correctly, 16 bit color depth, not a 24bit! What does it means? It means, that 8 (or even 9 bits) are removed from every color from the color palletes in pixel format conversion, so the range of possible color shades is significantly reduced, from 16 777 216 to 65 536 (or even 32 768), thus the overall graphics quality is reduced significantly as well.

It's 16bit color depth, but it doesn't mean that it's R5G6B5 pixel format. The DirectDraw also supports X1R5G5G5 pixel format, and it's up to driver which pixel format it chooses for 16bit color depth.

Anyway, few year back, I made a mod for Arcanum, that forces the game to run in windowed mode, which was disabled in Arcanum in release version, because it wasn't finished and fully working feature.

What's surprising, is that when the game is running in windowed mode, there are no pixel format conversions, and game uses a full 24 bit color depth range. That's great, because it means that it's possible to force the Arcanum to run in 24 bit fullscreen mode.

If my time allows it, I'll fix this problem in January, and release it as .exe patch. I found this issue thanks to my DirectDraw emulator, and hopefully it will be useful for finding bugs in other games as well :)

So after that, I rather decided to write my own library, which will be able not only to log all the calls, parameters and other stuff, but it will be also able to redirect all calls to original library, so it'll serve also as proxy library. Most importantly, it will be also able to emulate all the DirectDraw features.

That was the plan, but after two weeks working on it (well, in free time only), the only thing I've managed to implement is - all the DirectDraw interfaces and it's versions  (i.e. DirectDraw/Surface/Clipper/Palette - 1-7) , proxy calls and functions/methods & parameters logging. It's kinda powerful tool already, but something important is missing...

What's missing? the emulation part :) on which I just started working. I chose Direct3D9 for the rendering and stuff, but if I manage to make everything working, I'll also add support for OpenGL, if needed.

Anyway, I'm testing the library on Arcanum, and you can see the debug output here: http://pastebin.me/2582666d95f3f39b81a49edb8d00aee0

Kinda cool, right? I call this library DDrawEmu :)

Well, guess what... I'm still pretty busy, as I just started to work for two unnamed companies, so I got two jobs, and also I'll be preparing myself for final exams, for at least next 3 months, because I have to take the exams next semester (I can't delay them anymore, as I've already delayed them for 3 semesters).

Why I'm talking about this? Well, I've promised to post a long development report for the OpenArcanum project before the end of this year. Also, I've promised to fix the source code, and then release at least part of it.

Unfortunately, it seems that I won't be able to fulfill all these promises... so I'm really sorry guys! :(

I'll try my best to post at least something, however I was planning to post very detailed information about the engine, plans, and tasks, so it'll be nothing against what I was initially planning.

I wish I could split myself into two...

Few days ago, I got an idea for tile image compression for 2D isometric games like Arcanum, whose tiles are all in same shape, so today, I thought that it would be nice to try to implement it, and compare it with current compression algorithm, the RLE.

For those who don't know anything about Arcanum's file formats, in the Arcanum, all tiles has same dimension, 78x40 pixels, and it's colors are reduced to 256 color palette, then resulting indices are compressed by RLE compression, and saved with palette into Art file.

The idea about the new tile compression is very simple: save only what's visible, which means that the compression algorithm will remove all transparent pixels (mask) from left and right side of the rows in the image, and then save trimmed rows to output (raw) stream.

Anyway, I did implemeted it today, and I can say that the compression/decompression is very simple and fast, since the algorithm consist only of two single for loops and one memcopy call for every row, so it's even faster than RLE compression. Also the algorithm has fixed compression, so the compression ratio is constant, and it's not required to save any meta information for decompression, thus some space gets saved.

So I tried to compress and then decompress one tile (more precisely, it's indices) 100 000 times. And speed?

Tile compression:

Compression: 274 ms

Decompression: 433 260 ms

Compared to RLE:

Compression: 4 041 ms

Decompression: 2 447 ms

Oh yeah, in this case, the compression is almost 15 times faster than RLE compression, and more importantly, decompression is also faster, more than 5 almost 10 times faster.

The reason why decompression is slower than compression is that, it's required to fill output buffer with zeros (i.e. transparent pixels) before actual decompression, plus I'm using own memory fill function that is optimized for larger blocks (SSE2), thus making decopression slower. (Optimized!)

About compression ratio, one tile image has 3 120 (78*40) pixels, where 1 596 pixels are always transparent, so compressed files has 1 524 (3 120 - 1 596) pixels, thus the ratio is 0.488 (~1:2).

If you multiply those numbers by size of one pixel (in bytes), for example 3 for 24bit images, then you get almost 5KiB (4 788B) saved per tile. Now multiply that by 100 000...

What about RLE? well, it don't have fixed compression ratio, so it's hard/impossible to culculate/guess those numbers, but in the Arcanum, terrain is very varied and colorful, and most of the colors in palettes are used, so the RLE won't be as much effective as my tile compression.

Anyway, with such fast tile decompression, it's possible to decompress the tile on fly, when requested, no need to keep them decompressed in RAM anymore.

Pixel get/set functions can also work on compressed tiles very fastly using precalculated lookup table (for row offsets).

Also, all the tiles can be stored into one big tileset bank, and compressed with deflate or huffman compression, so we get even more space saved, and loading will be also faster.

The whole bank can be now loaded into memory for superfast access!

What's wrong on this implementation? it's slow! Not only that there must be some helper container constructed with a variables whose purpose is to keep the foreach state (to switch and interupt second for loop), but also, for every iterated element, there are several useless condition tests and var assignments, only to make it working.

That's a way slower than normal for/while loop with iterators.

How can we speedup the foreach loop? Can't we somehow remove the second for loop and the helper container? Surely we can, but it won't be easy (and for someone nice).

I spent all day researching and implementing different foreach loops, but in the end, I had to split it into two for loops anyway.

It's almost impossible to implement it to work with references. Pointers are easy thing, task for 5 min, but it isn't as convenient as foreach with references.

Anyway, that way the portable-way attempt, later I managed to implement the foreach loop as one for loop by hacking the reference via templated asm function. Someone may consider it as ugly hack, but who cares, when it's a way faster than other implementations, and you can always easily switch to generic implementation, because it's nothing more than a macro.

So how did I implemented it? Have a look at this picture:

This picture shows how my foreach implementation works, before I converted it to macro. It consist of 4 variables:

1. -First variable is containter's iterator object. Nothing special about it, use your own or stl implementation.

   
   

2. -Second is pointer to current iterated element. While iterating, it's compared to NULL, so it's very fast.

   
   

3. -Third variable (reference) is the most important one. It's actually pointer to fourth variable ("element"), which is reference to currently iterated element, so it's pointer to pointer to "element" reference. After calling pointerToReferenceEx, it will point to variable reference, so we can change reference's address to point to different (next) element.

   
   

4. -Fouth variable is currently iterated element, it will always point to new element after every for loop iteration.

   
   

As you can see, I'm using new Cpp0x features, auto and decltype, to determine type of variables, because C++ doesn't have any typeof operator (GCC has it as extension, which is used by Qt foreach loop as well).

That's nothing special, the magic relies in the pointerToReferenceEx function (I'm not going to release the implementation yet), which takes as first argument a pointer to element you want to dereference, and which is able to determine address of reference on the left side (varible "element") of the expression and assign it to variable "reference" before the function returns! In other words, the function is able to determine the address of reference where the input pointer will be dereferenced and assigned.

Using this function, we don't need second for loop, nor any other helper container, since we define the variables out of scope.

Yes, we can define the variables before for loop, and still be able to have more foreach loops in one scope without any variable name collisions!

How? Piece of cake, we make the variable names unique, by appending the current source code line number to them!

This can be done automatically by using predefined macro __LINE__ and using some helper macro for concating the variable names with the lines number.

Here's the final macro:

#define _ArgConcat(a, b) a##b
#define ArgConcat(a, b) _ArgConcat(a, b)

#define AppendCurrentLine(x) ArgConcat(x, __LINE__)

Usage example (must be on same line!):
bool AppendCurrentLine(myBool); if(AppendCurrentLine(myBool)) return;

So this way, the variables are unique, and we can have more foreach loops in one scope! Final foreach macro might look like this:

#define ForEach(Variable, Container) ...

ADynamicArray<AInt32u> array;
...
ForEach(x, array) // x is defined as AInt32u&
{
// do something fancy
}

Out of interest, in the AGate engine, there are also more advanced foreach loops, to allow to take control over more features, so there are several foreach macros:

#define ContainerIterateEx(Container, Variable, Iterator, Direction, Seek) ...
#define ContainerIterate(Container, Variable, Iterator) ...
#define ContainerForEach(Container, Variable) ...

That's it. Nothing more to add, maybe I'll only note that I'm not using any foreach loops in C++ at all :) I think that it's feature suitable for scripting languages only, but I did provided foreach loops in AGate engine for those who would like to uses them.

That's all for now. Crypton's out! :)