I'll be in Los Angeles in two weeks for Siggraph 2008.
Because of that, I've been coding a bit less and spending more time with Excel and Power Point, preparing for some meetings that I'll have there..
Using MS Office depresses me. I used to pride myself of not understanding what possible use for Office I would have.
Now, I think it's important to make Power Point presentations for all those people that aren't talking with you daily and that can't or won't understand code. One must learn to communicate and perhaps to inject some hype, too (MegaTexture, Geospatial, real-time Ray Tracing ;) ..but when time is scarce, then it's depressing to think that I could be effectively writing code and solving interesting problems rather than picking a nice font or resizing a picture in a corner of a presentation slide.
This reminds me a Dilbert strip that to me is now more real than ever: at an external site there is Wally's head coming out through a hole in table, inside a glass bell. Wally's face looks tired and unshaved. A spectator is looking at it while Dilbert explains: "this is what our 3D product would look like if we didn't have to waste time preparing demos". ...my feeling exactly 8)

Anyhow, I did write some code too. For one thing I'm at a good point with this "geometry processing framework". Basically a 3D engine tailored for geometry manipulation rather than rendering.
I once wrote something that was mixing both geometry processing and rendering, but it was a bit of a pain to maintain.
Personally, I think that it's more difficult and more important to put down those kind of frameworks rather than getting Direct3D/OpenGL to spin an object. Making API calls to a spoon-fed rendering interface is not the same as manipulating, optimizing, organizing data.

On the texture side, I've started writing some Power Point (indeed) and the goal there is to push the system rather than the format. The idea is that a JPEG-like format is being built but it's important to see how the eventual engine behind will handle it best, unpacking at selected texture resolutions on the fly depending on the VRAM and bandwidth budget.

Very important is also the build process. I've been asking artists not to optimize textures. Current assets are good enough to make high quality pre-rendered movies.. one character uses about 1GB of texture memory. However, many times these textures include an alpha channel that is not used and that can safely be removed. Bump maps, which only need one value can also be converted from RGB to grayscale (can't do it for color textures because in DX10 there are red-channel-only textures instead of luminosity textures (bha !)).
Anyway, the idea really is to make things as scalable as possible.. this is exciting because it gives authoring freedom.
One could ask an artist to make a model at a specific resolution or even to make one highres and then manually convert it to lower res with normal maps.. but that's all time wasted doing manual labor. Manual work is what should be avoided.. artists shouldn't worry about simplifying geometry unless they want to.

Time to sleep.. the Melatonin is finally making it's effect 8)
zzzzzzzzzzzzzzzzzzzzzzzz

I've finally setup a nice decompression thread that takes care of everything (almost) without any hiccups from the rendering side.

Right now I'm only decompressing textures in full size.
When a texture it's loaded, the average color (or 1x1 mip-map) is loaded right away (the 1x1 mip is stored in the file header without any need for decompression). Then a task for decompressing the full size texture is created and the decompression starts right away.
Right now it's either 1x1 or full-size, next I'll have to pick actual intermediate resolutions depending on the image being rendered.

One nice thing that I found out is that DX10 is thread-safe by default (no need to set special flags, making one fear for degraded performance). So, the actual texture creation and decompression is all happening in a thread separate from the main loop.
As I mentioned before, a texture decompressor will unpack data in slices using OpenMP. This allows me to use whatever cores I have to decompress one texture at the time as fast as possible (more cache coherent than trying to decompress different textures at once).
Currently, during load I can see all 4 cores being used to the maximum (more or less).. it's a nice feeling 8)

To minimize initialization times, I've also implemented a simple Direct3D texture object cache.
Every time a texture is released by the engine, it doesn't actually get immediately released in Direct3D but rather added to a free list (unlike the Wikipedia article, I'm actually not using linked-lists) where it can be picked up again if there is another request for a texture with the same characteristics.. otherwise it will be released from D3D after a few frames being unused.
Cache aside, I'm mostly counting on the multi-threading to cover up for all those potential stalls incurring from resource management in D3D.

The next big step will be to continuously "resize" textures depending on the needs of the frame being rendered.
This should happen lazily to avoid too much work by the decompression system.
The reason why one would want to resize textures is to economize on memory, but that doesn't have to happen at every frame for every texture. It makes more sense to keep a texture at 1024x1024 even if the next frame only needs a 512x512, rather than putting to work the decompression. The 1024x0124 may be needed soon again and the rasterizer is going to pick the right mips regardless of the maximum resolution.

In general I like the idea of progressive quality. I like the idea of drawing geometry with whatever texture resolution I have. This way I can theoretically keep a consistent frame rate and the quality of the textures will change depending on how fast the decompression can happen (which depends on the CPU/GPGPU power).

Next next I'll have to worry about geometry.. that's trickier especially when using a lot of complex materials.. but hopefully some coworkers can help there ;)

wooooo
zzzzzzzzzzz