Productivity, generality and OpenMP
Flash news.. I'm very busy at work 8)
I could work less, but I want to produce something good. I like the idea to take a more general approach to problems and make something bigger out of it.
One of my current goals is to develop in a scalable manner. In order to do that, things need to be rethought in a more generic form.
For example one can have a triangular mesh, or could develop a system to do a remeshing to turn the original geometry into a semi-regular data structure that can easily be compressed and streamed progressively.
I think that scalability is really a key to the much needed productivity improvement in game development.
At work we talk every day about how to go about some solution, and there the key question is always: "can we use a scalable and generic solution ?".
This is usually about development pipeline.. not about actual code. The idea of code reusability is less straightforward. I actually aim more at providing simple implementations, to modularize code so that it can easily be grabbed without too many dependencies.. rather than trying to fit all in a supposed grand scheme of hierarchy of objects and whatnot.
In the end the harder problems are really those about how to organize data and how to transform those data across the development pipeline.
On the side, I also used OpenMP for the first time. After a few odd results, I managed to parallelize a loop that uncompresses images in that progressive-JPEG-like format that I've been working on.
Like for JPEG, the image is processed by 8x8 sub-blocks. Using OpenMP pragmas I set the parallel section to happen on rows of blocks.
Parallelizing every row of blocks makes sense, but I could probably try to do multiple rows at once to see if I can reduce overhead of context switching and potential cache trashing. Parellelizing every block instead turned out to be overkill.
As a rule of thumb, if I think that I could wrap some code into a function with practically no overhead, then perhaps I can make parallel section out of it. In fact, I think that OpenMP eventually grabs that section and makes a function out of it anyway...
Aside from some early decoding artifacts due to my inability to share some variables from outside the "parallel for" (see the example at the bottom here), using OpenMP was really easy. Definitely much simpler than manually creating and reusing threads, also less involved than using Threading Building Blocks because one doesn't need to create functor objects and also because OpenMP is readily available with modern compilers with minimal effort.
cool
zzzzzzzzzz
- Davide's blog
- Login or register to post comments
EEhhhh... OpenMP. I dont
EEhhhh... OpenMP. I dont know much about it. Seems like its not too popular on the consoles though.
13+5=18
Incidently this is a question to all the power users there who like to tweak options etc etc...
Whenever I send some large attachment (1mb+) using Gmail or any other web based email program it often hangs up my router. If I send them through Outlook Express then everything is fine. If anybody has an answer as to why this happens, let me know!
threading for tasks ?
I first learned about OpenMP from the XBox 360 SDK 8)
.. but I realize that with very few threads and with a fixed target one is tempted to do "manual scheduling" by using pools and workers directly.
However, for the future, it seems like we are moving towards hundreds of threads (GPUs already do that, transparently thanks to the simplicity of shader languages). With so many threads in play, it's going to be very painful to think in terms of "jobs".
One has to thread loops with ease and that's where OpenMP really helps.. makes it easy so that one doesn't have to think too much about using MT.
As for the attachments. GMail is web-based so it obviously uses the HTTP(S) protocol for sending files. I'm not sure how ports are contracted, but it's definitely different from sending a standard mail where attachments basically become text using MIME BASE64/UUENCODE or stuff like that.
See if you have the same problem sending large pictures or binary files from the browser to sites like Flickr, Picasa, YouTube or any of those that allow temporary sharing of binary files (probably easier).
I think OpenMP is fairly
I think OpenMP is fairly limit for real game development use (perhaps it's usable on the tools side). The library was originally design for scientific computing purposes. So maybe if one is doing some kind of "predictable" work on a data set, then it's ok for use in that area.
Anyways, we'll see what the future brings ...
Why no OMP ?
Why do you think it won't fit needs in games ?
OpenMP is for threading loops mostly. It's for automatically splitting data in chunks and have different processors work on those chunks.
So, maybe it's not good for games because it doesn't give you a "tasks system" kind of multi-threading, but one should avoid that as much as possible. It's much better to to work by splitting large data in a deterministic way, rather than thinking in terms of living objects to synchronize manually with locks and semaphores.
I think that games logic is better off with serially executed virtual threads. Manually managed threaded tasks become necessary on specific cases such as data streaming. But for number crunching it's better to use higher level systems such as OpenMP.
Also interesting: Intel's Ct, (Software) Transactional Memory, the already mentioned TBB, Google's Map Reduce (see video).
P.S. How's life back in America ? Speaking so much American English all at once is making you skip past tenses when writing already ;)
I'm not ruling it out
I'm not ruling it out completely, but for things like AI (that relies on other systems like physics animation) it's not too ideal since there's an inherent waiting on needing certain information to be present to make AI "decisions". I'm not hearing too much about OpenMP with games anyways (papers, etc) which seems to imply that people are either using other solutions or rolling their own. Some interesting stuff here by the way: Insomniac Games :: R&D. I've heard a lot about Google's Map Reduce; I'll have to take a closer look at that soon.
How's life here? Well, it's not too bad ... people are complaining about the "high" gas prices, but I tell them that it's close to $7 a gallon in Japan and they quickly stop. 8P It's also nice to have some large living space again and being able to easy access books and actual people to talk things technical with. I have to admit that I do miss life in Japan (a little bit), but not to the point that I'm ready to pack up and move back! I need to buy a car at some point; fortunately I live close enough to work to be able to walk or bike (just like when I was in Japan!!!).
The "bad" English you're seeing is something that I've been struggling with since I got back ... for some reason, the Japanese part of my brain asserts itself more than the English part ... I'm actually having trouble speaking English! 8P I suppose it will take a few months to recover ... but I have an opportunity to speak Japanese daily here ... so maybe no hope of recovery for me. 8P
Mr Ragin is back in the
Mr Ragin is back in the USA??
Details!!!!
Insomnia
I think OpenMP is not as used because we aren't really dealing with hundreds of threads just yet. But next generation hardware may well go that way.
Interesting the site form Insomniac.. their presentations are a bit on the quantity than quality, but I know how painful it is to make presentations (and to put the compiler down while doing it 8( ).
At R&D where I work we started with a big wiki, but as things changed that went a big forgotten. Now we have some sort of custom made blog which works pretty well, but we may be moving to more intranet solutions... though the real problem is that intranet itself is not very appealing (to me). I really don't like the idea of having to write stuff for coworkers.. it's too limiting for me.
I'd rather remove sensitive info but expose work to everyone else out there.
I'm not holding my breath.. but hopefully when the project is finished, there is going to be some disclosing.
moooooo
Skeptikal
I'm still skeptical about OpenMP becoming one of the standards once the number of threads increases ... but I guess we'll have to wait and see what happens.
Yeah, writing for co-workers alone isn't too "cool" compared to writing for the general (technical) public, but it's better than nothing. Keeping things up to date is a serious pain too. Perhaps 4 hours each week should be allocated to this?
My guess is that eventually you'll get to publically disclose stuff ... maybe it'll be along the lines of this kind of presentation:
西川善司の3Dゲームファンのための「ロスト プラネット」グラフィックス講座 Xbox 360グラフィックスここに極まる! 日本発の次世代技術の秘密とは?
For you non-Japanese readers ... 8P
http://forum.beyond3d.com/showpost.php?p=919807&postcount=116
8 cores are wasted with Capcom ;)
I think I saw that article before.. but I definitely didn't try read it.. it would take too long and it doesn't seem like it's breaking any new grounds.
Impressive all that GUI system they have.. I guess it's also because Capcom is doing actual games with that editor thing.
I tried the Lost Planet demo on the 360 (downloaded a while ago) and I didn't think it was that good (low poly counts, simple lighting). There was also some sort of PR demo at IDF 2007 with Lost Planet running on 8 cores, but again it didn't look so good, perhaps because they aren't really doing anything special (I was there and I remember the cores being barely used indeed).
All thse games nowadays are so GPU-centric, we are pretty much in the hands of NVidia and AMD when it comes to real-time graphics. They tell us what to do, which systems and formats to use.. and the best engines out there are mix and mash of those tricks, lots of brute force, little logic.
About OpenMP, is there any specific reason why you are skeptical ? It seems to me that OpenMP is the only ready multi-platform solution for multi-processing. All it takes is a compiler switch and an include.
The biggest issue with
The biggest issue with writing those wikis is keeping them up to date. Sometimes its better to have no documentation than documentation that is wrong. But trying to remember to do updates everytime something changes becomes a fulltime job after a while.
Insomniac is very popular when it comes to Ps3 Cell programming. They have Mike Acton there who I believe came to them from IBM. Incidently he doesnt trust compilers or even intrinsics and insists on programming in asm =)
On the other hand they say that they heavily use some of the modern c++ concepts like policies, delegates etc... Which is a bit dubious if your goal is writing high performance console code...
technical writers maybe ?
I think there should be some sort of middle men. I guess you call those technical writers.. sometimes I wish someone would ask me what I'm doing in detail and write about it (possibly even in Japanese ;)
Unfortunately we don't currently have that luxury (if we had someone good enough to understand and write about such things, perhaps we'd put that person to code !).
Though we do have video recording for all the key meetings of the project.. I just wonder how's all that video going to be dealt with.. video editing ? ouchhhhh
OpenMP wasn't originally
OpenMP wasn't originally designed for games; rather that tweaking it to work for games, the whole mutli-processor programming issue should be deal with from the ground up. If you have time, check out this video lecture: Making Concurrency Mainstream. Might require IE to view though ... :-/
No IE on Mac here ;) There
No IE on Mac here ;)
There are a lot of things that we use that weren't meant for games... sometimes games push technology.. sometimes they don't. Multi-processor architetture is the way supercomputers exist, but it's something still fairly new to games and game programmers.
ole'
You might find this
You might find this interesting: We can transform single thread to multithread: Intel
If this actually works, it would be a lot better than OpenMP. Much better (possibly safer) than doing it manually.
I thought I mentioned Ct
I thought I mentioned Ct before, but I can't find the post 8)
Ct is still under development, but definitely very interesting.
I'm in touch with the developers and sometimes I get to exchange some opinions with them.
Ct is coming up with some interesting solutions. We'll have to see how they work in practice. Mixing Ct's subsystem and C++ is definitely tricky.
I like how Ct sometimes looks like it's heading towards shaders.. I'd be very happy to see HLSL and Cg pan away to give more freedom to developers (enough of toy languages !).
The interesting thing about Ct is that basically has its own compiler which will work transparently in the background at run-time (doing JIT is pretty much a must nowadays for parallel processing).
Another cool thing is that Ct also copes with vectorization, allowing to use SSE2, 3, 4 and whatever comes next..
Ah, I didn't know about Ct.
Ah, I didn't know about Ct. That looks pretty interesting. 8P I think I'd better get a machine with actually more than one CPU though. 8P