Molly Rocket :: View topic - Immediate Mode Graphical User Interfaces (IMGUI)

Molly Rocket
Molly Rocket games and everything even tangentially related

FAQ

Usergroups

Immediate Mode Graphical User Interfaces (IMGUI)
Goto page 1, 2, 3, 4 Next

Molly Rocket Forum Index -> IMGUI

View previous topic :: View next topic

Author

Message

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

	Posted: Mon Jul 04, 2005 7:34 pm Post subject: Immediate Mode Graphical User Interfaces (IMGUI)

I've been long overdue to get some information out there about this technique, which I think is a much better way to write GUI code than what's currently out there. Apologies for the large file size, but the makeshift lecture ended up being quite lengthy. http://mollyrocket.com/861 Anyone who wants to mirror, feel free. - Casey Last edited by casey on Thu Sep 11, 2008 1:45 am; edited 2 times in total

brianhook

Joined: 23 Feb 2005
Posts: 89
Location: Adrift in a Sea of Ennui

	Posted: Tue Jul 05, 2005 2:32 am Post subject:

I like it. I think it would probably greatly benefit, however, from a more thorough discussion about how to integrate with an event loop. There's a lot of assumption/glossing over when it comes to "widget checks input state" -- a common RM GUI "pattern" is that there is a way to filter events to all the elements, and IM GUI doesn't really do that.

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

Posted: Tue Jul 05, 2005 2:37 am Post subject:

brianhook wrote:

I like it. I think it would probably greatly benefit, however, from a more thorough discussion about how to integrate with an event loop. There's a lot of assumption/glossing over when it comes to "widget checks input state" -- a common RM GUI "pattern" is that there is a way to filter events to all the elements, and IM GUI doesn't really do that.

Since it's assumed that you're running in real-time, there is assumed to be only one meaningful event process per call of the UI. Ie., you can boil the entire input state down to a buffer of pressed characters, the mouse button transitions, and the mouse delta. That whole thing gets stuffed into the ui context, which the widgets are free to check as they please.

One key observation that most people typically overlook, I think, is that the average windows message loop processes only one or two UI messages at a time before going idle again! So structuring your code to handle meaningfully large input strings is often a really bad tradeoff, because it never gets utilized but forces you to write much more complicated code. In the rare circumstances where you think you might need supersampling, you can always pump your UI multiple times a frame, but I have never found that to be necessary so long as accurately track mouse events and have a keyboard buffer for widgets that need to accept typed input.

- Casey

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

Posted: Tue Jul 05, 2005 8:24 am Post subject:

Sean Barrett, who's upcoming article I referred to in the video, sent me some questions subsequently. I'm replying to them here so that everyone can see the answers:

Sean Barrett wrote:

you probably should explain better why hot works the way it does. it must seem weird to people that you do this test-if-inside-and-then-make-hot after you made use of the hot test, instead of just testing 'if-inside'. (I realize the reason is because since you do painter's, this means the last thing to set hot is on top, but you never explained that part.) In fact, I'm not sure I understand it all--while something is active, other things stop setting hot, but the active item doesn't, so you can tell whether the active item is still currently 'over'... I take it you clear 'hot' at the start of every frame? I guess that's all I missed.

Actually, it's not just because of painter's algorithm. Yes, for sorting purposes, it is convenient to have one frame of lag on the hot response, because it allows you to do a z-compare in there and sort on z if your UI has that concept. But that's not the main reason - I'd do it this way even if nothing in the UI overlapped. The main reason is because the UI context can deny you hotness if some other item is active. So you could, if you wanted to, move the check to the top of the function if you didn't care about sorting, but you would still need it to be more than just an is-inside. As for clearing the hotness, yes, it is cleared every frame. Active is not.

Sean Barrett wrote:

have you actually used 'hot' for a keyboard UI? I think you need a separate 'focus' context.

Yes - sorry, I didn't mean to imply that you could use the same hot for both. I meant "it could mean hot for mouse, or hot for keyboard". If you want a hot for mouse and hot for keyboard, then you need two. I don't normally do keyboard UIs so I haven't thought much about it beyond that though, since the only thing that gets keyboard input is edit box code which sets itself active and then snarfs all keyboard until it becomes inactive. There's the more advanced concept of "focus" that moves around with the TAB key and such, and I haven't ever played with that sort of thing.

Sean Barrett wrote:

one thing I ran into which your description seems like it would have a problem with, but maybe you're just oversimplifying: if you decide to stop calling a widget, everything is fine, unless that widget happens to be the active one, in which case it stays active indefinitely even though it's not being called, and thus you can't actually stop it and interact with anything else. (I have a thing where once an widget goes active, it has to keep setting a 'still_active' flag every frame, or else the library clears the active context at the end of a frame. (Sometimes I use a lighter-weight version of this; at the end of a mouse-up event I kill active, regardless. Actually, the refined version might only check 'still_active' after a mouse-up.))

Hmm. I don't think I've ever run into that problem, but that makes sense. My more complex UI code nowadays actually uses much more advanced stuff because I've been slowly building in the database stuff and editor-centric structures that handle other things that are not GUI-related (undo, automatic UI generation, etc.), so I think all the cases where I would've run into this only occur in that codebase, where the problem is not applicable.

Sean Barrett wrote:

you don't mention the issue with GUI elements that are "pure GUI" rather than representing app objects; like a table with columns that you can drag to resize and reorder, or having scrollbars when things don't fit onscreen--your app now has to store that state somewhere, and it's not stuff the app was already storing necessarily. I'm sure there's others, I just haven't done that much.

My opinion on this lately is that it actually is data that the app is storing, if the app wants to present a flexible UI that can do things like restore its state on save/load and so forth. So I treat things like the window scroll position and column widths as first-class data that needs to be saved and so on. Now, obviously, I put the code for handling all this into the library, so that the app can just call on pre-written code to do it all, so in the end you're still just calling DoScrollBar(), but the philosophy is that it is app data. State other than a transient interaction is app data, period. That's my take on it.

Sean Barrett wrote:

implementing and maintaining multi-select (trivial when it's just in the objects)

I don't think of that as trivial under any circumstances. Good multi-select is hard to implement no matter what GUI library you're using I think. Like, when you want to select, say, a bunch of different types of objects that all have a position, and then interact with the position, etc. That's one thing I really want to add to my current codebase and I know how I want to do it, but I just can't find the time for it unfortunately. Next game Smile

Sean Barrett wrote:

how do you implement a table with clickable columns to control sort order?

Immediate mode can retain data during the frame, remember. So I'd implement it by just bufferring all the calls to AddRow(), then sort, then display. This is one of the key things that I failed to realize when I first did IMGUIs: immediate mode does not mean that you cannot create internal structures to the GUI library. It just means that the app doesn't have to manage the creation thereof. This is exactly like graphics. The graphics card does all sorts of FIFO stuff and hidden caching god knows what else. But the important thing is that it always appears stateless to the application. It doesn't matter what happens under the hood as long as the app sees an IMGUI interface, because that's where the benefits come for the app. Having or not having state internally that is unknown to the application is just implementation details.

Sean Barrett wrote:

I realized that the biggest problem with the multipass stuff that I do (which isn't just because of one frame of lag, it's because if you tried to do layout with one frame of lag you'd need somewhere to store the info during the intervening time, and managing that would be a hassle--and also because you'd starting adding more frames of lag as things got complicated) is the thing you mentioned about e.g. needing to send up rendering contexts; in my system, I have to have a call 'doUI()' in the app that all it does is traverse the whole UI, and call it for each event (paint, keyboard, mouse, etc.); whereas in your system you can just at any time anywhere in your code (e.g. when you've already got the camera set up) call a UI function... that definitely seems a lot nicer in general, although I don't know how big a deal it is in practice. (I've never done mixed 2d&3d.)

I'm pretty much single-pass only these days. I don't like multi-pass and I haven't explored it further than the first few things I tried with it.

Sean Barrett wrote:

I've used game UIs that dropped keypresses when you were typing in savegame names and such. If for some reason your frame rate was low, don't you lose keystrokes? I guess you can queue them and deliver them one per frame.

I allow widget code to loop over the event queue internally if they're worried about the specific nature of the events that happened. So, for the edit boxes and such, they all loop over the keystrokes themselves to make sure they don't drop anything.

- Casey

sean

Joined: 01 Feb 2005
Posts: 1392
Location: Kirkland WA

Posted: Tue Jul 05, 2005 8:40 am Post subject:

casey wrote:

But that's not the main reason - I'd do it this way even if nothing in the UI overlapped. The main reason is because the UI context can deny you hotness if some other item is active. So you could, if you wanted to, move the check to the top of the function if you didn't care about sorting, but you would still need it to be more than just an is-inside.

Not really; once object X goes active, you don't ever need hotness; you just need active widgets to do an 'is mouse cursor over me or not' test on, say, mouse-up; you can just ignore hotness entirely. I'd argue that this notion of 'hotness' that you have is an overloaded hack: it means "which UI element would I interact with if I did something" when nothings active, and it means "am I over the active element" while something's active. It falls out of the implementation straightforwardly, with the one if test to check if something else is active, but they seem to me two different things.

It may be convenient, because it lets you put your hit-testing in one place inside your doWhatever() function, but you could instead just do:

Code:

bool doButton(...)
{
bool result = false;
bool hittest = isMouseCursorOverMe;
if (active==me) {
if (MouseUp) {
if (hittest) result = true;
}
} else if (NothingActive && hittest) {
if (MouseDown)
SetActive(me);
}
...
return result;
}

The only case this doesn't work, it seems to me, is when there's overlap.

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

Posted: Tue Jul 05, 2005 10:11 am Post subject:

Code:

That certainly seems reasonable, but it just strikes me as ceding too much control to the widget. I don't really think of hotness as a hack, but rather the way in which the ui context brokers the ability for a widget to be interacted with. Typically yes, it boils down to what you've written in the non-overlap case, but I also think you lose some abstraction there. But maybe there's really no other reasons why you need that brokering, so perhaps it is just extraneous in the non-Z case.

- Casey

NeARAZ

Joined: 05 Jul 2005
Posts: 51
Location: Kaunas, Lithuania

	Posted: Tue Jul 05, 2005 2:26 pm Post subject:

Great presentation, I must say. For my own stuff, lately I was also doing half of UI in "immediate style" - i.e. render what's needed, etc. Though my stuff wasn't as developed and complex (in a "handles complex scenarios" sense) as yours... Thanks for the presentation! (goes to throw away RM gui from my codebase and code some IM gui stuff)

brianhook

Joined: 23 Feb 2005
Posts: 89
Location: Adrift in a Sea of Ennui

Posted: Tue Jul 05, 2005 4:38 pm Post subject:

NeARAZ wrote:

Great presentation, I must say. For my own stuff, lately I was also doing half of UI in "immediate style" - i.e. render what's needed, etc. Though my stuff wasn't as developed and complex (in a "handles complex scenarios" sense) as yours... Thanks for the presentation! (goes to throw away RM gui from my codebase and code some IM gui stuff)

And so it begins... =)

brianhook

Joined: 23 Feb 2005
Posts: 89
Location: Adrift in a Sea of Ennui

Posted: Tue Jul 05, 2005 4:50 pm Post subject:

Quote:

Ie., you can boil the entire input state down to a buffer of pressed characters, the mouse button transitions, and the mouse delta. That whole thing gets stuffed into the ui context, which the widgets are free to check as they please.

Is it the app's responsibility to generate characters from key up/down pairs, or does each widget do this? For example an input box or password control.

Quote:

So structuring your code to handle meaningfully large input strings is often a really bad tradeoff, because it never gets utilized but forces you to write much more complicated code.

Okay, I intuitively agree with you on this, but I'm wondering if there isn't some corner case where if you don't empty the event queue that something bad happens.

How would you track state for something like dragging an icon around? Something like in the UI context you put in "drag_start"?

It would be nice if you wrote down the Principles of Immediate Mode UIs -- I don't remember if your presentation did this or not. Off the top of my head I'd think it was like this:

Application drives UI directly, no abstraction or indirection
Application manages data
No data mirrored between widget and data
Minimal widget-state -- application tracks global UI state with a context
Application handles all event management and converts to application data

billyzelsnack

Joined: 05 Jul 2005
Posts: 14

	Posted: Tue Jul 05, 2005 5:43 pm Post subject:

A video presentation.. Brilliant!! I hope this really catches on with the gamedever community. I've talked with you many times about your IM stuff and I never quite 'got it'. I think I understand now. A few random comments.. A couple of times you mentioned that one of the advantages of IM style is that the widgets don't need to store data with them, where a RM style does. That's not true of modern guis. A modern gui typically has an abstract concept of a Renderer and/or Editor that does the work in user code. Of course, that abstraction does come at a cost of added complexity and hoop jumping. So 'Hot' is what the cursor is over right now and 'Active' is just really a hint for the other widgets to know if they need to draw themselves differently (ie, hilighted or whatever)? I have something very simular to 'Hot' in my RM gui stuff that I like it a lot. It is called 'CursorFocus' (the cursor version of a KeyFocus). It is very useful and seeing that in your system make me much more open to it. Do you have any concept for locking 'Hot', to making dragging things around easier? Other than that, you've now convinced me to sit down and try your IM gui stuff.

brianhook

Joined: 23 Feb 2005
Posts: 89
Location: Adrift in a Sea of Ennui

Posted: Tue Jul 05, 2005 5:54 pm Post subject:

billyzelsnack wrote:

A video presentation.. Brilliant!! I hope this really catches on with the gamedever community.

I'm not so sure it's a good thing, primarily because you can't skim it and it's very hard to edit later. For example, if Casey wanted to fold in these comments he'd have to redo the whole presentation.

I love his Luddite easel & marker action. Presentation -- Win, Lose, or Draw style!!

Quote:

A modern gui typically has an abstract concept of a Renderer and/or Editor that does the work in user code.

What is an example of this?

Quote:

So 'Hot' is what the cursor is over right now and 'Active' is just really a hint for the other widgets to know if they need to draw themselves differently (ie, hilighted or whatever)?

Active also controls behaviour (e.g. mouse-up).

Quote:

Do you have any concept for locking 'Hot', to making dragging things around easier? Other than that, you've now convinced me to sit down and try your IM gui stuff.

I've found that it's real hard to do the first few times, because it's "natural" to drift back to RM style stuff of creating button widgets, having it take over the event loop, etc. Try to keep everything as parameters and avoid having a "button" structure I find helps keeps me focused on "the action is my focus, not the data".

A simple demo app probably would help people learn this stuff.

billyzelsnack

Joined: 05 Jul 2005
Posts: 14

Posted: Tue Jul 05, 2005 6:08 pm Post subject:

Quote:

Not being able to skip can be nice too. I know I probably would of just skimmed his paper on it and said... What the hell are you talking about, you are crazy! (Like the last several times I've talked with him about this. Smile

)

Quote:

Quote:
A modern gui typically has an abstract concept of a Renderer and/or Editor that does the work in user code.

What is an example of this?

That type of thing is all over Swing. I dunno, maybe check out JList or JTable or something.

Quote:

Quote:
So 'Hot' is what the cursor is over right now and 'Active' is just really a hint for the other widgets to know if they need to draw themselves differently (ie, hilighted or whatever)?

Active also controls behaviour (e.g. mouse-up).

Hmm. I wonder if that is necessary. I think it might be a responsibilty that could be placed on the widget code.

Quote:

Quote:
Do you have any concept for locking 'Hot', to making dragging things around easier? Other than that, you've now convinced me to sit down and try your IM gui stuff.

I've found that it's real hard to do the first few times, because it's "natural" to drift back to RM style stuff of creating button widgets, having it take over the event loop, etc. Try to keep everything as parameters and avoid having a "button" structure I find helps keeps me focused on "the action is my focus, not the data".

I don't see how you can do dragging without a lock. Maybe that's why 'Active' mucks with the input.

NeARAZ

Joined: 05 Jul 2005
Posts: 51
Location: Kaunas, Lithuania

Posted: Tue Jul 05, 2005 6:19 pm Post subject:

billyzelsnack wrote:

Quote:

Quote:
A modern gui typically has an abstract concept of a Renderer and/or Editor that does the work in user code.

What is an example of this?

That type of thing is all over Swing. I dunno, maybe check out JList or JTable or something.

Hey, don't get me started on Swing! Yeah, it's supposed to be "nice and whatever OOP design", and in a sense it is. But it's nothing even relatively close to "simple" or "elegant", the latter being very true when you actually are trying to deal with it's renderers and editors. Column sorted, filtered JTable with some colored cells, some bold fonts and some spinner controls for editing? I don't want to remember that pile of crap anymore Smile

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

Posted: Tue Jul 05, 2005 6:51 pm Post subject:

billyzelsnack wrote:

A couple of times you mentioned that one of the advantages of IM style is that the widgets don't need to store data with them, where a RM style does. That's not true of modern guis. A modern gui typically has an abstract concept of a Renderer and/or Editor that does the work in user code. Of course, that abstraction does come at a cost of added complexity and hoop jumping.

I'm not sure I follow you. Either you have to code the interactions yourself, or a standard GUI forces you to create a widget. There's no two ways around that in any GUI library I've ever seen. Sure, you can draw all your own buttons and code all your own interactions or something, but then you're not using the library anymore. For example, suppose you wanted a scrolling window with 1000 trackbars (or comboboxes, or whatever). In IMGUI you have a for loop that calls DoTrackbar. In a regular GUI, how do you do it without actually creating 1000 widgets that store at least the fact that they exist?

billyzelsnack wrote:

So 'Hot' is what the cursor is over right now and 'Active' is just really a hint for the other widgets to know if they need to draw themselves differently (ie, hilighted or whatever)? I have something very simular to 'Hot' in my RM gui stuff that I like it a lot. It is called 'CursorFocus' (the cursor version of a KeyFocus). It is very useful and seeing that in your system make me much more open to it.

For me, hot is who was the person who had the mouse focus last frame, and active is the person who's being interacted with. So elements typically ask the ui context "am I hot?" and it responds true if they are and there is no active item, or if they are hot and they are the active item. Then they can draw themselves, or respond to the initiation of an interaction. They also ask "am I active?" which just compares their ID vs. the active ID and returns that. If a widget is active, it should be responding to UI events in whatever way it's meant to work... pop up a menu, slide a slider, whatever. It may or may not care if it's hot anymore (once the user begins interaction, some UI elements interact with the user until mouse-up, regardless of whether or not the cursor is over them, for example).

billyzelsnack wrote:

Do you have any concept for locking 'Hot', to making dragging things around easier?

If you were dragging something, it would be considered active. Active is the thing that gets "locked", and it is in fact locked by default (you're active until you say you're not). Hot is always transient.

- Casey

casey
Site Admin

Joined: 18 Dec 2004
Posts: 1768
Location: Seattle

Posted: Tue Jul 05, 2005 7:02 pm Post subject:

brianhook wrote:

Is it the app's responsibility to generate characters from key up/down pairs, or does each widget do this? For example an input box or password control.

It is always the widget's responsibility, particularly because most of the time the widget needs to know more than just keypresses. For example, my edit box code does key repeat by remembering if a key was down and letting it insert every n seconds until it is released. So typically you want the IMGUI library to accept pretty raw event streams, and let it clean them out how it wishes, so sometimes there can be "aggregated" information that most widgets use, but other widgets can look at the full stream when they need it.

brianhook wrote:

How would you track state for something like dragging an icon around? Something like in the UI context you put in "drag_start"?

It depends on the situation, but you probably wouldn't have to do anything at all. If you are a widget that supports dragging, then you know that if you're active, you're being dragged (or clicked). When the user let's go, you know you've been "dropped" on the hot item (which may be you, in which case it's a click). So the only thing you'd really need to add to your system is a "this was dropped on you" variable pair that the widget can now set for the widget that received the drop to pick up on the next frame (ie., with a "if(IWasDroppedOn)" block). Presumably the dropped-on value is returned to the application as part of the drop-accepting widget's return value (or optional out parameter).

brianhook wrote:

It would be nice if you wrote down the Principles of Immediate Mode UIs

I did, in the video - there's only one: the interface to the library is immediate-mode Smile

That's really it. The other items you listed on that list may or may not be true depending on the library. Minimal widget state is not actually a property of IMGUI. It happens to be a nice byproduct for most libraries written in this style, I think, but it is not a necessity.

The power of IMGUI is simply that the interface is immediate mode. I can't stress that enough. I can envision (and in fact, have built) some very complex things in more advanced IMGUI stylings that actually build large, frame-to-frame coherent backing structures that the app knows nothing about. This is still IMGUI, and in fact, I think there are massively powerful things you can do here that you could not do with either the other two library styles (regular RMGUI, or "bare bones" barely anything retained per frame GUI).

- Casey

Display posts from previous:

	Molly Rocket Forum Index -> IMGUI	All times are GMT Goto page 1, 2, 3, 4 Next
Page 1 of 4

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum