Vertex Painter

Summary

A vertex painter that was used in the last two group projects during my time at The Game Assembly. It can paint on any mesh that has been loaded into the scene and uses an emissive decal as a 'paint brush' to highlight the area that will be painted.
Only selected objects can be painted. This is done for two reasons:

The vertex color data for all mesh instances is stored inside our custom binary model format.

Premise

During the fall of my second year at TGA, while we were working on our 5th game, our Artists requested the ability to paint on vertices. At the time I, nor any of the other programmers, had the time to implement such a large, optional feature, as we were busy enough implementing the required ones. Once we were done with the 5th project, just before going on Christmas break, and the 6th one had started; I decided that this was something I was willing to implement during the holidays, as it sounded pretty cool and would make for a neat addition to my future portfolio.
 I had already worked up an idea of how to implement it and so I requested a model, with matching textures, from our artists and got to work.

On the rest of this page I will go through how it currently works and some of its intricacies, but I wanted to mention a lesson that I learned before it reached that point. 

My first iteration worked great and initially I was happy with it. Then, after winter break, I showed the artists how it worked and quickly realized that I had lost the ability to see the forest for the trees as I was too concentrated on how to make it work, that I had forgotten to think, and ask, about how it was meant to be used. The thing that had slipped my mind was that it had to be able to paint on multiple models at the same time and at that point it could only paint on one at the time! It wasn't practical for them to use it this way and would most likely not be used at all unless I fixed it. In order to not waste all the time spent on it, I scrambled to fix the issue. Unfortunately the holidays were over and I had other assignments coming in, so I had to fix it quickly. The main logic was easy enough to convert and I thought it might not be so bad. Then I got started on fixing the ability to save what had been painted and it dawned on me. What I had thought was a clever way to store the vertex color data meant that, because of version control and that the data for all instances was inside the binary model file, only one person could use it at a time. I knew what I had to do to fix it in theory, but it was also a lot of work and because of limited time, and the fact that it was at least usable now, it meant that I had to leave it there.

Implementation

You start in selection-mode which is just using the editor normally, but when you have at least one item selected in a scene; you can try to activate paint mode inside the vertex painter UI interface. Clicking the button will verify and prepare the mesh data for painting. The preparation consists of copying the necessary data into more cache-friendly structs, pre-calculating the mesh faces + face normal for intersection tests, and transforming the vertices into world space to avoid doing that every frame. 

Once in paint-mode; you can no longer select / deselect items, or change properties. This is both because the same key binds are used to paint as it is to select items, and to prevent the need to handle all cases involving changing the model of any item that has been painted on. There is also a brush created when hovering any selected item. The brush is actually a decal with an emissive color, and three line drawn circles that form a sphere that shows the bounds of the brush. The color of the circles match the RGB layers that you've got selected.