Experiment: Mesh-based character interaction

The current animation workflow in Blender can be a bit bogged down by bone shapes being hard to find, hard to select, disappearing inside the mesh, etc. So it's no wonder that one of the more frequent requests I've been hearing from my animators during the production of Sprite Fright was a bone picker.

The biggest unique benefit to such 2D pickers is that you get a strong sense of which controls were used and to what extent to achieve the current pose you have in front of you. But they also have their bagge:

• Take up tons of screen space
• Require tons of mouse movement
• A 2D interface to interact with a 3D character limits rig design

So, although a 2D picker made its way onto my TODO list, I decided to first focus on something that I think would be easier to implement and solve a bigger set of problems: Mapping controls to interactable areas of the mesh itself.

In Blender 2.80, we got a Python Gizmo API, but if the documentation is anything to go by, it was never quite finished.

My goals were to see if I can implement what I need in Python, and if not, set clear and targeted development goals so I can ask for help from a C developer. This would also mean agreeing on a design with the Animation Module, so that a developer will only have to worry about the code.

I started by diving into the code of add-ons that do similar things:

• PizMo by pKrime
• LilyGizmos by Élie Michel

So many thanks to them. A couple weeks of head scratching, copy pasting, re-writing, UX designing and optimizing later, I created an add-on that brings the idea of mesh-based bone interaction to the fullest extent that I think is possible with the current API:

It looks great, right? Well, it's actually pretty bad:

• The pre-selection highlighting, and therefore the selection, is not depth-aware, so if you have overlapping gizmos, the one in the back is likely to be hovered.
• I had to create extra meshes that are slightly offset from the original, in order to avoid Z-fighting. This workaround however, adds a crazy maintenance overhead.
• The mouse movement interaction threshold user preference is not being accounted for, so if you have auto-keying enabled, you can't select a bone without also keyframing it.
• If you change the mesh or the weights, you need to manually refresh the gizmo shapes.

And those are only the bigger problems! But finding these problems was exactly the purpose of this whole exercise, so I'm still considering it a great success. This is a start that I'm sure we can build on.

You can find more details and a link to the add-on itself in this task in case you want to try it out or give testing and feedback.

I also presented the add-on (along with its flaws) in an Animation Module meeting and asked for feedback. Some pain points were pointed out:

• Gizmos are stored on a per-bone basis, so they cannot be directly hooked up to a custom property. You would have to drive your custom property with a bone transform.
• Since only one gizmo is stored per bone, you can't have two separate gizmos for translation and rotation of the same bone. You would have to select the bone, then execute your preferred transformation, or have two bones parented to each other; A parent for translation, and a child for rotation.

For now, I find these compromises acceptable to keep a clean and intuitive design: One gizmo per bone means that as far as the user is concerned, the gizmo doesn't feel like a separate entity from the bone. There is only the bone, and that bone can be represented by a custom shape, a gizmo, or both. I find it super straight forward.

So, what's next? It's now up to a lovely C developer to either volunteer or be assigned to this project, so they can assess the feasibility of the necessary changes, and also give their input on the design from a more technical perspective.