So I was trying out playing with controller for a bit and at some point I turned it off to go back to playing with mouse and keyboard.
Normally the game does a good job of switching between keyboard & mouse UI/HUD and controller UI/HUD automatically, but I think something borked after I turned off the controller while the last input was from the controller, part of the UI was stuck in the controller version and could not be turned back.
Went away after a reboot.