Ok, updating (finally) with a fix that worked for me.
I've got a buddy who is a former Triumph technician who helped me out. He had seen this exact same issue on a few different bikes, and in all cases it was related to the SAI system. Basically he told me that if one or more of the SAI hoses gets blocked up somehow, you get this problem. My solution was to do a SAI delete, just removed the system entirely, installed the block-off plates from British Customs, and uploaded a no-SAI map appropriate for my setup (the Arrow 3-1 since I'm running a Leo Vince 3-1 pipe).
The problem is completely solved, as is the weird hesitation I was feeling around 3k-4k rpm.
I hope that helps someone else. Check your SAI system to make sure it's actually flowing air when it should, or just do a SAI delete and upload an appropriate map.
2005 Speed Triple
2004 Speed Four