Just wanted to help out others who may have this problem. I am developing a custom people picker and am testing changing the group owner and I received the error
"A domain group cannot be the owner of a group."
But I was picking people, not groups so the error was confusing. I finally realized the claim I was using as the user identity claim in the claim provider was NOT the same claim that the SPTrustedTokenIssuer uses as its identity claim. These two things must be the same (obviously, it was a fat finger).
Please mark my response as an answer if appropriate.
Learn.SharePoint.com