We don't need a teammate (call them X) to have access to the Inbox, we just need him to belong to a team.
Here are the replication steps
1. Assign teammate X an Inbox seat license
2. Add teammate X to the Sales team successfully
3. However as soon as I remove the Inbox seat from teammate X, he disappears from the Sales team.