feat: Improve connection removal handling by skipping connections not managed by the controller