Battle of the egos?
At this point it's just the fact that Cas really doesn't like Sloane anymore. He's not trying to get his opinion across to him anymore because it really doesn't matter. He's done dealing with it.
He also told him to move because... you know, if he doesn't he's going to die terribly.