Drifter bristles defensively when your Ghost mentions that you experienced a vision from the Emissary of the Nine. He, too, is experiencing visions, and he wants you to keep your distance.
Whether you respect his wishes is, of course, entirely up to you.