CIA agent Kern gives Brown a napkin with the Chinese character 'ke' on it, and tells Brown he will face a choice when he hears the words "baltic amber." The next scene is 3 months later, and the napkin shows no visible wear from being carried all that time.
Brown is still carrying the napkin 3 months after meeting Kern, despite a) recognizing him immediately, and b) remembering the identifier "baltic amber." Kern never gave him an instruction to keep the napkin; it was probably a reminder of one of the meanings of the Chinese character 'ke,' namely 'commitment.'
Blane makes it clear to Brown that he does not trust Kern given a history of playing for whichever side pays the most, so why would he allow Kern close enough to him to take his passport?