Human memory prioritizes the storage and recall of information that is emotionally-arousing and/or important in a process known as value-directed memory. When experiencing a stream of information (e.g. conversation, book, lecture, etc.), the individual makes conscious and subconscious value assessments of the incoming information and uses this as a metric to determine what to remember. In order to improve automatic recall of memory, previous memory augmentation systems have sensed users' physiological state to determine which sensory media should be prioritized. Here, we propose to prioritize memories using the affective state of individuals that the user is interacting with. Thereby, the proposed wearable Affective Memory Augmentation system uses affective information from the user's social companions in order to facilitate value-directed memory.