Yes, this episode has many plot holes and reviewers are rightly frustrated that little detail is offered about Kellin's biological processes that render others' memories unable to capture her. But I don't think it is fair to ask for less focus on the love story.
Chakotay has gotten the short end of the stick with personality-expanding episodes, or maybe it just feels that way because of how stoic he is. Because of his unemotionality, the Vulcan Tuvok actually gets a lot of attention surrounding his emotional reactions to events. Chakotay has certainly been the star of multiple episodes by this point late in season 4, but he has been fading into the background, seeming more impassive even than the Vulcan. He is a stalwart piece of furniture on the bridge.
Chakotay's major traits are his loyalty and strong work ethic, so much so that the only times viewers see him passionate are when he cares deeply about solving an engineering or diplomatic problem his own way. When we discover he had a fling with Seska, it is only relevant because his work ethic and loyalty cause him to want to protect Voyager all by himself.
If anything, "Unforgettable" did not offer enough scenes of this man feeling feelings.