When Victoria looks down upon neighbor Emily's house, the house seems near. But when Emily looks up at the mansion, it seems further off in the distance.
Given how sharp and intelligent Frank is, it is unlikely he didn't recognize Emily from the party 10 years back. He interacted with her for quite some time and even shooed her away from the pool house while he murdered Roger Halstead. Frank is head of security, so it should have occurred to him that Emily looks just like the waitress at the Grayson's party some years back.