While on the phone, Andy mentions that Raleigh is over 100 miles from Mayberry. In season 3 "Andy's Rich Girlfriend," Barney mentions that "The Capital" (Raleigh) is 55 miles from Mayberry.
It would be unlikely that lawyer Lee Drake would drive out from Raleigh early in the morning, a distance of 100 miles, to follow up with Andy on some details of their work from the previous day without calling first to make an appointment.