I know people have looked at inferring "users never make that particular left turn, it must be disallowed" or "users miss it a lot, there must be /something/ wrong with it", but I don't know how it worked out.

Driving a car, on an unfamiliar route, and on top of that managing where the directions deviate from reality, all takes so much of the user's brain that I'd be pretty reluctant to ask them to talk to their car too. The idea of marking it for later commentary is interesting, I wonder if you could do that without the realtime mark? When you're home, if there's a "My Trips Today" view that shows: on this trip you left the route we thought -- open the trip and it brings up the specific step of deviation -- and you could indicate "yeah, those directions didn't work" / "directions would work, I just had my own reasons", plus add more detail if you like.