Understanding by context can help you get the general idea but there are times when you really have to understand every thing they say.
Native Speaker speaking fast: So...I'm going to take the car and then come back at around 6:30PM, maybe 7PM. But friend A has the apartment key so you have to be back here by 6:30PM at the latest. What do you think?
In this situation it is important that you know that you have to be at the house at 6:30PM; the native speaker changed their mind from the original 6:30 or 7:00PM return to the updated 6:30. But if you just catch *most* of the words, you may not understand that. And you might not realize that you have to be back or else the native speaker can't get into the apartment.