Why do we never forget swimming, riding, driving even after years while our memory deceives us in other walks of life?

Al this falls under procedural memory, which includes ‘muscle memory’ or ‘motor learning’. The cerebellum of our brain is exceptional at encoding sequences of muscle movements into long-term memory storage. Our procedural memory is so strong that people even suffering from dementia and amnesia can often retain such memories. They do not forget how to play a musical instrument or riding a bike without any conscious efforts.

As far as driving is concerned, this is practice-based skill and even if you do not do it for long and then try it after years, you may not be as good as you used to be at the moment you left it but you will quickly come back and intuitively.