The word habit is typically used for personal actions - things that you do frequently and repeatedly without even thinking about them:
I have the habit of falling asleep in front of the TV.
My brother has the bad habit of biting his nails.
Diana is trying to develop healthy eating habits.
I don't trust him. He's a habitual liar.
(the adjective form is "habitual")
The word custom is used more for cultural or religious traditions.
People in this country have the custom of removing their shoes before entering the house.
According to religious custom, wearing white symbolizes purity.
It's customary to give people gifts on their birthdays.
(the adjective form is "customary")
The word custom also has another meaning - as an adjective, it is the short form of customized - meaning something is changed to make it more personal, individual, or unique:
I got a custom cell phone case with a picture of my dog on it.