He says that sometimes i push him away.?

Sometimes a person doesnt want to be touched. I get it. Why are people thinking it must mean something like depression. Not everyone is super touchy feely. You said it hurts you when he feels bad and you dont WANT to reject him, you just need a little space sometimes. Sounds acceptable to me. Maybe just talk to him and let him know its not about him when you feel that way, its about you. As long as you arent doing it all the time he should respect you needing space from time to time.