Should I let go of our relationship?

The best thing you can do is talk to him. Tell him how you feel and what he want to happen. Then you can decide if you want to let him go or keep him.