So the person who first used this phrase probably thought they were doing this:
But hang on a mo. There's a description for the belief system that involves believing in Jesus. It's.... what's the word? Oh yes. Christianity. You can't "give up" on Christianity just by deciding you don't like the definition of the word. What you're really saying is you're giving up all that judgemental / doctrinal / stuffy / traditional stuff, to follow Jesus. Leaving all those judgemental / doctrinal / stuffy / traditional people behind. But that still makes you a Christian, sorry. It's what the word means.
What I suspect the expression really means is this:
Or, being more charitable perhaps: