I think that is why the term 'non-theist' better describes your position. 'A-theist' suggests that the person stands outside the whole debate, which leads to the assumption you describe.