Solutions:Make sure you are using compatible checkpoint and controlnet, they have to be both for SD1.5 or SDXL.