Good question, do you depend on a system that can't be proved to be error free - particularly if it's based on current LLM technology?

Personally I think no, we'll have to get a bit better at it first - maybe in the way when we developed mathematically provable compilation for critical software systems perhaps.