100% for this, as people have said you can get AI to answer anyway, as the “guardrails” are really just part of the closed harness and can only do so much as like with everything with a LLM its just a matter of probability that it will ignore it at some point, regardless better a have than a have not ( although there is an argument this could make people think its safer i guess).

To REALLY solve this, you would have to edit the training data to take this stuff out (the reason they don’t is because for the big models, they just scrap the Internet for data to save cost).

But the real answer is we just need to make people aware of how they work they dont think or reason or even really know anything; they just try to guess what answer a human would want.

If you want to do real work with them you need you need be able to train a model on your own data and control the harness and weights (which is why i think open source will win out) even then am pretty sure there main use case is roleplay (for like RPGs) and sex bots as there mistakes dont matter here. ( and yeah there is an issue with people using this stuff for replacement friends so even then not great).