How do I set limits before filming nudes or porn?
Use normal words.
"Nudity is fine" can mean completely different things to different people. So can erotic, explicit, softcore, hardcore, sensual and almost every other label people use because it sounds easier than describing the actual thing.
If the distinction matters, talk about the actual thing.
Will the face be visible? Full nudity? Genital visibility? Kissing? Touching? Oral sex? Intercourse? Close-ups? Another performer? Photography as well as video?
You do not need a fifty-page checklist. You need both people imagining the same project.
A clear no is useful. It lets the producer decide whether the concept actually fits the performer. If the project only works when somebody does something she does not want to do, then it is simply the wrong project.
A maybe can be useful too, as long as it remains a maybe. Sometimes you genuinely do not know how something will feel until you are there. Fine. But a producer should not build the entire production around the assumption that uncertainty will conveniently turn into yes later.
There are also boundaries that have nothing to do with sexual acts. Face visibility, public name, smoking, alcohol, public locations, dialogue, hair, makeup, another performer, which stills can be used for promotion, where the finished material is published. Those can matter just as much.
Oddly enough, clear boundaries normally make the creative side easier. Once everybody knows where the walls are, there is much more room to improvise inside them without wondering whether somebody secretly expects something else.
That is how I prefer to work. If the concept and the person's limits do not fit, it is a casting mismatch. There is nothing creatively interesting about arguing somebody into a different answer.
You do not need to make the conversation clinical either. "I'm fine with full nudity and oral, I don't want anal, and I want to discuss any cum-shot setup before we book" is much more useful than a polished paragraph about personal boundaries. The producer now knows what the actual production can be built around.
It also helps to separate a hard no from something you are simply inexperienced with. "I have never done that and I don't know yet" is information. It should not be treated as a hidden yes, but it is different from "I do not want that". Good casting is partly about understanding those differences without turning them into a negotiation game.