Your prompt worked in March. You tried it forty times, In June it started returning You did not pin the version. You pointed at a name This is the part of building It is not an API The contract is loose. The shape of the output The behaviou...
Your prompt worked in March.
You tried it forty times,
tuned the wording,
got the output shape you wanted,
wrote a test,
and shipped it.
In June it started returning
a slightly different answer,
and nobody changed a line.
You did not pin the version.
You pointed at a name
that means latest,
and latest moved.
This is the part of building
on someone else's model
that teams learn late.
It is not an API
in the way a payments API is one.
The contract is loose.
The shape of the output
is a suggestion.
The behaviour has a release schedule
that is not yours,
and a deprecation date
that is not negotiable.
So treat the model id
as a dependency version,
because that is what it is.
Pin it explicitly.
Write it in configuration,
not in a string in the middle
of a function.
Log which one answered,
next to the input and the output,
so that when the complaint arrives
in three weeks
you can tell whether the model changed
or the customer did.
Then build the boring thing
you have been avoiding.
Twenty saved inputs
with the answers you consider correct.
Run them when you change the prompt.
Run them when you change the version.
Run them on a schedule,
because a provider can adjust
something underneath you
without a version bump.
It does not have to be clever.
A file of cases
and a script that shows the diffs
will catch more real regressions
than any evaluation framework
you never finish setting up.
And plan for the upgrade,
because the pin is temporary.
The version you froze
will be retired,
usually with less notice
than you would like,
and the migration will land
in a week that was already full.
A prompt is code.
It has a version,
it has tests,
it has a deprecation date,
and it will break silently
if you pretend otherwise.
– Serguey Asael Shinder