Nils Durner's Blog Ahas, Breadcrumbs, Coding Epiphanies

Bring your own prompts

There is a recent spawn of tools “powered by AI”, “powered by ChatGPT”, … e.g. the “Kubectl OpenAI plugin”. Such third party clients may limit, or outright sabotage, you in certain ways however:

  • often based on the older, more primitive language model endpoints (davinci-003 in this case)
  • unaware of breaking changes in the target subject
  • same for new features I don’t know about the velocity of Kubernetes development nowadays, but with certain Python packages, I believe, one colleague and me have had broken code generated. Two properties of GPT bite here:
  • training data just until 2021 (eternity!)
  • no real notion of release version numbers/dates (at least used to) As a consequence, you may be generating legacy today or results may simply not work, potentially unnoticed initially As a potential remedy, I have toyed with custom contexts in GPT-4: first the original answer, with default parameters:

    ?: What’s the record duration of nuclear fusion? !: (reference to some Korean experiment in 2018) Primed with a short note in German about today’s breakthrough in Greifswald: ! As of my last update on March 21, 2023, the Wendelstein 7-X experimental fusion reactor in Germany set a record […] It would be interesting to see if advances in IT, perhaps in the form of Knowledge Base articles, may similarly inform up-to-date responses. However, clients like the one you mention neither allow users to do this by themselves, nor do the clients cater to that.

One may argue that this “is part of the hallucination-kind of issues with GPT one needs to deal with”. That’s very true for ChatGPT, but with the GPT-4 API, it largely comes down to parameter choice and prompt engineering, in my experience.