1
$\begingroup$

I have learned a bit recently about prompt strategies. For example, there was a paper about how just by saying “Let’s think step by step” can increase answer quality by like 40%. I have also come to appreciate that models like GPT4 sometimes actually do better with short prompts than long ones. One paper I think found that LLMs have “recency bias”, so if you give sample text, then an instruction, it does better than instruction, then sample text, because in that case, it pays less attention to the instruction.

I have struggled a lot with basically zero or few shot prompting GPT-4 to give me highly concrete information and/or a specifically actionable plan or set of steps.

To give an example, it very often gives you very vague, general advice like,

“If you’re looking for a job, trying looking around on online job websites, or contacting a local employment agency”.

If I give it more specific information, and try really quite hard to get it to tell me something way more specific, at best it might add in some very common sites and places, like,

“First, think of what jobs you might like, based on your skills. Then, search for those keywords in a job listing site, like Monster.com or Indeed. Also, consider contacting the local municipal job center, [City Job Center’s name, address, phone number.”

It has been quite hard for me to try to get GPT-4 to be way, way more like a hardcore data-crawling machine, so to speak. It would be really nice to know if there was a special trick that has been discovered - just like the surprising efficacy of 5 words, like “Let’s think step by step” - where you basically tell it that you only want specifics, and you don’t want just like, the top three - ideally, you want it to figure out every single known job website or app on the internet, every single known job center in your county, every single employment agency and recruitment firm too, all of their names, links to their webpages, etc. Given that some GPT-4 systems are able to search the web, the requested task could make it clear that the model is free to use any information it already possesses internally; search amply and procedurally on the web to find more information that it needs; but furthermore, if it does not know, that is fine, but in that case, it should provide further, actionable steps for the human to take, like a specific place they could ask, or specific google keywords they recommend searching for.

Similar to information, I find it difficult to get GPT-4 to make a set of instructions that totally eliminates as much open-endedness or choice as possible - in which every single conceivable way of breaking down a task into tiny actions is present. Instead of saying, “make accounts on glassdoor and LinkedIn.com. Register with your email. Fill out your profile with relevant information”, I want to understand how to get it to say something like, “Ok, your name is ___. What’s your email address? And main skills? Got it. Ok, let’s start with LinkedIn because _____ (intelligent justification, even statistically backed, for why it has a high success rate). Based on this data analysis I found / made, it turns out there’s very high demand for this very specific job title right now, on LinkedIn: ___. And I can easily imitate some common resumes of people in those fields. So, here is the text of your resume: ___. Download and save that as a Word document. Now click this link here: _____, and click “apply” - that job is nearby you and it’s probabilistically likely you may get it. Submit it. Next, check your email once every 3 hours, because ____”.

The question here is not so much wanting a true AGI / AutoGPT, but just trying to know powerful simple keywords, prompts, commands, etc., that just help the model understand the difference between specific and vague. The word “specific” has not helped me at all, unfortunately, unless I go through some iterative prompting / fine-tuning, but it’s not convenient enough for daily use.

Any research-backed findings on this?

Thanks.

$\endgroup$
1
  • 2
    $\begingroup$ Interesting question! 2 things. Please try to make your title as specific as possible. It feels very broad at the moment. Second, if you say 'you've read a paper', please reference so that we know what you are talking about. Lots of us would like to read them as well if we havent done so :) It helps a lot to put your question into the right context $\endgroup$ May 4, 2023 at 13:02

1 Answer 1

1
$\begingroup$

It may not be a complete answer to your question as it is quite broad, but I would offer three approaches for you to have a look at:

Fine-tuning via JSON: Ranedeer.

Mr. Ranedeer AI Tutor is a customizable GPT-4-based learning tool offering personalized learning experiences for diverse users. It requires a ChatGPT Plus Subscription with GPT-4 or above models and is not recommended for GPT-3.5 or GPT-4 API. The tutor allows users to adjust knowledge depth, learning styles, communication types, tone styles, and reasoning frameworks. To use, visit ChatGPT, select GPT-4 or above, and input the Mr_Ranedeer.json content into the prompt. Mr. Ranedeer supports various commands for feedback, tests, configuration updates, and lesson planning

Logit Processor: Clownfish

In the text, the challenges and limitations of using ChatGPT without humans in the loop is discussed, as models can hallucinate actions and behave dangerously. To address this, the author proposes "ControLogits," an algorithmic manipulation of probabilities to guide the model in generating text that fits a specific JSON schema. The Clownfish implementation of ControLogits can be used to validate and direct token generation according to a schema. However, this approach is not directly applicable to GPT-4 due to limitations in exposed logits.

JSON Generator: Jsonformer

Generating structured JSON from language models is a challenging task, as it requires the JSON to be syntactically correct and conform to a specific schema. Current approaches are often brittle and error-prone. Jsonformer is a new, more efficient approach that wraps around HuggingFace models. It fills in fixed tokens during generation and only delegates content token generation to the language model. Jsonformer supports a subset of JSON Schema, including number, boolean, string, array, and object types. It ensures bulletproof JSON generation, enhances efficiency, and is flexible and extendable due to its compatibility with any model that supports the HuggingFace interface.

$\endgroup$
5
  • $\begingroup$ Thanks, those are some really impressive prompts I had not encountered, so I’m definitely glad I asked. Although, the approach I had taken today, after asking, was utterly different - and honestly, really productive and interesting. I had an extended conversation with GPT-4, detailing my ideas, getting its own. I’ll post it as an answer here soon. Basically, we tried to focus on a purely logical definition of what “specific” even means. GPT-4’s solution really surprised me. It basically, under my guidance, devised an NLP metric - TF-IDF + 2 others - we told GPT to attain a score on it & it did $\endgroup$ May 4, 2023 at 23:55
  • $\begingroup$ I also explored some new ideas for myself. 1. Using multi agent systems basically as “AI-native” “adversarial networks” - planning on coding it in LangChain asap. I’m sure it’s already being done. But trying to pit a few GPTs, with roles, against each other, to iteratively refine a prompt, which is useful zero-shot. I was thinking of calling powerfully optimized, and very short, zero-shot prompts, “atomic bombs” (if they exist), as you discover a special trigger which is very effective relative to the number of words. $\endgroup$ May 5, 2023 at 0:01
  • $\begingroup$ I am also going to explore if it’s possible to compress iterative, relative prompting into a single prompt, somehow. I analyzed how the difficulty with words like “specific” is they are relative, or as I call them, affine - they only have meaning in comparison to something else. Although using this “floor jack” technique works great, it doesn’t fulfill the desire of zero shot, and brief. I’ll be exploring if GPT-4 itself could somehow “compact” the effect of those iterations into a single prompt, somehow. The idea fascinates me. $\endgroup$ May 5, 2023 at 0:07
  • $\begingroup$ However, by the end, I was able to conclude on some much simpler ways, which were honestly, what I was looking for. One, is just to give it the simplest single example of the difference between vague and specific. I chose an arbitrary topic - a description of myself. I gave it three examples, from vague to specific. I told it I wanted it to be more specific than the third one. And it worked quite well. I also considered even simpler ways - just state the number of steps you want - subdividing, chopping up the same thing into any number of parts - “make me a sandwich, in 100 steps”. $\endgroup$ May 5, 2023 at 0:09
  • $\begingroup$ And there were also some interesting ideas about linguistics - like, asking for determined nouns instead of general classes - if you say bicycle, I require that you state which one, in the real world. That idea also intrigued me. $\endgroup$ May 5, 2023 at 0:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .