# CAPTCHA based on text comprehension and random tokens

I developed a novel type of CAPTCHA based on text comprehension and random tokens. Given a task Pick the first pair of adjacent letters and a random token 8NBA596V, the user has to provide the solution NB. It offers basic protection and an attacker can solve individual tasks with specific effort. I am curious, whether contemporary AI can solve it generically?

There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task – essentially a predicate in a tiny universe of discourse – and then applying it to the random token.

• Have you tried feeding this task into any of the available GPT models to see how it answers? Such as GPT-2 or GPT-J-6B Sep 3, 2021 at 9:50
• I tried it on GPT-J-6B and it repeatedly failed this task. They can sometimes do previously unknown tasks just based on a description of the task, but apparently, not this one. Sep 3, 2021 at 10:02

(Assuming English.) In your specific example, there would be $$26^2$$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $$26 * 26 * 7 = 4732$$ possible combinations. My intuition is that the key space is too small.
It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $$(26 + 10)^6 = 2,176,782,336$$ possibilities. That keyspace is $$460,013$$ times larger than your system with the longer string. It gets worse though. An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. It would not be a guaranteed win for the attacker. But it would be guessing at a higher probability than $$\frac{1}{4732}$$.