CAPTCHA based on text comprehension and random tokens

Question

I developed a novel type of CAPTCHA based on text comprehension and random tokens. Given a task Pick the first pair of adjacent letters and a random token 8NBA596V, the user has to provide the solution NB. It offers basic protection and an attacker can solve individual tasks with specific effort. I am curious, whether contemporary AI can solve it generically?

You can access more example tasks here: https://www.topincs.com/manual/captcha

There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task – essentially a predicate in a tiny universe of discourse – and then applying it to the random token.

Answer 1

(Assuming English.) In your specific example, there would be $26^2$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $26 * 26 * 7 = 4732$ possible combinations. My intuition is that the key space is too small.

It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $(26 + 10)^6 = 2,176,782,336$ possibilities. That keyspace is $460,013$ times larger than your system with the longer string. It gets worse though. An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. It would not be a guaranteed win for the attacker. But it would be guessing at a higher probability than $\frac{1}{4732}$.

Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.

I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just "Pick the first pair of adjacent letters." But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.