I think the colloquial understanding of Gödel's incompleteness theorems allows them to be too broadly applied. Gödel's second incompleteness regards the consistency of a formal system, which is a technical concept of formal systems that means the system cannot prove every formula. It is commonly framed as a system not being able to prove both a formula and its negation (e.g. $2+2=4$ and $2+2 \neq 4$), since many logical systems allow anything to be proven from a contradiction.
The second incompleteness theorem states that if a consistent formal system is expressive enough to encode basic arithmetic (Peano arithmetic), then that system cannot prove its own consistency. This implies that we must use a stronger system B to prove the consistency of A. The system needs to be able to represent arithmetic because that is what is used to define the representability conditions of Gödel's proof that allowed him to formally construct the self-referential formulae central to the incompleteness theorems.
Here I diverge with my own opinion on this, I feel the concept of consistency in formal systems has no obvious bearing on the limits of artificial intelligence. An intelligent agent need not know anything about formal consistency to reach its level of intelligence -- the vast majority of humans have never encountered this concept, and yet they are still intelligent. Even many mathematicians don't give it a second thought unless they are in the trenches of mathematical logic. One would have to take an overly narrow view of artificial intelligence to allow Gödel's second incompleteness to serve as a limitation to it.
I caution against the popular informal restatements of Gödel's incompleteness theorems. These theorems were undoubtedly earth-shattering in the study of foundational mathematics and still have grand implications today, but projecting those results too far away from their rigorous origins is going to lead to many stray conclusions.
