Intelligence, Fast and Slow
Psychologist Daniel Kahneman, whose study of human biases laid the foundation for modern behavioral economics, in his book Thinking, Fast and Slow outlines a dichotomy: human thinking is performed in two different ways. Fast and instinctive thinking of "System 1" and slower, deliberate thinking of "System 2."
A question one would employ type 1 thinking to find a solution to is "what is 6x3?" A question suited for type 2 thinking is "what is 6x33?" The mechanism used by the human brain to arrive at those is fundamentally, substantially different. When asked about an explanation, the asked person would produce similar step-by-step arithmetical solutions to both. However, in the type 1 thinking, the process explained did not occur. No math-savvy person adds three sixes when faced with such a simple calculation. This, or any other valid breakdown, will be made up after-the-fact, as opposed to "6x33" question, where the person asked how they concluded on the result would likely answer truthfully, giving an account of arithmetics actually performed in their heads.
A person prompted with "6x3" does not add three numbers so fast that they don't even notice it. They do not make the calculation at all, even unconsciously. The operation of producing the answer is certainly more similar to information retrieval than calculation. But rather than accessing some storage space for results of trivial arithmetic, the answer is encoded within neural connections. It is connectionist and behavioral, rather than symbolic and cognitive. By benefit of constant reuse (behavioral aspect), this knowledge is programmed into weights of the biological neural network: the human brain (connectionist aspect). The only thing that happens in the brain confronted with "2x2" is neurons firing in a sequence producing 4. Weights are sub-symbolic, and there is no explicit mathematical representation, any recognizable intermediate steps.
In type 2 thinking, one uses one's brain to emulate symbolic information-processing, just as one emulates a virtual machine on one's desktop. One creates algorithms, plans, and puts effort into imagining numbers, patterns, and structures. An imaginary canvas for equations appears; it is an act of conjuring the symbolic representation. In the posthumanism of Stiegler's Technics and Time, the 'faculty of symbolization' is an interaction of the human mind with its environment—the externalization of the central nervous system. In type 2 thinking, 'one thinks, one invents, one makes discoveries—but they have acquired, unnoticed, a displaced, "symbolic" meaning.'
And, biologically, the cerebral cortex is just an augmentation of the monkey brain the human has—the outermost layer, subjugated by a primitive ape to build airplanes and write poetry. This gives transhumanists hope for a tertiary level: a silicon superintelligence attached to the cortex, which would work for the cortex just as the cortex works for the brain's inner parts. It's a promise of superhuman intelligence without losing what it means to be a human.
Type 2 thinking is what was used by top-down AIs of the XX century. Pre-programmed expert systems, deploying logic and probability to make decisions. Type 1 thinking is largely the only thinking currently used in artificial intelligence. Type 2 might be emergent out of the heavily connectionist-behaviorist approach, just as it is with biological intelligence, but the deep neural networks of today make no explicit symbolic reasoning. That is what creates the issues with interpretability. How is it possible to interpret how a network arrived at its conclusion when no meaningful symbolic process was employed? And making the AI explain its actions would make them use type 1 thinking to generate a valid (sounding) reasoning that fits the already obtained result.