Design choices for future AI - finding a way forward

Sir John Lazar CBE FREng

1. So much current AI innovation assumes pure eternal abundance – abundance of compute, energy, talent and capital. This inevitably leads to “brute force” design choices. But for most of the world, constraints abound. And understanding constraints, and designing to address them, results in astounding innovation grounded in real lives. Necessity is the mother of invention. This is the basis for our Enza Capital thesis for how Africa can exercise agency in the age of AI, and why we are launching our new AI Africa Fund. 
2. We have built chatbots that are articulate, verbose, sycophantic and addictive, and are driving cognitive offload, and even cognitive surrender, especially for our children. But all of us could also choose to design AIs that are not designed simply to maximise engagement but introduce a more positive Socratic “friction” that maintains human cognitive endurance. These are the issues we are trying to address at the Raspberry Pi Foundation, led by CEO Philip Colligan, who was recently named in Time Magazine’s 100 most influential people in AI. 
3. We in the technology community saw the world change dramatically with the release of Claude Opus 4.5 on 24 November 2025, marking a clear step change in the acceleration of AI agent performance. Fast forward to the OpenAI / Hugging Face breach, and all the other concerning examples we have witnessed recently. But why did the dominant AI companies think it was a good idea to make the overriding design choice a strong drive for autonomous agents, that can spawn a myriad of sub-agents, running independently for as long as possible, often based on badly-formed and incomplete instructions, occasionally coming back to a bored human for a cursory human-in-the-loop “tick”?
   
   We seem to have forgotten, as a number of people have written (Rohit Krishnan, Henry Farrell, Cal Newport, amongst others), that our civilisation has been built on marshalling “super-intelligent” structures like the corporation, the market, academic institutions with rigorous peer review, Wikipedia, and so on. These systems bring together disparate and often specialised human intelligences to produce remarkable results – as Brad DeLong aptly calls it: “anthology intelligence”. These super-intelligences all deal with the reality that humans fail, through incompetence, lack of training, exhaustion, sometimes criminality. Anyone who has ever managed a team will know how adept humans are at "reward hacking", but we know how to deal with it! And we have management processes, insurance, documentation, budgets, schedules, auditing, tech roadmaps, contingency plans and regulatory compliance to help us. This is how we manage composite risk in complex systems. 
    
   It is possible to make a different design choice: we can knit and plumb these new, powerful, fast-moving but “jagged” AI agents into productive peer relationships with humans, within a re-imagined super-intelligent system that enables sensible oversight, balanced risk management, productive friction, adaptation to the differing speeds of humans and AI agents, and most importantly, as Ethan Mollick has highlighted, retains interest and engagement for humans: “if agents make every interesting decision and leave people with the approvals, the exceptions, and the failures, we will have automated the wrong half of the job.” 
    
   Crucially, as Seb Krier and Nick Jennings have written, this means broadening our focus on alignment of a single “cognitive intelligence”, to “social intelligence. As Seb says, “Proper scaffold engineering and multi-agent segmentation doesn’t do away with the need for aligned models, but they transform alignment from a psychological problem of internal model disposition into a constitutional problem of institutional design”. 
    
   Thinking sensibly about these systems, and how we manage risk, is the essence of the ongoing cross-disciplinary work that we are doing on Agentic AI and sovereignty at the Royal Academy of Engineering.

Why am I wading in now?

• Assuming a very narrow band of design choices pushes us into two camps: those who back unfettered acceleration, and those who find themselves part of a profound backlash. Of course we need sensible regulation! But a deep negative backlash with overbearing regulation will also stunt enormous opportunities that sensible well-designed tools could open up. For example, if we designed a safe, supportive AI tutor that did not drive dependency and laziness and could run on a low-end mobile device, think about how it could change the educational trajectory of a kid sitting in a classroom in Africa with 80 other classmates. 
• This is also a question of sovereignty and agency. Sovereignty means retaining meaningful choices over critical dependencies, and the capability to design, build, procure, deploy and govern systems in line with our own objectives. Unless we step back and realise that we can forge alternative design options, we have very limited sovereignty and agency! We are stuck on a train of helplessness as it hurtles down the tracks.