Written by Joseph Park, Lulu Ito and Tim Burress
Since our last article on neuro-symbolic and probabilistic programming, I’ve received a fair amount of responses from my network. This interest stems partly from the intriguing idea that these technologies seem to represent viable alternatives to Large Language Models (LLMs), which many people nowadays consider synonymous with ‘AI’. To my surprise, more individuals than I imagined, including those involved in the AI industry, expressed unfamiliarity with symbolic AI.
We previously depicted LLMs as essentially a black box, highlighting that “LLMs and neural networks simply consist of countless numbers and unstructured interconnections without the flexibility of a modular, structured system that one can easily edit, understand or retrain.”
This description raised questions about what it means for an artificial intelligence model to be modular, and why the currently dominant paradigm, LLM, employs a different approach (a monolithic approach, as we’ll explain later) if the benefits of modularity are so obvious. Let’s dive into these questions.
Modular vs. Monolithic Approaches
In fact, monolithic and modular are both common approaches in programming, rather than concepts specific to neuro-symbolic AI or probabilistic programming.
Monolithic Approach
Most neural network approaches, including many LLMs, employ what’s called a monolithic approach, where the programmer uses a single, self-contained code to create the whole program. The best analogy for this is handcrafting an object with a lump of clay—one material that, when complete, becomes a concrete whole. It’s easy and fast, but crucially, once you’ve built something and you’ve realized something is not right, it’s difficult to make the change without compromising the whole. It’s rigid because sometimes your only option is to start over from the beginning, and you can’t simply remove one faulty piece of code and replace it with another.
There are parallels one can draw to an AI model enabled by a monolithic approach. It’s convenient when you build the model for the first time, but it would be very cumbersome and expensive to train it again if something has gone wrong, making it far from an ideal system. As the famous black box analogy highlights, it’s very difficult to pinpoint the cause of the problem exactly. Although since our last article was published some LLM companies have tried to improve this aspect, it still doesn’t look like a comprehensive solution.
(It’s important to note that monolithic is not a defining characteristic of all LLMs, as some neural networks can be ‘modular’ in a different sense too. However, for the purpose of this article, we are making this association since most LLMs currently available to us seem to be monolithic.)
Modular Approach
The modular approach is in many ways the opposite. The best analogy here is LEGO blocks: Each component that makes the system is independent from each other, but can be combined to make one whole object. It’s easily modifiable without affecting the other parts – you can revert to any stage of the building and start again from there.
For similar reasons, modular designs are typically more reliable, because each of the components can be tested separately to ensure that they will behave the way they’re supposed to. Therefore they’re easier to debug, and easier to improve, compared to monolithic approaches. However, modular approach remains less prevalent in AI development due to its complexity in implementation and the need for more intricate system design. Additionally, the success of monolithic models in achieving impressive results has led many researchers and companies to continue investing in and refining the more traditional, unified approach.
Modularity in Probabilistic Programming
While all this has been known for decades, where modularity was one aspect of structured programming, what seems relatively new in the context of probabilistic programming is being able to write one model, then apply multiple (modular) inference algorithms to it, or to take several models and combine them into a larger one (or to take one big model and divide it into several smaller ones), or to use one language (like GenSQL) to ask queries about any model.
Much of this power stems from modularity itself. The concept of combining different types of AI, such as neural networks for pattern matching and probabilistic programs for reasoning, relies heavily on the modularity of each component. Again, think of LEGO blocks — you can construct complex structures from well-defined blocks with clear interfaces, without having to create new blocks from scratch each time. Once you have a particular inference algorithm “block,” you can simply plug it in. You don’t even need to worry about its internal workings (as long as you trust it). This modularity significantly speeds up prototyping and allows humans to think, experiment, and play at higher levels of abstraction instead of being forced down into the 1’s and 0’s.
There’s an appeal to modular software design that’s deeply rooted in the way the human mind works, naturally organizing information into modular hierarchies. We see this organizational principle reflected in many aspects of our world: universities are divided into departments, disciplines are broken down into specialties, and living things are classified using taxonomic ranks.
This modular approach to understanding extends to how we comprehend complex systems, including software. In trying to understand someone else’s code—or more challenging, to reverse-engineer it—we instinctively seek out a modular structure. The process typically begins with identifying subroutine calls, focusing on the most relevant ones. We then work to understand the function of each subroutine, gradually building a comprehensive understanding of the entire system from these modular components.
However, the need for modularity in design differs significantly between humans and machines. While humans rely on modular structures for comprehension and problem-solving, machines operate on a fundamentally different level. Compilers, for instance, often rewrite human-readable code to optimize performance and efficiency, sometimes dismantling the original modular design in the process. This transformation doesn’t impede the code’s functionality because CPUs execute instructions without needing to “understand” them in a human sense.
This dichotomy between human comprehension and machine execution mirrors the contrast between explainable, symbolic AI and the often opaque nature of neural networks. Symbolic AI, with its modular and interpretable structure, aligns more closely with human cognitive processes. In contrast, neural networks, while powerful, often function as black boxes, processing information in ways that can be challenging for humans to interpret or explain.
The tension between these approaches lies at the heart of many current debates in AI development. Finding ways to bridge this gap and creating systems that are both high-performance and interpretable remains a central challenge in the field.
Conclusion: Towards a Diverse AI Ecosystem
As AI continues to evolve and permeate various aspects of our lives, the balance between monolithic and modular approaches will likely shift. This shift could lead to a more diverse AI ecosystem, where the choice of approach is guided by the specific needs and constraints of each application.
The recent, rapid development of AI has been largely driven by big technology companies who own foundational AI models. LLMs with monolithic approaches — fast and scalable — have been preferred from a business point of view. However, as AI technology becomes more integrated into our daily lives, we anticipate that the limitations of LLMs will make them incompatible with many of our habits, expectations, and preferred ways of operating. This is especially the case in areas of our lives and work that demand high explainability, fine-grained control, or domain-specific expertise such as healthcare and medicine. The flexible, modifiable nature of modular alternatives like probabilistic programming are primed to become the go-to option.
Different AI approaches will have their place. LLMs will still be useful for some domains, while others will benefit from alternative forms of AI models like probabilistic programming. The future of AI likely involves a coexistence of diverse forms of artificial intelligence, each suited to different tasks and requirements.
By understanding the strengths and limitations of modular and monolithic approaches, we can better navigate the evolving landscape of AI technologies and choose the most appropriate solutions for specific challenges. This understanding will be key in shaping the future of AI, where various approaches complement each other to create more robust, flexible, and intelligent systems.
Ultimately, the future of AI may not be a competition between monolithic and modular approaches, but rather a symbiosis. By embracing this diversity and fostering collaboration between different AI paradigms, we can create a future where AI truly serves and enhances human capabilities across all domains of life. This hybrid approach could pave the way for AI systems that are not only more capable but also more aligned with human needs and values.
Joseph Park is the Content Lead at DAL (joseph@dalab.xyz)
Lulu Ito is an intern at DAL and a student at Keio University with a primary interest in Statistical Machine Learning (luluito@keio.jp)
Illustration: Asuka Zoe Hayashi
Edits: Janine Liberty
