(All Things Open 2025) Open Small AI Toolbox
Building for Performance, Privacy, and the Planet
Introduction: A Journey Toward Responsible Innovation
Many people who read my newsletter or hear me speak about AI might think I’m anti-AI. I’m not. But I do believe we have to be more intentional about how we use it. This belief is the result of a long journey working in emerging tech.
From my earliest days in the tech industry, I’ve been drawn to emerging technologies. I have always been fascinated by the intersection of technology and society—how new tools influence the way we interact. I started my career exploring mixed reality, seeing its potential to augment our communication and learning. Later, at institutions ranging from Clark Atlanta University to Caltech to UNC Chapel Hill, I dove into telepresence and sensor research that contributed to the early Internet of Things.
When I moved from academia to the startup world, it was during the early days of social media. I saw how these platforms connected people in unprecedented ways, powering global social movements such as the Arab Spring, the Occupy movement, the # MeToo movement, and Black Lives Matter. I experienced social media’s immense connection capabilities firsthand while I was building communities like Women Who Code Atlanta and REFACTR.TECH from the ground up. It seemed that technology was a tool for social good, connecting communities and progressing society. At the time, I viewed this intersection as a net positive, poor sweet summer child that I was.
But over the last five to ten years, my perspective began to shift. I started to see the harms that can be created or perpetuated by the ways we innovate. I also noticed another trend: as each emerging technology matures, each iteration exponentially increases the negative impact we place on each other and our environmental resources. Mobile, wearables, and IoT quickly transitioned us to an always-on, big data world. The metaverse has had its impact, building on the previous emerging tech generation’s hyper-connectivity and sensor networks, while introducing new safety, privacy, and surveillance concerns. Blockchain and crypto mining had an even bigger environmental impact than their metaverse predecessor. Now, with AI, it feels like we are taking a monumental toll, both environmentally and societally, again with no guardrails in place. This trend isn't sustainable; it’s destructive.
This has led me to focus on a central question: How can we innovate with AI responsibly? How can we build great products that people love while uplifting communities, utilizing our environmental resources in sustainable ways, and mitigating societal harms, such as big tech-fueled surveillance? I believe that open-source, small language models (SLMs) offer a balanced approach to advancing AI, providing us with the superpowers to achieve both. This post is a toolbox for building AI that is more accurate, performant, private, and less harmful to the planet.
1. Smaller, Faster, Cheaper: The Efficiency of Small Models
There’s a common misconception that small models are merely poorly performing versions of large models. This isn’t the case. SLMs offer a distinct set of advantages rooted in efficiency, making them not just an alternative to large language models (LLMs), but often a superior choice for specific, real-world applications.
Performance Benefits of SLMs
Speed: Because they have fewer parameters, SLMs are inherently faster. When run locally, they eliminate network latency entirely, making them ideal for real-time applications where instant responses are critical.
Cost-Effectiveness: Running models locally allows you to avoid the recurring API fees associated with cloud-based LLMs. They can run on affordable consumer hardware, including CPUs, and in production, they can be deployed at 1/10th to 1/50th of the cost of their larger counterparts.
Specialization: SLMs can be fine-tuned for highly specific, domain-focused tasks. For tasks such as summarizing legal documents, classifying customer support tickets, or analyzing medical texts, a specialized SLM can be more reliable and even outperform a massive, generalist LLM that has been trained on broad internet data.
For me, this is one practical answer to the unsustainable trend I observed, where each new technology wave demanded more. This efficiency isn't just a technical detail—it's a strategic advantage. It’s what can make AI practical and scalable for businesses, startups, and individual developers alike. It transforms AI from a high-cost experiment into a reliable, lower-risk tool that can be embedded into existing workflows while quietly driving bottom-line impact.
2. Your Data, Your Hardware: Upholding Privacy and Security
When you use a local language model, the computation happens on your hardware. This simple fact has profound implications for privacy, security, and reducing surveillance, as it eliminates the need to send sensitive information to a third-party cloud provider. This level of control is crucial for industries with stringent data governance requirements, such as healthcare, finance, and law.
The risks associated with some public, centralized services can be significant and are often buried in privacy policies. For example, one prominent AI service states:
The personal information we collect from you may be stored on a server outside the country where you live. We store the information we collect in secure servers located in the People's Republic of China.
For a lot of projects, especially those involving sensitive community or business data, a policy like this is an immediate non-starter. By running models locally, you maintain full control over your information, ensuring it remains within your trusted environment.
This control is a foundational step toward what can be called "digital sovereignty." It allows organizations and communities to build and operate AI systems on their own terms, without being locked into foreign-controlled platforms that may not share their priorities or values. This is how we ensure the tools we build truly serve the communities they're meant for.
3. A Greener AI: Addressing the Environmental Footprint
Running giant AI models in massive data centers consumes a tremendous amount of energy for computation, cooling, and data transfer. As an industry, we must be transparent about the environmental cost of the current AI paradigm shift. SLMs offer a more eco-friendly alternative.
Because SLMs have lower computational and memory requirements, they translate directly to lower energy consumption. Running them on local or decentralized hardware also reduces the energy footprint associated with large-scale data transfer across global networks. This aligns with a more sustainable vision for technology, one that acknowledges its resource impact.
While I believe the reduction in operational energy is the most significant factor, we must take a holistic view, as the environmental calculus is complex. A massive proliferation of smaller, decentralized devices could create a new e-waste challenge. However, the significant reduction in operational energy—the power consumed during day-to-day use—makes SLMs a critical tool for building AI systems that are more responsible stewards of our planet’s resources.
4. For Everyone, By Everyone: How Open Source Democratizes AI
The combination of small models and open source is a powerful force for democratization. Open-source SLMs lower the barrier to entry, empowering a global community of developers, researchers, and startups to contribute to building the future of AI—not just a handful of large, well-funded tech companies.
A clear example of this is InkubaLM, Africa’s first multilingual SLM. By making the model's architecture openly available, its creators empowered a community to apply techniques like quantization and distillation, compressing it by 75% and making it small enough to run offline on entry-level smartphones. This enables the creation of localized tools for underserved languages and cultures, something that is often an afterthought for large, commercially-driven models.
Image Credit: Lelapa.AI
This approach allows open-source models to serve as a form of digital public infrastructure (DPI)—foundational tools that are inspectable, auditable, and governed in the public interest. This fosters trust and provides a pathway for communities to shape AI to their own needs.
Open models offer a more realistic path to autonomy, providing the ability to shape and govern AI tools without needing to own the entire compute stack.
This, to me, is the most crucial point: it’s about decentralizing power and ensuring the future of AI reflects the diverse world it’s meant to serve. This democratization ensures that the future of AI will reflect a diverse range of perspectives and societal priorities, leading to technology that is more equitable, trustworthy, and beneficial for everyone.
Conclusion: Rethinking Scale for a Smarter Future
While large, cloud-based models have demonstrated the incredible potential of AI, the future of practical, responsible, and scalable deployment lies in a more nuanced and "right-sized" approach. Small language models are not a step backward; they are a strategic evolution. They offer a powerful combination of performance on specialized tasks, robust privacy, cost-effectiveness, and environmental sustainability that their larger counterparts cannot match.
By embracing open-source SLMs, we equip ourselves with a toolbox to build a better future with AI—one that is more efficient, more private, more sustainable, and more democratic.
As we move beyond experimentation and into deployment at scale, it is clear that the smartest systems are not necessarily the biggest. The critical question for builders and leaders is no longer "How big can we build?" but "What is the right-sized AI to create the future we want?"
