I was setting up a local LLM via Ollama to serve a couple of models, the main being gpt-oss-120b. I am also installing ComfyUI for some basic image generation.

While its true that gpt-oss-120b is not the latest one, my experience and general opinion has been this is the best model to run locally if your system allows. My Strix Halo machine has been setup with Ubuntu and 96GB of RAM dedicated for the inference.

There were a few hiccups, mainly because I was installing it when the team was switching between Moltbot and Openclaw and some documentations were also changing fluidly.

Here are few pointers if you are trying the same.

1. If you remember my previous post, I had a Ubuntu Dell Laptop lying around without much responsibility. I did consider spinning up a VM in my (other) Proxmox machine, but then decided to use this laptop for it. The first thing I did was create a dedicated user in the system and install Openclaw under this user. This provides a sort of isolation from Openclaw bricking the laptop by mistake.

2. My Strix Halo machine is running the Ollama and opening up only the port to Ollama and that too from known IP address (of my Openclaw machine). This might cause some delay if you follow this approach.

3. You need to make sure that Ollama has sufficient Context Reset interval. When I tried with the defaults (keep-alive : 5 minutes) and the API was not sending this, it kind of went into a short-term memory loss situation.

4. You might already be doing this, but the Ollama context size should also be sufficiently higher (as much as your local rig allows). I kept it at 8192.

   

   

5. I also found that linking with Whatsapp was, although simple, weird. Because you are left with talking to yourself unless you get a dedicated e-sim for Openclaw. I might do that later, but for now, I moved to setup a bot in Telegram and this has been pretty smooth.

6. Finally, I was doing everything at once. Installing Ollama, gpt-oss, openclaw and routing between two local machines etc. Looking back, I would have taken a simpler approach. Maybe start with your claude as your model provider and then switch to local LLM once you have ironed out the issues with OpenClaw.

While it was definitely fun, I don’t think its sustainable to keep this approach if I want to make sure that the Openclaw is somewhat useful while I could reliably use my local rig for other AI experiments. So, sadly while this was a proof of concept that you could run Openclaw locally, I will be moving back to Claude. I can’t imagine anyone doing this, in their local laptop unless they dedicate it for this purpose. Maybe one day. Not now.

Let me know if you have tried something with Openclaw. I am curious to learn what use cases are going to surface up with this new tool.

Anyway, between the Christmas holidays and now, I started working on the evenings and weekends to get it up running. At this moment, I think I have all the necessary hardware I would eventually want for a fully functioning homelab. (Except for 2 NAS HDDs, because, geez!, they are also getting expensive now). The actual installation is going slowly though.

By no means, is mine completed or well defined. Without further ado, here is my (partial) setup.

As you can see, most of the hardware are in place, while the software part is getting built every week.

The main motivation for me to get this out already, even before completing my setup, is this article I read on GitHub. I want to increase my “Luck Surface Area”. While I am trying to find my way around the networking complexities like GPU passthrough for my Coral TPU so that the Frigate will work, I am also seeing a lot of benefit in selfhosting most of these services. Eventually, this might become one of my best investment.

Over the coming weeks, I will be writing more about each component and my learnings. Meanwhile if you have been on a similar journey or have any question/feedback about my setup, I would love to hear!

I attended a few of those sessions and tried to capture my learnings here.

Key Sessions for me

• Securing AI Pipelines
  Abhinav Sharma, KodeKloud
  I learned how Kubeflow can be used to build pipelines on Kubernetes and how to secure pods, apps, and storage from unwanted traffic. Abhinav also introduced MinIO as an open-source storage layer.

• Building Intelligent Applications Beyond Prompts
  Peter Friese, Google
  Peter explained how Genkit and the Model Context Protocol (MCP) help developers build AI applications more easily. The key idea is to let LLMs interact with existing APIs, systems, and specialized tools, enabling orchestration of complex tasks without reinventing everything.

• Scaling AI Inference with Serverless
  Anmol Krishan Sachdeva & Paras Mamgain, Google
  This session highlighted how Knative and Kubeless on Kubernetes can scale inference workloads, distributing model training jobs across ephemeral pods. This makes it possible to cut inference time from hours to milliseconds.

• Open-Source Alternative: Ubicloud Showcase
  Ubicloud team
  Ubicloud presented itself as an open-source alternative to AWS. They also demoed a fully local RAG setup that runs on a laptop, letting you “talk to your docs” privately with open-source LLMs. It is a compelling option for sensitive or air-gapped environments.

• Day 2 Keynotes

  • Stephen Chin, Neo4j curated the program with thoughtful flow and energy.

  • Keynote on BAML: Vaibhav Gupta introduced a new programming language for building AI agents.

  • Manos, from Oumi made the case that open models are catching up to closed models, creating a path for both enterprises and startups to innovate without being locked in.

• Improving AI Inference Using KServe and vLLM
  Red Hat team
  This talk introduced OpenShift AI and explained how KServe functions as a model serving platform. With vLLM, inference can be optimized further through techniques like continuous batching, paged attention, speculative decoding, and quantization using the LLM Compressor library.

• Blueprints & any_llm/any_suite.
  Stefan French from Mozilla.ai
  Mozilla.ai showcased their open-source projects Blueprints and any-suite. The aim is to help developers who are new to AI adopt it in a simple, approachable way.

• From Local Experiments to Production
  Oleg Šelajev, Docker
  Oleg shared a repeatable process for turning local AI workflow experiments into production-ready deployments using containerized, static configurations.

There were many more sessions but I really had to make a difficult choice of selecting few of them to match my limited understanding and the relevance for me. Going by the attendance and chatter I am pretty sure I missed something interesting but I am still glad I had a chance to attend this event.

Conversations That Counted

The most valuable part of AI_Dev came from speaking with practitioners. I met people from defense, retail, finance, construction, government, and deep tech. These conversations were practical and unfiltered, and the common themes were clear:

• Lack of standards is holding people back.

• Rapid innovation means today’s choice could be outdated tomorrow.

While much of the discussion was tactical, it gave me insights no curated online course could match. In just two days, I learned more about the state of AI adoption than months of reading articles or LinkedIn posts could have given me.

Why Open Source Matters

History tells us that technology only reaches maturity when it becomes open. Open source creates standards, reduces vendor lock-in, and keeps you close to where innovation actually happens. For enterprises trying to adopt AI responsibly, those benefits are not theoretical, they are survival tactics.

Why AI_Dev Was Worth It

This event gave me a genuine boost into the AI space. The value came from the absence of sales pitches. The content was about learning, not selling, which is what made it authentic and worth my time.

Takeaway for Architects and Leaders

It is easy to scroll LinkedIn and be swayed by the loudest voices. The real insights come from listening one layer deeper, where practitioners are working with the technology every day. What I saw at AI_Dev convinces me that open source is the natural path forward for AI adoption in enterprises.

Reflection

I was once a skeptic. Now I am fully invested in AI’s potential. These two days showed me just how widespread adoption has already become, and how many possibilities are still ahead. AI is not a future discussion. It is happening everywhere, right now.

I finally want to thank the Linux Foundation and AI_DEV for organizing this fantastic event. I hope to attend their future sessions too.

#AIAdoption #OpenSourceAI #SoftwareArchitecture #AI_Dev #MLOps

When I was asked to take on a global leadership role, my first reaction was hesitation. Up to that point, most of my work with non-functional engineering had been from an architect’s perspective: ensuring systems were scalable, resilient, and compliant. Taking responsibility for shaping a global competency center that spanned four regions felt like a leap into the unknown.

But that is the beauty of opportunities — they rarely arrive when you feel perfectly ready. I am grateful that I was trusted with this responsibility, because the experience gave me insights into how organizations really function that I might never have gained otherwise.

More Than Just Tools

When people think of standards, they often think only of tooling. In practice, the scope was far wider. We had to consider reporting frameworks, clarify the boundaries of responsibilities between project and product teams, and even define templates for annual goals.

It quickly became clear that this wasn’t something I could do alone. I leaned heavily on subject matter experts and experienced colleagues who volunteered their time and knowledge. Their input was invaluable, and the collaboration reminded me that building standards is as much about community as it is about process.

Influence Over Enforcement

One of the most interesting challenges was the diversity of regional practices. Each region already had solutions that worked for them, often tailored to local needs. Without a formal mandate to enforce a single global standard, we had to take a different path: influence over enforcement.

That meant organizing voluntary learning sessions, offering practical support to teams who wanted to adopt but lacked the know-how, and above all, positioning the competency center as a partner rather than an authority. Progress was slow at times, but small wins began to accumulate.

Small Wins, Big Impact

Those small wins didn’t come from forcing compliance, but from creating visibility. Over time, the competency center became recognized as a one-stop shop for expert advice, shared knowledge, and best practices. That visibility was our biggest achievement. It gave teams a trusted place to turn to and gradually built confidence in the value of global alignment.

What I Learned

The biggest lesson I took away is that organizational buy-in matters more than technical superiority. Even the best designed standards won’t stick without governance and executive sponsorship. If I were to set up a global competency center again, I would ensure that both are in place from day one.

I am thankful to have had the chance to play this role, because it reminded me that technical excellence is only one side of the equation. The other side, perhaps the harder one, is alignment, culture, and trust. That’s what turns standards into adoption and good ideas into lasting practice.

I’d love to hear your perspective: what challenges have you faced when trying to roll out standards across regions or teams? If you’ve been part of a global change effort, whether in architecture or beyond, share your story, the hidden challenges are often where the best lessons live.

Engineers and analysts are now advocating for smaller, tightly scoped models that offer higher reliability and drastically better ROI. Think targeted tools, not one-size-fits-all giants.

💡 Here’s the extrapolation: Enterprises already sit on a goldmine of domain-specific data. Instead of chasing generic mega-models, they can train lean, custom models tailored to their internal ecosystems—models that improve over time, gain flavor with use, and get better with seasoning. Like a cast-iron pan, the more it’s used, the better it performs.

✅ More reliable

✅ Economically sustainable

✅ Purpose-built for real workflows

It's not about fighting scale—it's about embracing precision.

What are your thoughts on this? Do you think we are overseeing this Achilles' heel of large models?

#GenAI #LLM #AIstrategy #EnterpriseAI #TechLeadership #MachineLearning #ROI #SmallerIsSmarter

Subscribe now

Subscribe now