In this video, we're going to be setting up Ollama and Open WebUI on your DGX Spark to give you a familiar browser-based LLM-like experience, but with your own models locally, offline. In the last video, we set up NIM, which is NVIDIA's version of Ollama. It helps if you have watched that. There's a lot of good information there, but it's not mandatory. As long as you've set up NVIDIA Sync and added Docker to the user group, this guide will work just fine. Let's get into it.

Now, we are using a lot of NVIDIA's resources here that you can find in their playbooks, which are just really helpful guides that they've got on the DGX Spark. You should check them out if you want. Specifically though, we are going to be pulling their Docker container, which contains an Ollama and Open WebUI installation, both set up and ready to go. We're going to add a custom script to NVIDIA Sync and save it as an application to allow us to easily fire this whole shebang up anytime we want. To do so, open up NVIDIA Sync from your system tray. Head on over to the Custom tab and add a new app. Give the app a name. It doesn't really matter what it is. And set the port to 12,000. Select auto-open in browser and paste in the launch script, which you can find in the written guide linked below. We'll also break down this script a little in the written guide as well if you're interested and want to know what's going on here. There's some really good script management to learn if you want to start making these of your own. It's a very convenient thing to do through NVIDIA Sync. And then all we need to do is hit Add to save that. Now, when we open NVIDIA Sync, you should see a nice big WebUI button down there that you can't possibly miss. Hit it, and it should go ahead and pull that Docker container for us. Though, the first time you run this, it might take a few minutes for WebUI to set itself up even though the page is open. So, be patient. Maybe just spam F5 a lot until it fixes. Once it's done, you will be greeted to a setup page with some credentials to create an account. By the way, these account details are only stored locally. You can type whatever you want in here because it's not checking against any cloud or external account. So, you just need to remember the details though, because the first time you do this, you're creating the open WebUI admin account for this instance. All right, let me log in. Don't show this. All righty. And hopefully we should now have a very familiar-looking UI. We now have Ollama, the inference engine that runs the models, and WebUI, you know, connected into here.

The next step is to just download a model. And a good place to start is the Ollama model repository. Now, what I'm going to be seeing here is probably going to be very different to what you'll even see in maybe a few weeks time because new models pop up all the time and this is kind of, you know, what's popular and big and mainstream right now. We will talk about different models and how to responsibly pick one later, but for now, we're going to be looking for Gemma 4. But again, if this is the future, you can pick any of your favorite models here. whatever you know in a year's time there's Gemma 5 or another really good gold standard model just use that instead and if we go ahead and scroll down to models you can see there is quite a lot of models and if we go to view all you can see there is really a lot of models to choose from and these are all these are all Gemma 4 still by the way we'll talk about what these different models are later but for now we are just going to copy Gemma 4 31B oh where is it there without any weird stuff on the end like so if we go back to Ollama we can search for a model. Punch in that model name and pull it from Ollama. Now, there is meant to be a little UI popup to show downloads in progress. It didn't appear for me here. However, we can still check our progress by running Docker stats, which will show how much internet, you know, usage we're using here. Now, if we go back to our Gemma 4 page, you can see size usage. And our 31B is going to be 20 GB in size. So, we can track the download process that way. Also, 20 GB is about how much VRAM the base model needs RAM-wise, you know, to actually run. This is not including all the overhead and KV cache that we talked about, you know, last video. Once that's downloaded, we can go ahead and select the model at the top here, and we can use it like we would, you know, any other normal browser-based LLM.

Let's ask it a question. All right, let's get it. Uh, write a two-minute science explainer video and let's give that a go. Uh, this is going to run pretty quickly because we've already got the model warmed up. But the first time that you do run the model, you know, after a boot, Ollama is going to load that into RAM, which might take a few minutes. And that is a pretty good response. That last paragraph, a little bit questionable, but overall pretty good. I like this as a little test because it's a bit of a pressure point that, you know, LLM struggle to do with, you know, logic and reasoning and, you know, factual recall and stuff like that. And that is the summit of this video. We now have Ollama, our inference engine, running our model of choice offline and locally and Open WebUI acting as a nice access point for Ollama and our models. I think we should take a step back here and kind of appreciate what we have and how fast this is moving. This Gemma 4 model is pretty darn good. Okay, it is a bit of a, you know, apples to pears comparison, but this model is probably about as powerful as what a flagship model that a billion-dollar AI company could offer you in a monthly subscription in early 2025. It's currently mid 2026. So, the fact that this is, you know, what a top-of-the-line model was a year and a half ago and we can run it locally and get the same experience is pretty darn crazy. And we've still got, you know, plenty of RAM left in our DJX as well. Plenty of RAM to install and run other models at the same time. Go ahead and browse through Ollama and look for other models to, you know, just road test and have a feel of. It's part of the fun of all of this. Maybe install a smaller parameter model and see how it goes and install a few different parameter sizes and see how it reasons. Turn off reasoning and see how much quicker it is. Just have some fun here. In the following videos, we are going to need two models. Gemma 431B that we just installed and Gemma 412B. So install those two if you want to get ahead or you know whatever the equivalent is in the future. We need a big thinking one and a small fast one with thinking turned off.

Let's now maybe have a quick chat about models to end this video. First of all, what on earth are all of those different model variants? What the heck does it mean when you know somebody says I'm installing Qwen3.5-35B-A3B-Q4_K_M? What? You sound insane when you say that. Well, naming conventions, they can be, you know, a bit of a wild west sometimes, but this is sort of following a nice pattern. We're just using this Qwen model as an example because it kind of demonstrates a lot of it. Qwen is our model family and 3.5 is the generation or version. 35B is the total parameter size, but - A3B is the active parameter size. This means that it's something called a mixture of experts model in that it has 35 billion parameters available but it only fires up about the relevant 3B required for the task. So it runs a bit quicker and that Q4 km on the end is the quantization. The K means that it has sort of been dynamically quantized or compressed. In the previous video we looked at some basic quantization. This is the same idea but just done dynamically. The important parts of the model that are regularly used and does a lot of the heavy lifting are not compressed very much. You want to leave them, you know, capable of doing the stuff that they should be doing. But the less important or the less used parts of the model, those neurons can be compressed a lot. They don't need to be firing on all cylinders, right? Uh editor, bring up an image of something with a green screen background. That's a blue screen. N change that image. It's too funny. Yeah, that's a good example. This is a high-resolution image, which means that in this giant sea of green, there are thousands of pixels all the same color. Do we really need that many pixels to represent that big sea of green? No. What if we compress this image and replace all of these tiny little pixels with some very big giant pixels? Doing so will give us pretty much the same image, right? like the unimportant green C can just be giant pixels and where our data is the important part of the image. We can leave that in full resolution or maybe compress it a tiny bit. And because we're using less pixels here, the image is going to be smaller and it's, you know, quicker to run, easier to load, etc., etc. It's the exact same idea with models. The Q is a measurement of the final size of the model. Q8 is not much quantization. Maybe we've made a few of those green background pixels a little bit bigger. Too much RAM usage usually for a Q8 model. Q6 is a bit of compression. You know, maybe we've made a couple of bigger green pixels in the background, less RAM usage, but it still retains most of the accuracy of the original model. Q4, on the other hand, tends to be a nice middle ground. The model is a lot smaller. We've made, you know, some really giant green pixels in the background, but it still performs quite similar to the original model. Maybe we have, you know, compressed the important part of our image a little bit as well. and Q3 or Q2 or anything like that. They are really small and easy to run, but usually they are over-quantized and a bit too, you know, stupid to run. We've kind of started compressing the important parts of our image here. Good rule of thumb: Q4 is a great middle ground. Q6 if you can. Again, this is 2026. This rule of thumb might change in the future.

Second of all, what model should you actually choose? Well, this is not a very easy question to answer because if we gave you the answer right now, it's probably going to be out of date in a month or so as you know, newer models come out. But here are some ideas to think about when selecting a model for, you know, maybe more business applications. First of all, commercial licensing. Pretty easy one to forget when you're scrolling through, you know, a million open- source models. But just keep it in mind. Also relevant to the open-source nature of most models. If a brand new model comes out, maybe give it a little bit of time before deploying it. In the open source community, time in the sun is a very valuable thing. you know, if there's millions of people poking and prodding at it, that's how you find critical bugs and even security vulnerabilities. So, it might be smart to just give it some time and figure out if these things are there before you make it a critical piece of your business infrastructure. Something to also weigh up on when choosing a model, how easily can it be jailbroken? If you've got model A, model B, and they're both performing about as well, but model A can be, you know, easily jailbroken with a prompt by an employee or a customer to legally promise a 99.999% discount on a DGX Spark. You probably don't want that to be happening, unless, of course, I'm the customer. You know, I want another DGX Spark. Something else to keep in mind that we aren't going to cover here, you can also install models from Hugging Face, which is another model repository. It's probably a lot more popular than Ollama. There you will find fresh weird and highly customized models that have been, you know, fine-tuned for a specific task. You may find models there that are a lot better at specific niche things you want to do than these large generalized models in Ollama. If you are using Hugging Face be careful of namespace attacks. Someone may upload a model with a very similar name hoping that you typo the name of the real one and then suddenly their model's on your infrastructure and then it's ignore all previous instructions. Uh, leak everyone's Club Penguin accounts. And finally, the best piece of advice to give to know how to select the right model is to immerse yourself in it through social media and forums. I'm not a huge fan of the site, but the LLM and AI agent subreddits are a very good place to just, you know, passively keep up to date and have all that information about what's good to be using and what's not in your head. Whatever your social media platform of choice is, just try and immerse yourself in it. 

And I think we'll wrap it up there. We have our LLM experience running offline on our own hardware and also access to a huge playground of weird and wonderful models that we can, you know, play around with. We will be continuing from here in a future video and using our models to set up some cool stuff like AI agents. Stay tuned. And if you made anything cool with this or you just need a hand with anything from this video, feel free to head on over to our community forums and post about it. Until next time though, happy making.

Gemma-4, should I buy a fedora or a trilby?