So, you got your DGX Spark, and you want to get going with it. This is the video for you because we're going to get a few different LLMs running on our Spark all at once. This is also going to be a bit of a crash course on the world of LLMs, as well as explaining some of the buzzwords and terms that you will hear. Now, we are assuming that you do have some level of experience with LLMs. Using ChatGPT or Claude on the regular is more than enough. So, let's get into it. Starting off, make sure you have gone ahead and set up your DGX Spark and updated. You can do this remotely with its Wi-Fi hotspot with the details on the card that came in the box. But we just plugged in a monitor, keyboard, and mouse to set it up with the regular route. Once you reach the desktop and everything is updated, we are ready to roll.

The first thing that we'll do is install NVIDIA Sync on another computer, the PC that you actually, you know, want to develop from. NVIDIA Sync is a desktop app that allows you to easily monitor and interact with your DGX Spark over your local network. And it also lets us easily SSH into it. Download and install it, and you'll be prompted to punch in a name for the device. Your hostname, which is on the card. It's the uh hotspot SSID. Just punch that into that field as well as your username and password that you set up with your Spark on, you know, the first time that you set it up. Good chance that after doing this, you will have another update for your Spark. So, let that install as well. Now, if you head on over to your system icon tray and select the NVIDIA Sync app, you can go ahead and open up your DGX dashboard. This is a handy little tool, and it lets you see your current new RAM usage as well as your GPU utilization. From here, you can also go ahead and open up a new terminal, which is SSH into your machine. From now on, this terminal is how we're going to be using our Spark.

All righty. To run an LLM, we need an inference engine which is a fancy way of saying, you know, software that runs the AI model. Now, you have likely heard of Ollama, which is the biggest and most popular one. We will be setting up Ollama in a very near future video. But in this video, we will be spooling up NIM, which is NVIDIA's enterprise-level inference engine built off VLM. We will talk more about when to use which one later, but for now, let's just set it up as a first demo. Starting off, you will need to create an account for NVIDIA NGC. This is their sort of enterprise machine learning AI developer platform. We need an account because it'll let us download their specifically trained NIM models. Once you've created your account, go over to the NGC catalog page for the model you wish to use. We're just going to be using a Qwen 3 model here. If you click on get container, it will likely prompt you with more things required before you can start downloading the models. Once you've gone ahead and done what it asks you to do, it'll give you an API key. Save this somewhere secure and private, and make sure you do not lose it because we will need it. It's also hard to recover, and you'll need to regenerate a new one. Just make sure you save it somewhere. If you have the API key and you can click get container, and you see this little, you know, image path pop up here. Your account most likely has the right permissions to continue from here. Annoying little thing we got to do there, but now we can go ahead and install NIM.

The first thing that we'll do is add Docker to our user group with this command here. This means that we don't need to use pseudo every time that we want to use a Docker command. Super helpful. In the written guide link below, you will find all the commands that we'll be using in this video, so you can just paste them in as you need. Please do not write anything in this video by hand. And yes, a lot of this will be using Docker, which is a way to bundle bits of software together into nice little self-contained containers that don't have to worry about compatibility or software conflicts. When we say that we're installing NIM, we instead mean that we'll be downloading and deploying a Docker container for a specific LLM with NIM installed, you know, inside that Docker container as well as all the other little bits of machinery required to make it run. If you aren't familiar with Docker, it's definitely worth doing a bit of research, as you may be using it quite a lot on this thing. Next, we'll go ahead and set up Docker with our NGC authentication by punching in these two lines here. Of course, in this first line, you'll need to actually put the API key that you generated earlier. Now, let's go ahead and download and deploy our Docker container with this big wall of text here.

As our first demo, we're going to be deploying a 32B Qwen 3 model. The 32B just means it's a 32 billion parameter model. It's kind of the size of the brain. And depending on your internet speed, this can take 10 to 30 minutes. So, it might be time to go grab a cup of tea. And we're back magically after a Costco hot dog. Once that is done, though, your Quinn model should be up and running. And if we take a look at our dashboard, it is using a fat amount of RAM. We will fix that later, but you should be able to open up a new terminal. So, let's just put that one in the top right and use our video sync. Grab a new terminal. You're going to have a lot of terminals. So, if you've got snapping enabled on your system, it's going to be very helpful. And if we punch in this CURL command, we will send a prompt to Qwen asking it to name 10 red fruits. And if we take a look in here, oh, let's highlight it all. It's our response. How darn cool and quick was that to set up? And this, by the way, is kind of the summit of our video. We now have a model running locally, offline, and private that we can prompt with this curl command here with whatever we want. If you are in the know, you can take this and go ahead and start integrating it into your agentic AI systems or whatever shebang you got set up. For the rest of this video, though, we are going to dissect a bit of what's going on here, fix that excessive RAM issue, and get a few more models running all at the same time on our Spark.

Starting off, what the heck is that CURL Command? Let's go ahead and clear that just to make this a bit more readable. And we don't really need to see our Qwen terminal running like that. So let's zoom in and make that a bit bigger. This JSON here is sort of a standardized string format for sending inference requests to a model. You will see it crop up a lot. You don't need to know super in-depth how it works as in time you'll probably have these generated for you by other LLMs. But just quickly at the top here we specify the model and its address on our machine. We have our prompt that we're, you know, feeding it and getting that's the prompt that we want to give it as well as this system prompt. This system prompt is the instructions that we give the model to kind of change its nature and how it views the world. It's sort of, you know, it's the thinking hat that it puts on while it reads and processes our prompt. For example, let's go ahead and say uh speak with a funny British accent. Then we have our keyword arguments or KWARGs. These are the settings that we can tweak that go deeper into the model's behavior. They change from model to model. But here we are turning off thinking or reasoning as we don't need the model to think deeply about what fruits are read. Obviously turn this on as it's needed. Then at the bottom we have our parameters. These are the knobs that we can turn to alter how it thinks. For example, we can set the maximum allowed tokens for this request. If it hits that number, it immediately stops, you know, processing and sends whatever it's got. While we're here, let's list 10 red fruits with that funny British accent. Right, then I've got 10 red fruits for you. Let's see. I'll list them in me best cheeky British fashion, shall I? Sure.

This Quinn model looks like it's using an ungodly amount of RAM. But this is actually just NIM allocating those resources to our Qwen model in case it needs it. How much RAM does it actually need? Well, that really depends on your setup. The Qwen model itself is using about 16ish GB of RAM. NIM and all the other bits of machinery to run the model are probably about 5 gigs of RAM. Then we also have to allocate memory to the token context window, which is the LLM's working memory. By default, we have a context length of about 8,000 tokens, which needs about 2 GB of RAM for quint 3. The needed RAM will change from model to model, though. You know, it's not always 2 GB of RAM for an 8K context window. Now, this size window is usually enough for most AI agentic applications. If you were limiting this bot to, you know, really small prompts and getting it to only output, you know, one paragraph answers, maybe it's a really simple cog in your machinery, you could drop the context length to maybe 4K, which would only use 1 gigabyte. You don't need a big working memory there. Or let's say you wanted Qwen to read really dense 40-page financial reports. you might need to push it up to a 32k context window, which is Qwen's maximum native context window. It's the limit you can do without doing, you know, special modifications. However, 32K is four times more than 8K. So, this will need 8 GB of RAM, which is four times more. You'll need to plan ahead and size your context window according to your needs. Let's for now keep it at 8K token window. So that's 2 GB plus 16 gigs of our, you know, Qwen model and then say seven more gigs for NIM and overhead and maybe a bit of breathing room. We only need 25 gigs of RAM, right? Well, yeah, maybe a bit more though. Something to consider though is what happens if you have multiple inference requests coming in at the same time. If three people try to send this container of Qwen a prompt at the same time, it will process all three requests at once, and you'll need enough RAM for three lots of 8K context windows, or 6 GB instead of two. How many simultaneous requests you plan for comes down to, you know, what you're designing for and what you intend to do with it, but you want to ensure that it has maybe a bit more RAM than needed so that your model doesn't crash. We're not going to get into it here, but there are also ways to queue up and organize prompt traffic so that your model only has to process one, maybe two at a time. So, we're going to run 8K context window, and we'll prepare for six simultaneous requests at once. So, that brings our RAM usage to 33 GB. We'll round that up to 36 GB just to be a little bit safe.

To actually run this model, we'll need to first stop our current Quinn container cuz that's still running in the background. A handy command here is docker stats, which shows all the current containers and what they're doing. Doesn't really work when we're zoomed in, but if we zoom out, you can see, you know, we've got a nice little layout here. Let's hit Control+C to exit that. And then we're going to go ahead and docker stop. And we're just going to input the name there. Uh, nim-Qwen-32b. That'll go ahead and shut it down. And as you can see, our system RAM drops back to just idle. And here is our command to start the container again, but this time with less memory. As you can see here, we're manually setting our maximum allowed context window here. Change this to whatever size you want. And we're also allocating our RAM with this number here. It's a bit funky. This number is a multiplier of the, you know, total RAM available. We have 128 gigs of total RAM. And 28% is close to 36 GB, which is what we were aiming for. Something else to note on this one, we don't have those make directory and chmod commands anymore as we only need them the first time that we run this container. This here is the command that you probably should be using anytime you want to boot this Qwen model back up. Let's go ahead and run that and it should only take a few minutes to come back. Once that has fired up, if we take a look at our dashboard, as you can see, we are using far less memory now. And we should be able to send the exact same curl command as before and get our request back. And in 3 2 1 go. There we go. Close enough.

All right. Sweet. Let's boot up another model. You can keep your Qwen container running as long as it's not hogging up all the RAM. We're going to go ahead and set up and deploy a Llama 3.1 container with this command here. And if you get an authentication error here, punch in your NGC API key line. Again, also note that this is pretty much the same setup command as before. We're just instead pointing it to use that Llama 3.1 model and giving it 15% of our systems RAM. It's an 8B model, so it's smaller and needs less RAM. Also, we are instead setting it up on port 8001 instead of 8,000. Each instance of a model needs its own port. So, you know, we can send a curl command to it. All righty. I'm going to go ahead and open up a new terminal again. Pop that over those two. And I'm going to go ahead and post in our curl command to ask it a question. Now, the request JSON for this one is slightly different. First of all, we're looking for the llama model on the 801 port, which we set it up as. And we don't need to specify reasoning here anymore because Llama 3.1B doesn't have reasoning. So, let's go ahead and run that. This one should be pretty quick to reply because it's a bit of a smaller model. Beautiful. Look at how quick that was. And fun fact about Llama, they're based on a family of models developed by Meta AI.

And just to round it off, let's boot up a third concurrent model, Nemotron Nano9B. I'm really running out of room for terminal, so we're just going to start overlaying everything. And we'll go ahead and do so with this command here. It's pretty much the same thing. You should be used to this by now. And after that's done, we can go ahead and open up yet another terminal. I'll pop it over like so. We're really running out of room here. And we can send a curl request with the following line here. Again, pretty much the same curl as before, but Nemotron is a reasoning model as well. In the system role here, we have this /no think, which turns off reasoning. Remove it if you want to turn it on. And let's go ahead and run that. It should be a pretty quick thing. It's only a 9B model. Come on. There we go. And we have some information about quantization there. Now, something to play around here with. Nemotron has a massive maximum context window of 128,000 tokens. Have a play around with this if you want. Just be sure to allocate enough RAM accordingly. And that is three models running all at the same time. You can send a call request to them all at the same time. You can send multiple curl requests to one model and it will chug through them all. And I know we've kind of flown through the process of spooling these up, but it really is that easy. And all we really need to do is ensure that we allocate our RAM responsibly.

There is one more thing that we should talk about before we wrap this up. And that is quantization. All the models that we're running here have been quantized, which is a fancy way of saying that they've been, you know, compressed. A great analogy is an image. If you have a fancy camera and take a photo, it'll likely be saved in a raw format. This is the highest most, you know, absolute version of the photo with all the original data in it. You know, it's like it's the highest quality version of, you know, the pure pixel data coming through. But if you go online and look at an image, you will find that most images are JPEGs, not RAW. Why is this the case? Well, here is a RAW photo and here is a high-quality JPEG. The JPEG is four times smaller in file size. It uses four times less RAM and uses less processing power to view. And it looks the same as the RAW. This is because our RAW, you know, has so much unnecessary data in it. We can, you know, make it a little bit smaller and compress it and still, you know, get pretty much the same result. We could make this JPEG four times smaller again by compressing it even more. This photo is now noticeably slightly worse, but it's still usable. There is however, a limit to this because if we keep compressing our JPEG too much, it becomes completely unusable.

And this is kind of the same idea for our AI models. Qwen 3, for example, the original model was trained at FP16 or technically something called BF16. Anyways, this is our high-quality raw photo that has all the data. But FP16 is very RAM and processing-intensive to run. So instead, it was quantized or compressed down to FP8. This halves the amount of RAM that the model needs and makes it, you know, run a little bit faster in the process as well. Now, there is a performance drop going from FP16 to FP8, but it's often, you know, pretty small and negligible. If however we tried going from FP8 to FP4, we would again half the amount of RAM required and get you know a little bit faster on the processing speeds but there would be likely a big drop in performance and you know model intelligence and that's maybe a little too much of a loss there which is why FP8 models you know are kind of a bit more common. However, that is changing as very smart people are figuring out ways to get more out of less. Some FP4 models are not that bad nowadays. And there's also cool things like MVFP4 and NVFP4. NVFP4 is a NVIDIA quantization that gives FP4 RAM sizes and processing speeds, but mostly keeps the performance and intelligence of FP8. Kind of the best of both worlds there. It's not always that case, but when it works, it usually works pretty well. In fact, the reason that we started with NIM is because both Qwen 3 and our Nemotron models are quantized to NVFP4. And it's just an easy way to get going with the NVFP4 versions of those models. And just some good rules of thumbs to go with them. Our Qwen 3 model, it's a 32B model. If it's quantized at NVFP4, it uses roughly 16 GB of RAM, which is half of that 32B number. If it were instead an FP8 model, it would use double the amount of RAM and it would need 32 gigs. If it were FP16, it would need 64 gigs of RAM and so on and so on.

And that is a good place to end this video. We now have a way to send a request to a model running locally on your machine and we have some levers that we can pull to alter how that model runs. This again is a crash course. There is a lot you can go play around with. We really do recommend you go and do that because that is the best way that you learn how you know all those knobs and little dials and how you can prompt and everything like that to make this a better experience. But this setup can be taken and integrated into larger agentic systems that you might be building. We've essentially built a nice little building block here. Also, we're going to be expanding from here with some future videos, so keep an eye out for that. As always, if you made anything cool with this or you just need a hand with anything from this tutorial, feel free to head on over to our community forums and post about it. Until next time though, happy making.