The Hugging Face Hub

The Hugging Face Hub is the website where most open-weight models are published. An open-weight model is one whose trained numbers (its weights) can be downloaded and run on your own machine. Every model on the Hub lives in a repository, and a repository is a git repository: a folder of files with a history of commits, plus branches and tags. If you know git, you already know the basic idea. If you do not, think of it as a folder that remembers every earlier version of itself and gives each version a name.

Running a model locally starts with downloading the right files from such a repository. This lesson goes through each file that a local-LLM tool reads and says what it is for. It then shows how to ask for one exact version of a repository, how to prove who you are for restricted models, how to download with the hf command and with llama.cpp, how to work without a network, and which files can run code on your machine when loaded. After it, you can open any model repository and say which files an engine reads and why.

You do not need network access for the examples. Each Java program parses a JSON document written inline in its source, in the shape that the Hub and the huggingface_hub client (the official Python library for the Hub, version 2.1.1, read for this lesson) use. The numbers in the example files are illustrative unless the text names their source. The Hub HTTP API and Cache Layout in Java makes the real HTTP requests.

Example 1: the files of a model repository and what each is for

A model repository holds many files, and an inference engine (a program that runs a model) needs only some of them. This program takes the file list of a typical small instruct model published in safetensors format and prints what each file is for. Read the right-hand column of the output as a short glossary for the rest of the lesson.

import java.util.List;

public class RepoFiles {
    static String purpose(String name) {
        if (name.equals("config.json")) return "architecture and hyperparameters";
        if (name.equals("generation_config.json")) return "default sampling settings and stop tokens";
        if (name.equals("tokenizer.json")) return "full tokenizer: vocabulary, merges, pre-tokenizer, special tokens";
        if (name.equals("tokenizer_config.json")) return "tokenizer settings and the chat template";
        if (name.equals("vocab.json") || name.equals("merges.txt")) return "BPE vocabulary and merges (older format)";
        if (name.endsWith(".safetensors")) return "weights";
        if (name.endsWith(".safetensors.index.json")) return "map from tensor name to shard file";
        if (name.endsWith(".gguf")) return "weights, tokenizer, and metadata in one llama.cpp file";
        if (name.endsWith(".bin") || name.endsWith(".pt") || name.endsWith(".pth")) return "weights in PyTorch pickle format";
        if (name.endsWith(".py")) return "custom model code (runs only with trust_remote_code)";
        if (name.equals("README.md")) return "model card: license, usage, YAML metadata";
        if (name.equals(".gitattributes")) return "which files git stores with LFS";
        if (name.startsWith("LICENSE")) return "license text";
        return "other";
    }

    public static void main(String[] args) {
        List<String> files = List.of(".gitattributes", "LICENSE", "README.md", "config.json",
                "generation_config.json", "merges.txt", "model.safetensors", "tokenizer.json",
                "tokenizer_config.json", "vocab.json");
        for (String f : files) System.out.printf("%-24s %s%n", f, purpose(f));
    }
}

.gitattributes           which files git stores with LFS
LICENSE                  license text
README.md                model card: license, usage, YAML metadata
config.json              architecture and hyperparameters
generation_config.json   default sampling settings and stop tokens
merges.txt               BPE vocabulary and merges (older format)
model.safetensors        weights
tokenizer.json           full tokenizer: vocabulary, merges, pre-tokenizer, special tokens
tokenizer_config.json    tokenizer settings and the chat template
vocab.json               BPE vocabulary and merges (older format)

Three groups of files appear in almost every model repository:

• Model description: config.json says which architecture to build and with which sizes. generation_config.json gives default generation settings.
• Tokenizer: the tokenizer turns text into numbers (token ids) that the model reads. tokenizer.json is the complete tokenizer in the format of the Hugging Face tokenizers library. vocab.json and merges.txt hold the same BPE (byte-pair encoding) data in the older GPT-2 format, and repositories often ship both. tokenizer_config.json adds settings and, in most current models, the chat template.
• Weights: one model.safetensors file, or several shards (model-00001-of-00004.safetensors and so on) with model.safetensors.index.json mapping each tensor name to its shard. A shard is one piece of a file that was cut into parts because it was too large.

The other files are for git and for people. .gitattributes tells git which file patterns are stored with Git LFS (Large File Storage), the git extension that keeps large binary files outside the normal git objects. The repository itself then holds a small pointer file for each of them (the next lesson parses one). README.md is the model card: a YAML header (structured settings at the top of the file) with the license and tags, followed by documentation.

An engine such as vLLM reads the first two groups. llama.cpp reads none of them directly. Making, Quantizing, and Testing Your Own GGUF showed that convert_hf_to_gguf.py copies everything it needs into one .gguf file.

Example 2: config.json field by field

config.json describes the shape of the model: how many layers it has, how wide they are, how many attention heads it uses. An engine reads it before loading any weights, because it must know what to build. This program parses a config in the format of Qwen2-architecture models and calculates the numbers that The Transformer Block and the Full Model and Memory Math use. Look at the parameter count and the file size at the end of the output.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Locale;

public class ConfigJson {
    public static void main(String[] args) throws Exception {
        String json = """
            {
              "architectures": ["Qwen2ForCausalLM"],
              "model_type": "qwen2",
              "hidden_size": 896,
              "intermediate_size": 4864,
              "num_hidden_layers": 24,
              "num_attention_heads": 14,
              "num_key_value_heads": 2,
              "max_position_embeddings": 32768,
              "rope_theta": 1000000.0,
              "rms_norm_eps": 1e-06,
              "hidden_act": "silu",
              "vocab_size": 151936,
              "tie_word_embeddings": true,
              "torch_dtype": "bfloat16",
              "bos_token_id": 151643,
              "eos_token_id": 151645
            }
            """;
        JsonNode c = new ObjectMapper().readTree(json);
        long d = c.get("hidden_size").asLong(), ff = c.get("intermediate_size").asLong();
        long layers = c.get("num_hidden_layers").asLong(), vocab = c.get("vocab_size").asLong();
        int heads = c.get("num_attention_heads").asInt(), kvHeads = c.get("num_key_value_heads").asInt();
        long headDim = d / heads;
        System.out.println("architecture: " + c.get("architectures").get(0).asText() + " (" + c.get("model_type").asText() + ")");
        System.out.println("head_dim = " + d + " / " + heads + " = " + headDim + ", query heads per KV head = " + heads / kvHeads);
        long attn = 2 * d * d + 2 * d * kvHeads * headDim;          // q, o and k, v projections
        long mlp = 3 * d * ff;                                       // gate, up, down
        long embed = vocab * d * (c.get("tie_word_embeddings").asBoolean() ? 1 : 2);
        long total = layers * (attn + mlp) + embed;
        System.out.println(String.format(Locale.ROOT, "parameters (no biases or norms): %.3f billion", total / 1e9));
        System.out.println(String.format(Locale.ROOT, "weights at %s: %.2f GiB", c.get("torch_dtype").asText(), total * 2 / Math.pow(2, 30)));
        System.out.println("context " + c.get("max_position_embeddings").asInt() + ", rope_theta " + c.get("rope_theta").asDouble()
                + ", eos " + c.get("eos_token_id").asInt());
    }
}

architecture: Qwen2ForCausalLM (qwen2)
head_dim = 896 / 14 = 64, query heads per KV head = 7
parameters (no biases or norms): 0.494 billion
weights at bfloat16: 0.92 GiB
context 32768, rope_theta 1000000.0, eos 151645

The fields of the config, in the order they appear:

The derived values in the output follow from these fields, and you can check each one by hand.

• The head dimension is 896 / 14 = 64. The hidden size is split evenly across the 14 query heads.
• 14 query heads share 2 KV heads, so each KV head serves 14 / 2 = 7 query heads.
• The parameter estimate adds up the matrices of one layer and the shared embedding. One layer has two d x d projections, two d x (kv_heads * head_dim) projections, and three feed-forward matrices. The embedding is counted once because the weights are tied (tie_word_embeddings is true). The total is 0.494 billion, which is why such a model is called "0.5B".
• At 2 bytes per BF16 weight, the file is about 0.92 GiB.

Some configs nest these values. Multimodal models (models that also read images) often put the language model's fields under text_config, and models with RoPE scaling add a rope_scaling object. A program that reads config.json must handle missing fields with defaults rather than assume every key exists.

Example 3: generation_config.json: default sampling and EOS id lists

When a model generates text, it must pick each next token from a list of probabilities. The rules for picking are called sampling settings, and the model's publisher recommends values for them. They live in generation_config.json, along with the tokens that tell the engine to stop. This program reads such a file and handles eos_token_id (the id of the end-of-sequence token) either as one number or as a list.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;

public class GenerationConfig {
    static List<Integer> ids(JsonNode n) {                      // an int or a list of ints
        List<Integer> out = new ArrayList<>();
        if (n == null || n.isNull()) return out;
        if (n.isArray()) n.forEach(v -> out.add(v.asInt())); else out.add(n.asInt());
        return out;
    }

    public static void main(String[] args) throws Exception {
        String json = """
            {
              "bos_token_id": 151643,
              "pad_token_id": 151643,
              "do_sample": true,
              "eos_token_id": [151645, 151643],
              "repetition_penalty": 1.1,
              "temperature": 0.7,
              "top_p": 0.8,
              "top_k": 20
            }
            """;
        JsonNode g = new ObjectMapper().readTree(json);
        System.out.println("stop on any of: " + ids(g.get("eos_token_id")));
        System.out.println("sampling: do_sample=" + g.path("do_sample").asBoolean(false)
                + " temperature=" + g.path("temperature").asDouble(1.0)
                + " top_p=" + g.path("top_p").asDouble(1.0)
                + " top_k=" + g.path("top_k").asInt(50)
                + " repetition_penalty=" + g.path("repetition_penalty").asDouble(1.0));
        System.out.println("min_p present: " + g.has("min_p"));
    }
}

stop on any of: [151645, 151643]
sampling: do_sample=true temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1
min_p present: false

generation_config.json holds the defaults that the Transformers generate method uses when a caller does not override them: do_sample, temperature, top_p, top_k, and repetition_penalty. These are the samplers of Sampling and Penalties, Logit Bias, and Sampler Chains. Model publishers set them to the values they recommend for the model. An engine that does not read this file uses its own defaults instead.

eos_token_id is the field to read with care. In config.json it is often a single id. Here it is a list, [151645, 151643], which means generation stops at either token. An instruct model usually ends a turn with an end-of-turn token (<|im_end|> here) and also has a document-end token (<|endoftext|>). A client that stops only on the id from config.json can keep generating past the end of the answer. The helper ids() accepts both forms, which is what every reader of this file needs.

Missing keys get defaults. path("top_k").asInt(50) returns 50 when the key is absent, which is the Transformers default. The last output line shows a key the file does not set (min_p), so the engine's own default applies.

Example 4: tokenizer.json: model type, vocabulary, merges, pre-tokenizer, added tokens

A tokenizer needs several pieces: a vocabulary, rules for combining characters, and special tokens. tokenizer.json stores all of them in one file. This program parses a shortened tokenizer.json with an 8-entry vocabulary. It then applies the merges to the word "hello", as Building a BPE Tokenizer did in lesson 7. A merge is a rule that joins two neighboring pieces into one larger piece. Watch how the list of pieces shrinks after each merge.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;

public class TokenizerJson {
    public static void main(String[] args) throws Exception {
        String json = """
            {
              "version": "1.0",
              "added_tokens": [
                {"id": 8, "content": "<|endoftext|>", "special": true},
                {"id": 9, "content": "<|im_start|>", "special": true}
              ],
              "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": false},
              "model": {
                "type": "BPE",
                "vocab": {"h": 0, "e": 1, "l": 2, "o": 3, "he": 4, "ll": 5, "hell": 6, "hello": 7},
                "merges": [["h", "e"], ["l", "l"], ["he", "ll"], ["hell", "o"]]
              }
            }
            """;
        JsonNode t = new ObjectMapper().readTree(json);
        JsonNode model = t.get("model");
        System.out.println("model type: " + model.get("type").asText() + ", vocab size: " + model.get("vocab").size()
                + ", merges: " + model.get("merges").size());
        System.out.println("pre-tokenizer: " + t.get("pre_tokenizer").get("type").asText());
        for (JsonNode a : t.get("added_tokens")) System.out.println("added token " + a.get("id") + " " + a.get("content").asText());

        List<String> parts = new ArrayList<>(List.of("h", "e", "l", "l", "o"));
        for (JsonNode m : model.get("merges")) {                    // "a b" strings or ["a", "b"] pairs
            String left = m.isArray() ? m.get(0).asText() : m.asText().split(" ")[0];
            String right = m.isArray() ? m.get(1).asText() : m.asText().split(" ")[1];
            for (int i = 0; i + 1 < parts.size(); i++)
                if (parts.get(i).equals(left) && parts.get(i + 1).equals(right)) { parts.set(i, left + right); parts.remove(i + 1); i--; }
            System.out.println("after merge " + left + " + " + right + ": " + parts);
        }
        System.out.println("token id: " + model.get("vocab").get(parts.get(0)).asInt());
    }
}

model type: BPE, vocab size: 8, merges: 4
pre-tokenizer: ByteLevel
added token 8 <|endoftext|>
added token 9 <|im_start|>
after merge h + e: [he, l, l, o]
after merge l + l: [he, ll, o]
after merge he + ll: [hell, o]
after merge hell + o: [hello]
token id: 7

The top-level keys of a tokenizer.json are:

• model holds the algorithm ("type": "BPE", or "WordPiece", "Unigram"), the vocab object that maps token text to an id, and for BPE the ordered merges list.
• pre_tokenizer describes how text is split before BPE runs. Byte-level BPE models use ByteLevel, often inside a Sequence with a regex Split.
• added_tokens lists tokens added on top of the vocabulary, with their ids and a special flag. Chat markers such as <|im_start|> live here.
• normalizer, post_processor, and decoder describe Unicode normalization, the tokens added around a sequence, and how ids turn back into text.

Merges come in two spellings. Older files store each merge as one string with a space ("h e"). Files written by recent versions of the tokenizers library store a two-element array (["h", "e"]), which also allows tokens that contain spaces. The program accepts both.

In the output, the 4 merges are applied in order. h e l l o becomes he l l o, then he ll o, then hell o, and finally one token, hello, whose id in the vocabulary is 7. The order matters: the list is sorted so that the merge learned first is applied first. A real vocabulary has 32,000 to over 200,000 entries, and its merge list is almost as long.

Example 5: tokenizer_config.json: chat template, BOS, and EOS

A chat model expects its input in a fixed text layout, with markers that show who is speaking. The recipe for that layout is the chat template, and it is stored in tokenizer_config.json together with the special-token settings. This program reads the file, finds the id of the EOS token (end of sequence), and inspects the template.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Iterator;
import java.util.Map;

public class TokenizerConfig {
    public static void main(String[] args) throws Exception {
        String json = """
            {
              "add_bos_token": false,
              "bos_token": null,
              "eos_token": "<|im_end|>",
              "pad_token": "<|endoftext|>",
              "model_max_length": 131072,
              "added_tokens_decoder": {
                "151643": {"content": "<|endoftext|>", "special": true},
                "151644": {"content": "<|im_start|>", "special": true},
                "151645": {"content": "<|im_end|>", "special": true}
              },
              "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}"
            }
            """;
        JsonNode c = new ObjectMapper().readTree(json);
        String eos = c.get("eos_token").asText();
        int eosId = -1;
        for (Iterator<Map.Entry<String, JsonNode>> it = c.get("added_tokens_decoder").fields(); it.hasNext(); ) {
            Map.Entry<String, JsonNode> e = it.next();
            if (e.getValue().get("content").asText().equals(eos)) eosId = Integer.parseInt(e.getKey());
        }
        System.out.println("eos_token " + eos + " = id " + eosId);
        System.out.println("bos_token " + c.get("bos_token") + ", add_bos_token " + c.get("add_bos_token").asBoolean());
        String template = c.get("chat_template").asText();
        System.out.println("chat_template: " + template.length() + " characters, uses add_generation_prompt: "
                + template.contains("add_generation_prompt"));
        System.out.println(template.substring(0, 47));
    }
}

eos_token <|im_end|> = id 151645
bos_token null, add_bos_token false
chat_template: 196 characters, uses add_generation_prompt: true
{% for message in messages %}{{'<|im_start|>' +

tokenizer_config.json names special tokens by their text, not their id: "eos_token": "<|im_end|>". The id comes from added_tokens_decoder, a map from id (as a string key) to token. The program searches that map and finds 151645.

bos_token (beginning of sequence) is null and add_bos_token is false. This tokenizer adds no BOS token, while Llama-family tokenizers do. An engine that adds a BOS token where the model was trained without one feeds the model an input it never saw.

chat_template is a Jinja template, a short program in a text template language. The shortened template here loops over the messages and wraps each one in <|im_start|>role ... <|im_end|>. When add_generation_prompt is true, it appends <|im_start|>assistant, which tells the model that its turn begins. Chat Templates, Roles, and Special Tokens renders such templates in Java. Some repositories store the template in a separate chat_template.jinja file instead of this field. A tool that reads only tokenizer_config.json then finds no template.

model_max_length (131072 here) is a tokenizer setting and can differ from max_position_embeddings in config.json. The model config is the one that describes the model.

Example 6: revisions: branches, tags, commit hashes, and pinning

A repository changes over time. A publisher can fix a chat template or replace weights, and a download that worked last week may give different files today. To get exactly the same files every time, you need to refer to one exact version. This program parses a response in the shape of GET /api/models/{repo}/refs (the Hub request that lists a repository's branches and tags). It then builds a download URL for a branch and for a commit. The repository name and the commit hashes are illustrative.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class Revisions {
    public static void main(String[] args) throws Exception {
        // Shape of GET /api/models/{repo}/refs (fields read by huggingface_hub's list_repo_refs).
        String json = """
            {
              "branches": [
                {"name": "main", "ref": "refs/heads/main", "targetCommit": "7ae557604adf67be50417f59c2c2f167def9a775"},
                {"name": "dev", "ref": "refs/heads/dev", "targetCommit": "1d2c4b9f0e8a7d6c5b4a39281706f5e4d3c2b1a0"}
              ],
              "converts": [],
              "tags": [
                {"name": "v1.0", "ref": "refs/tags/v1.0", "targetCommit": "0f9e8d7c6b5a49382716051f2e3d4c5b6a798867"}
              ]
            }
            """;
        JsonNode refs = new ObjectMapper().readTree(json);
        for (String kind : new String[] {"branches", "tags"})
            for (JsonNode r : refs.get(kind))
                System.out.printf("%-8s %-5s -> %s%n", kind, r.get("name").asText(), r.get("targetCommit").asText());

        String repo = "example-org/example-model", file = "config.json";
        String main = refs.get("branches").get(0).get("targetCommit").asText();
        System.out.println("moving:  https://huggingface.co/" + repo + "/resolve/main/" + file);
        System.out.println("pinned:  https://huggingface.co/" + repo + "/resolve/" + main + "/" + file);
        System.out.println("full 40-hex commit: " + main.matches("[0-9a-f]{40}"));
    }
}

branches main  -> 7ae557604adf67be50417f59c2c2f167def9a775
branches dev   -> 1d2c4b9f0e8a7d6c5b4a39281706f5e4d3c2b1a0
tags     v1.0  -> 0f9e8d7c6b5a49382716051f2e3d4c5b6a798867
moving:  https://huggingface.co/example-org/example-model/resolve/main/config.json
pinned:  https://huggingface.co/example-org/example-model/resolve/7ae557604adf67be50417f59c2c2f167def9a775/config.json
full 40-hex commit: true

A revision is anything git can resolve to a commit. There are three kinds:

• A branch (main, dev) is a movable name. resolve/main/config.json returns whatever main points to today.
• A tag (v1.0) is a name that usually stays on one commit.
• A commit hash is 40 hexadecimal characters and never changes.

The output shows each branch and tag with the commit it points to. The two URLs differ only in one path segment: main in the first, the 40-character hash in the second. The first can change its answer tomorrow. The second cannot.

The JSON has branches, tags, and converts arrays (and pullRequests when requested). Each entry has name, ref, and targetCommit, which are the fields list_repo_refs reads in huggingface_hub. llama.cpp reads the same endpoint. get_repo_commit in common/hf-cache.cpp takes the targetCommit of the main branch, checks that it is 40 hex characters (is_valid_commit), and writes it to refs/main in its cache.

Pinning means downloading with the commit hash instead of the branch name, as in the second URL. A pinned download gives the same bytes every time. This makes a setup reproducible and protects it from an unreviewed change upstream. The hf download --revision option and the revision argument of every huggingface_hub download function take any of the three forms.

Example 7: GGUF repositories: one file per quant, naming, and split files

GGUF repositories look different from safetensors repositories. They hold many weight files, one for each quantization (a way of storing the weights with fewer bits), and they have no config.json. A tool that downloads from such a repository must therefore choose one file. This program re-implements find_best_model and gguf_filename_is_model from common/download.cpp in llama.cpp and applies them to a file list. Compare the four requests in main with the four lines of output.

import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class GgufRepoNames {
    static final Pattern SPLIT = Pattern.compile("-(\\d{5})-of-(\\d{5})\\.gguf$");

    static boolean isModel(String f) {                          // gguf_filename_is_model in llama.cpp
        return f.endsWith(".gguf") && !f.contains("mmproj") && !f.contains("imatrix");
    }

    static boolean firstShardOrWhole(String f) {
        Matcher m = SPLIT.matcher(f);
        return !m.find() || Integer.parseInt(m.group(1)) == 1;
    }

    static String pick(List<String> files, String tag) {         // find_best_model in llama.cpp
        List<String> tags = tag.isEmpty() ? List.of("Q4_K_M", "Q8_0") : List.of(tag);
        for (String t : tags) {
            Pattern p = Pattern.compile(Pattern.quote(t) + "[.-]", Pattern.CASE_INSENSITIVE);
            for (String f : files) if (isModel(f) && p.matcher(f).find() && firstShardOrWhole(f)) return f;
        }
        if (tag.isEmpty()) for (String f : files) if (isModel(f) && firstShardOrWhole(f)) return f;
        return null;
    }

    public static void main(String[] args) {
        List<String> files = List.of("README.md", "imatrix.gguf", "mmproj-model-f16.gguf",
                "Model-7B-Instruct-IQ4_XS.gguf", "Model-7B-Instruct-Q4_K_M.gguf", "Model-7B-Instruct-Q6_K.gguf",
                "Model-7B-Instruct-Q8_0-00001-of-00002.gguf", "Model-7B-Instruct-Q8_0-00002-of-00002.gguf");
        for (String tag : new String[] {"", "q6_k", "Q8_0", "Q2_K"})
            System.out.printf("tag %-6s -> %s%n", tag.isEmpty() ? "(none)" : tag, pick(files, tag));
        Matcher m = SPLIT.matcher(files.get(7));
        if (m.find()) System.out.println("shard " + Integer.parseInt(m.group(1)) + " of " + Integer.parseInt(m.group(2)));
    }
}

tag (none) -> Model-7B-Instruct-Q4_K_M.gguf
tag q6_k   -> Model-7B-Instruct-Q6_K.gguf
tag Q8_0   -> Model-7B-Instruct-Q8_0-00001-of-00002.gguf
tag Q2_K   -> null
shard 2 of 2

The rules, as written in the pinned llama.cpp source, are:

1. Only .gguf files count, and files whose names contain mmproj (the vision projector of multimodal models) or imatrix (an importance matrix, not a model) are skipped. The source also skips draft-model files (mtp-, eagle3-, dflash-, dspark-). The Java version leaves those out for brevity.
2. With a tag such as q6_k, the file name must match the regular expression TAG[.-], case-insensitively. The [.-] after the tag prevents Q4_K from matching Q4_K_M.
3. Without a tag, the tags Q4_K_M and then Q8_0 are tried, and then the first model file of any kind.
4. For split files, only shard 1 qualifies (-00001-of-00002.gguf). The loader opens the other shards itself.

The output follows the rules:

• No tag selects Q4_K_M, the first default.
• q6_k matches Q6_K despite the different case.
• Q8_0 selects the first shard of a two-part file.
• Q2_K matches nothing and returns null. llama.cpp reports this as an error with a list of the available files (Example 10).

The last line prints the shard number and count parsed from the split name. The split pattern -NNNNN-of-NNNNN.gguf is the one llama-gguf-split writes.

Example 8: gated models, access tokens, and HF_TOKEN

Some models are gated. Their files are available only to accounts that have accepted the publisher's conditions on the model page. A program that downloads them has to prove that it acts for such an account. It does this with an access token, a secret string created in the Hub account settings. This program implements the token lookup order of get_token() in huggingface_hub and llama.cpp's token format check. It runs the lookup three times with different environment variables.

import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Map;

public class TokenLookup {
    // huggingface_hub get_token(): HF_TOKEN, then HUGGING_FACE_HUB_TOKEN, then the token file.
    static String token(Map<String, String> env, Path home) throws Exception {
        for (String key : new String[] {"HF_TOKEN", "HUGGING_FACE_HUB_TOKEN"})
            if (env.containsKey(key) && !env.get(key).isBlank()) return env.get(key).trim();
        Path hfHome = env.containsKey("HF_HOME") ? Path.of(env.get("HF_HOME"))
                : Path.of(env.getOrDefault("XDG_CACHE_HOME", home.resolve(".cache").toString()), "huggingface");
        Path file = env.containsKey("HF_TOKEN_PATH") ? Path.of(env.get("HF_TOKEN_PATH")) : hfHome.resolve("token");
        return Files.exists(file) ? Files.readString(file).trim() : null;
    }

    static boolean llamaCppAccepts(String t) {                  // is_valid_hf_token in llama.cpp
        return t != null && t.length() >= 37 && t.length() <= 256 && t.startsWith("hf_") && t.substring(3).matches("[A-Za-z0-9]+");
    }

    static String masked(String t) { return t == null ? "none" : t.substring(0, 6) + "..." + t.substring(t.length() - 2); }

    public static void main(String[] args) throws Exception {
        Path home = Files.createTempDirectory("home");
        Files.createDirectories(home.resolve(".cache/huggingface"));
        Files.writeString(home.resolve(".cache/huggingface/token"), "hf_FileTokenAbcdefghijklmnopqrstuvwxyz12\n");
        String envToken = "hf_EnvTokenAbcdefghijklmnopqrstuvwxyz123";
        for (Map<String, String> env : new Map[] {Map.of(), Map.of("HF_TOKEN", envToken), Map.of("HF_HOME", "/nonexistent")}) {
            String t = token(env, home);
            System.out.println(env.keySet() + " -> " + masked(t) + ", llama.cpp accepts: " + llamaCppAccepts(t));
        }
        System.out.println("header: Authorization: Bearer " + masked(envToken));
    }
}

[] -> hf_Fil...12, llama.cpp accepts: true
[HF_TOKEN] -> hf_Env...23, llama.cpp accepts: true
[HF_HOME] -> none, llama.cpp accepts: false
header: Authorization: Bearer hf_Env...23

Requests send the token as Authorization: Bearer <token>. For gated or private repositories, the token must belong to an account with access.

huggingface_hub looks for a token in this order (from get_token and the constants module):

1. An OIDC exchange when HF_OIDC_RESOURCE is set (used by CI systems, which are automated build servers).
2. The environment variable HF_TOKEN, then the older HUGGING_FACE_HUB_TOKEN.
3. The token file at HF_TOKEN_PATH, which defaults to $HF_HOME/token. HF_HOME defaults to $XDG_CACHE_HOME/huggingface or ~/.cache/huggingface. The command hf auth login writes this file.

The output shows the order working. With no variables, the file token is used. With HF_TOKEN set, it overrides the file. When HF_HOME points to an empty directory, nothing is found.

llama.cpp reads HF_TOKEN too, and -hft / --hf-token overrides it. It sends the token only if it passes is_valid_hf_token: the token must start with hf_, be 37 to 256 characters long, and contain only letters and digits after the prefix. Otherwise it logs "invalid token, authentication disabled".

Without a valid token, a request for a gated repository fails. huggingface_hub turns a response with the header X-Error-Code: GatedRepo into a GatedRepoError ("Cannot access gated repo"). A 401 on a repository API URL without that header is reported as "Repository Not Found". A comment in the client's _http.py explains that the Hub returns 401 both for missing repositories and for private or gated ones when the request is not authenticated. These status codes are described from the client's error handling in huggingface_hub 2.1.1, not from a published specification of the Hub API.

Tokens are secrets. The program masks them before printing, and tools should never write them to logs.

Example 9: the hf CLI: login and download

The hf command-line tool comes with the huggingface_hub package. It covers the two jobs you do most often: signing in and downloading files. The commands below go from the simplest case, one whole repository, to a filtered download of one quantization into a chosen directory and a download pinned to a commit. Text after # is a comment.

pip install huggingface_hub
hf auth login                      # browser login or a pasted token; stores it in $HF_HOME/token
hf auth whoami                     # shows the account the token belongs to
hf download Qwen/Qwen2.5-Coder-0.5B-Instruct                      # whole repository into the cache
hf download Qwen/Qwen2.5-Coder-0.5B-Instruct config.json tokenizer.json
hf download Qwen/Qwen2.5-Coder-7B-Instruct-GGUF --include "*q4_k_m*.gguf" --local-dir ./models
hf download Qwen/Qwen2.5-Coder-0.5B-Instruct --revision <40-character-commit>

The subcommands and options come from hf --help, hf auth --help, and hf download --help of huggingface_hub 2.1.1:

Without --local-dir, files go into the shared cache (~/.cache/huggingface/hub by default). The cache has a fixed layout with blobs/ (the file contents), snapshots/<commit>/ (one folder per version, with links to the blobs), and refs/ (which commit each branch name points to). Every tool that uses huggingface_hub shares this cache, so a second download of the same file is skipped. The Hub HTTP API and Cache Layout in Java builds this layout in Java.

The pattern *q4_k_m*.gguf in the GGUF example depends on how the publisher names files. Glob matching is case-sensitive, so the pattern must match the actual names.

Example 10: llama-cli -hf repo:quant direct download

llama.cpp can download GGUF files itself, without the hf command. The -hf option takes a repository name and an optional quantization. The first command below shows the normal online use. The next two commands run against a hand-built cache with --offline, which shows the same file-selection logic without needing a network.

# online (repository name from the pinned llama.cpp README): picks a file, downloads it into the cache, runs it
llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# offline replay: a cache built by hand with one file, tiny-Q8_0.gguf, for the repo "demo/tiny-GGUF"
LLAMA_CACHE=$PWD/fakecache llama-completion -hf demo/tiny-GGUF:Q8_0 --offline -no-cnv -p "Hello" -n 4 -v
LLAMA_CACHE=$PWD/fakecache llama-completion -hf demo/tiny-GGUF:Q4_K_M --offline -no-cnv -p "Hello" -n 4

The hand-built cache has the layout llama.cpp expects. The file models--demo--tiny-GGUF/refs/main holds a 40-character commit, and snapshots/<commit>/tiny-Q8_0.gguf is a symbolic link (a file that points to another file) to blobs/<sha256 of the file>. The first block of output below is representative output from the pinned build with the random-weight tiny model. Timestamps are removed, the path is shortened, and the first run is abridged to the line that names the loaded file, which -v (verbose) makes visible.

llama_model_loader: loaded meta data with 22 key-value pairs and 39 tensors from /tmp/fakecache/models--demo--tiny-GGUF/snapshots/0123456789abcdef0123456789abcdef01234567/tiny-Q8_0.gguf (version GGUF V3 (latest))

The first run found tiny-Q8_0.gguf through the snapshot path and loaded it. The second run asks for a quantization that is not in the cache:

E common_download_get_hf_plan: no GGUF files found in repository demo/tiny-GGUF
I Available GGUF files:
I  - tiny-Q8_0.gguf
error: --model is required

The error lists the files that are available, so you can pick a name that exists.

The -hf help text at the pinned commit describes the argument as <user>/<model>[:quant]. The quant "is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist". An mmproj file is downloaded too when the repository has one (--no-mmproj turns that off). The source tries Q8_0 after Q4_K_M before falling back, as Example 7 showed. -hff / --hf-file names an exact file instead.

Online, llama.cpp asks the API for the main commit (/api/models/{repo}/refs) and for the file list (/api/models/{repo}/tree/{commit}?recursive=true). It then downloads from {endpoint}{repo}/resolve/{commit}/{file}. The cache directory uses the Hugging Face layout, so a file downloaded by hf download is found by llama-cli -hf, and the other way around. --offline skips the API and uses only the cache, which is how the replay above worked without a network.

Example 11: offline mode and alternative endpoints

Sometimes a machine has no network access, or a company runs its own copy of the Hub (a mirror). Both tools need to be told. This program works out the endpoint (the base URL of the Hub), the cache directory, and the offline flag the way huggingface_hub and llama.cpp do. It does this for four sets of environment variables, so you can compare the two tools side by side.

import java.util.Map;
import java.util.Set;

public class HubEnvironment {
    static boolean isTrue(String v) { return v != null && Set.of("1", "ON", "YES", "TRUE").contains(v.toUpperCase()); }

    static String hubEndpoint(Map<String, String> env) {        // huggingface_hub constants.ENDPOINT
        return env.getOrDefault("HF_ENDPOINT", "https://huggingface.co").replaceAll("/+$", "");
    }

    static String llamaEndpoint(Map<String, String> env) {      // common_get_model_endpoint in llama.cpp
        String e = env.getOrDefault("MODEL_ENDPOINT", env.getOrDefault("HF_ENDPOINT", "https://huggingface.co/"));
        return e.endsWith("/") ? e : e + "/";
    }

    static String hubCache(Map<String, String> env, String home) {
        if (env.containsKey("HF_HUB_CACHE")) return env.get("HF_HUB_CACHE");
        String hfHome = env.getOrDefault("HF_HOME", env.getOrDefault("XDG_CACHE_HOME", home + "/.cache") + "/huggingface");
        return hfHome + "/hub";
    }

    static String llamaCache(Map<String, String> env, String home) {   // hf-cache.cpp get_cache_directory
        if (env.containsKey("LLAMA_CACHE")) return env.get("LLAMA_CACHE");
        if (env.containsKey("HF_HUB_CACHE")) return env.get("HF_HUB_CACHE");
        if (env.containsKey("HUGGINGFACE_HUB_CACHE")) return env.get("HUGGINGFACE_HUB_CACHE");
        if (env.containsKey("HF_HOME")) return env.get("HF_HOME") + "/hub";
        if (env.containsKey("XDG_CACHE_HOME")) return env.get("XDG_CACHE_HOME") + "/huggingface/hub";
        return home + "/.cache/huggingface/hub";
    }

    public static void main(String[] args) {
        String home = "/home/dev";
        Map<String, String>[] envs = new Map[] {
            Map.of(),
            Map.of("HF_HOME", "/data/hf", "HF_HUB_OFFLINE", "1"),
            Map.of("HF_ENDPOINT", "https://hub.internal.example/", "TRANSFORMERS_OFFLINE", "yes"),
            Map.of("LLAMA_CACHE", "/models", "HF_HUB_OFFLINE", "0")};
        for (Map<String, String> env : envs) {
            boolean offline = isTrue(env.get("HF_HUB_OFFLINE")) || (!env.containsKey("HF_HUB_OFFLINE") && isTrue(env.get("TRANSFORMERS_OFFLINE")));
            System.out.println(new java.util.TreeMap<>(env));
            System.out.println("  huggingface_hub: endpoint " + hubEndpoint(env) + ", cache " + hubCache(env, home) + ", offline " + offline);
            System.out.println("  llama.cpp:       endpoint " + llamaEndpoint(env) + ", cache " + llamaCache(env, home));
        }
    }
}

{}
  huggingface_hub: endpoint https://huggingface.co, cache /home/dev/.cache/huggingface/hub, offline false
  llama.cpp:       endpoint https://huggingface.co/, cache /home/dev/.cache/huggingface/hub
{HF_HOME=/data/hf, HF_HUB_OFFLINE=1}
  huggingface_hub: endpoint https://huggingface.co, cache /data/hf/hub, offline true
  llama.cpp:       endpoint https://huggingface.co/, cache /data/hf/hub
{HF_ENDPOINT=https://hub.internal.example/, TRANSFORMERS_OFFLINE=yes}
  huggingface_hub: endpoint https://hub.internal.example, cache /home/dev/.cache/huggingface/hub, offline true
  llama.cpp:       endpoint https://hub.internal.example/, cache /home/dev/.cache/huggingface/hub
{HF_HUB_OFFLINE=0, LLAMA_CACHE=/models}
  huggingface_hub: endpoint https://huggingface.co, cache /home/dev/.cache/huggingface/hub, offline false
  llama.cpp:       endpoint https://huggingface.co/, cache /models

The variables and their effects are:

The four runs in the output show the resolution:

• With no variables, both tools use the default endpoint and the default cache.
• Setting HF_HOME moves both caches. HF_HUB_OFFLINE=1 switches huggingface_hub to offline mode.
• HF_ENDPOINT changes both tools' base URL. TRANSFORMERS_OFFLINE=yes switches huggingface_hub to offline mode because HF_HUB_OFFLINE is absent.
• LLAMA_CACHE moves only llama.cpp's cache, and HF_HUB_OFFLINE=0 keeps huggingface_hub online.

Offline mode is useful on machines without internet access, in CI, and to guarantee that a run uses exactly the cached revision. An alternative endpoint is used for internal mirrors and proxies that serve the same API. A client must then send the same paths (/api/models/..., /{repo}/resolve/...) to the new base URL.

Example 12: safety: trust_remote_code, pickle files, and the publisher

A model repository is code and data written by someone else, and loading it runs it on your machine. Most files only hold data, but a few kinds can run code. This program scans a file list and a config.json for the two risky patterns. The output has one line for each risky file, so check which file names appear.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;

public class RepoSafetyScan {
    public static void main(String[] args) throws Exception {
        List<String> files = List.of("config.json", "configuration_custom.py", "modeling_custom.py",
                "pytorch_model.bin", "model.safetensors", "tokenizer.json", "training_args.bin");
        String config = """
            {"architectures": ["CustomForCausalLM"],
             "auto_map": {"AutoConfig": "configuration_custom.CustomConfig",
                          "AutoModelForCausalLM": "modeling_custom.CustomForCausalLM"}}
            """;
        JsonNode c = new ObjectMapper().readTree(config);
        for (String f : files) {
            if (f.endsWith(".bin") || f.endsWith(".pt") || f.endsWith(".pth") || f.endsWith(".pkl") || f.endsWith(".ckpt"))
                System.out.println("pickle file: " + f + " (loading it with torch.load can run code)");
            else if (f.endsWith(".py"))
                System.out.println("python file: " + f + " (runs only with trust_remote_code=True)");
        }
        if (c.has("auto_map"))
            c.get("auto_map").fields().forEachRemaining(e ->
                System.out.println("auto_map " + e.getKey() + " -> " + e.getValue().asText()));
        boolean hasSafetensors = files.stream().anyMatch(f -> f.endsWith(".safetensors"));
        System.out.println("safetensors weights available: " + hasSafetensors);
    }
}

python file: configuration_custom.py (runs only with trust_remote_code=True)
python file: modeling_custom.py (runs only with trust_remote_code=True)
pickle file: pytorch_model.bin (loading it with torch.load can run code)
pickle file: training_args.bin (loading it with torch.load can run code)
auto_map AutoConfig -> configuration_custom.CustomConfig
auto_map AutoModelForCausalLM -> modeling_custom.CustomForCausalLM
safetensors weights available: true

Two kinds of files can run code:

• Pickle files (.bin, .pt, .pth, .pkl, .ckpt). PyTorch's traditional checkpoint format is Python's pickle, and unpickling can call any Python function. A crafted checkpoint can therefore run commands when it is loaded. The safetensors Format shows why. PyTorch's torch.load restricts what it unpickles when weights_only=True, which is the default since PyTorch 2.6 according to its release notes. Older versions, and code that passes weights_only=False, load the pickle in full. training_args.bin, a file many fine-tuning repositories contain, is also a pickle.
• Python files with auto_map. When config.json has an auto_map that points to modules in the repository (modeling_custom.CustomForCausalLM), Transformers loads that code only if the caller passes trust_remote_code=True. The code then runs with the caller's permissions.

safetensors and GGUF files contain only tensors and metadata, and their readers do not execute anything from the file. A repository that offers them can be used without loading any pickle. llama.cpp never runs Python from a repository.

Run these checks before downloading from an unfamiliar repository:

Interview points

• A Hub model is a git repository with LFS-stored weights. Engines read config.json, the tokenizer files, and safetensors weights, while llama.cpp reads one GGUF file with all of them inside.
• config.json gives the architecture and sizes, and the head dimension, the GQA ratio (query heads per KV head), and the parameter count follow from it.
• eos_token_id can be a list in generation_config.json, and stopping on only one id runs past the end of turns.
• tokenizer.json holds the model type, vocabulary, merges (as strings or pairs), pre-tokenizer, and added tokens. tokenizer_config.json holds the chat template and the special-token names.
• Branches move, tags rarely move, and commit hashes never move, so pinning a commit makes downloads reproducible.
• llama.cpp's -hf repo:quant matches QUANT[.-] case-insensitively, defaults to Q4_K_M and then Q8_0, skips mmproj and imatrix files, and uses only shard 1 of split files.
• Tokens come from HF_TOKEN first and then from the token file in HF_HOME, and llama.cpp sends only tokens that start with hf_.
• A 401 from the Hub can mean missing, private, or gated without authentication, and the header X-Error-Code: GatedRepo identifies gated access.
• HF_HUB_OFFLINE=1 stops network calls in huggingface_hub, llama.cpp uses --offline, and both share the models--org--name cache layout.
• Pickle checkpoints and trust_remote_code are the two ways a repository can run code on load.

Summary

• Weights, tokenizer, and configuration are separate files in a Transformers repository and one file in a GGUF repository.
• .gitattributes lists the patterns stored with Git LFS.
• A Qwen2-format config with 896 hidden size, 24 layers, and tied embeddings gives about 0.494 billion parameters.
• generation_config.json carries the publisher's default sampling settings.
• Merges in tokenizer.json are either "a b" strings or ["a", "b"] pairs.
• added_tokens_decoder maps special token ids to their text.
• bos_token: null with add_bos_token: false means no BOS token is added.
• The refs endpoint returns branches, tags, and converts with targetCommit hashes.
• A resolve URL with a commit hash always returns the same bytes.
• GGUF split files are named -NNNNN-of-NNNNN.gguf, and tools select shard 1.
• hf download takes --include, --exclude, --revision, --cache-dir, and --local-dir.
• LLAMA_CACHE and MODEL_ENDPOINT are llama.cpp-specific, while HF_HOME, HF_HUB_CACHE, and HF_ENDPOINT are shared.
• safetensors and GGUF files cannot execute code when read, and pickle files can.