Lesson 31 of 73
Downloading Large Files Reliably in Java
A quantized 7B model (a model with 7 billion numbers, stored in a compressed form) is a file of 4 to 8 GB, and a 70B model is several files of that size. A web page or a small JSON answer arrives in a moment. A file this large takes minutes or hours, and many things can go wrong in that time. The connection drops at 60%. The server is busy and refuses the request. The disk fills up. The file on the server is replaced by a newer version halfway through.
This lesson builds the download code of loop pull (the command from the course's loop tool that fetches a model) one problem at a time, using java.net.http.HttpClient, the HTTP client that comes with the JDK. Each example solves one problem and shows it with a small program. The last example combines all of them in one class.
After the lesson, a reader can write a download that resumes where it stopped after an interruption, never accepts a corrupted or mixed file, and shows the user progress. The table lists each problem in plain words, the technique that solves it, and the example that shows it.
| Concern | Technique | Example |
|---|---|---|
| Saving a body | BodyHandlers.ofFile, ofInputStream |
1, 4 |
| Hub downloads redirect | Redirect.NORMAL, Location |
2 |
| Hanging servers | connect and request timeouts | 3 |
| Progress needs a length | Content-Length vs chunked |
5 |
| Partial content | Range, 206, Content-Range |
6 |
| Interrupted downloads | resume from a .part file |
7 |
| Files replaced on the server | ETag and If-Range |
8 |
| Corruption | SHA-256 while streaming | 9 |
| Speed | parallel ranges on virtual threads | 10 |
| Temporary errors | retries with backoff and jitter | 11 |
| Crashes | .part plus atomic rename |
12 |
| Full disks | FileStore.getUsableSpace |
13 |
| Feedback | a progress bar with an injected clock | 14 |
| All of the above | a Downloader class |
15 |
To make the examples safe and repeatable, every network example starts its own small web server inside the program. The server is a com.sun.net.httpserver.HttpServer that listens on 127.0.0.1 (the address of the same machine) with port 0, which tells the operating system to pick any free port. Nothing leaves the machine, and no real download is needed. The Hub HTTP API and Cache Layout in Java (lesson 26 of the course, cited below by its number) introduced this fake-server technique, and the sockets lesson covers HttpClient basics.
The "model files" in the examples are random bytes from java.util.Random with a fixed seed. A fixed seed makes the generator produce the same sequence every time, so file sizes and hashes are the same on every run.
Example 1: GET to a file with BodyHandlers.ofFile
The first problem is how to save a response body to disk without holding all of it in memory. A 5 GB model does not fit in the memory of most programs. In HttpClient, a body handler decides what happens to the response body as it arrives. BodyHandlers.ofFile(path) writes the body into a file piece by piece.
The program starts a server that holds 100,000 bytes under one name and answers 404 (not found) for every other name. It downloads the file, then shows two traps of this handler: what it does with an error response, and what it does when the file already exists.
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.Random;
import static java.nio.file.StandardOpenOption.*;
public class GetToFile {
public static void main(String[] args) throws Exception {
byte[] data = new byte[100_000];
new Random(1).nextBytes(data); // the "model file" the server holds
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/files/", ex -> {
boolean known = ex.getRequestURI().getPath().equals("/files/model.gguf");
byte[] body = known ? data : "Entry not found".getBytes(StandardCharsets.UTF_8);
ex.sendResponseHeaders(known ? 200 : 404, body.length);
try (var out = ex.getResponseBody()) { out.write(body); }
});
server.start();
String base = "http://127.0.0.1:" + server.getAddress().getPort() + "/files/";
HttpClient client = HttpClient.newHttpClient();
Path dir = Files.createTempDirectory("downloads");
Path target = dir.resolve("model.gguf");
HttpResponse<Path> ok = client.send(HttpRequest.newBuilder(URI.create(base + "model.gguf")).build(), BodyHandlers.ofFile(target));
System.out.println("status " + ok.statusCode() + ", content-length " + ok.headers().firstValue("content-length").orElse("?")
+ ", file " + ok.body().getFileName() + " " + Files.size(target) + " bytes, identical: " + Arrays.equals(Files.readAllBytes(target), data));
Path missing = dir.resolve("missing.gguf");
HttpResponse<Path> notFound = client.send(HttpRequest.newBuilder(URI.create(base + "missing.gguf")).build(), BodyHandlers.ofFile(missing));
System.out.println("status " + notFound.statusCode() + ", file exists: " + Files.exists(missing) + ", content: " + Files.readString(missing));
Files.write(target, new byte[250_000]); // an older, larger file with the same name
client.send(HttpRequest.newBuilder(URI.create(base + "model.gguf")).build(), BodyHandlers.ofFile(target));
System.out.println("ofFile(path) over a larger file: " + Files.size(target) + " bytes");
client.send(HttpRequest.newBuilder(URI.create(base + "model.gguf")).build(), BodyHandlers.ofFile(target, CREATE, WRITE, TRUNCATE_EXISTING));
System.out.println("with TRUNCATE_EXISTING: " + Files.size(target) + " bytes");
server.stop(0);
}
}status 200, content-length 100000, file model.gguf 100000 bytes, identical: true
status 404, file exists: true, content: Entry not found
ofFile(path) over a larger file: 250000 bytes
with TRUNCATE_EXISTING: 100000 bytesThe server holds 100,000 bytes for /files/model.gguf and answers 404 Entry not found for any other name. The Hub sends the same text for a missing file.
The four output lines tell this story:
- The first download returns status 200,
content-length 100000, and a file of 100,000 bytes identical to the server's data.ok.body()is thePaththe handler wrote to. - The second request returns 404, and the file
missing.ggufexists anyway, containingEntry not found. A body handler runs for every status code;ofFilewrites whatever body comes back. A downloader must check the status code before treating the file as the model. - The third download writes the same 100,000 bytes over an existing 250,000-byte file. The result is still 250,000 bytes.
ofFile(path)opens the file withCREATEandWRITEonly. That overwrites the start of the file and leaves the old tail in place. The file now holds the correct first 100,000 bytes followed by 150,000 bytes of the older file, which is a broken file that looks fine from its name. - With the options
CREATE, WRITE, TRUNCATE_EXISTING, the file is cut to zero length before writing, and the result is 100,000 bytes.
Both traps lead to the same rule for the rest of the lesson. Write into a temporary file, check the status, the size, and the hash, and only then give the file its final name (Example 12).
Example 2: redirects, Redirect.NORMAL against NEVER
A Hub resolve URL does not serve the file itself. It answers 302 Found with a Location header that points at a storage server (lesson 26, Example 7). This is a redirect: the server says "the thing you want is at this other address, ask there". The question is what HttpClient does about it. Its redirect policy decides whether the client follows the Location automatically.
The program starts a server with a redirecting address and a storage address. It sends the same request with three different clients, one with the default policy, one with NEVER, and one with NORMAL, and prints what each one sees.
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpClient.Redirect;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
public class Redirects {
public static void main(String[] args) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/org/model/resolve/main/model.gguf", ex -> { // like the Hub: redirect to storage
ex.getResponseHeaders().add("Location", "/cdn/sha256-ab12?expires=1760000000&signature=c0ffee");
ex.sendResponseHeaders(302, -1);
ex.close();
});
server.createContext("/cdn/", ex -> {
byte[] body = "GGUF...".getBytes();
ex.sendResponseHeaders(200, body.length);
try (var out = ex.getResponseBody()) { out.write(body); }
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/org/model/resolve/main/model.gguf");
HttpRequest request = HttpRequest.newBuilder(uri).build();
HttpClient defaults = HttpClient.newHttpClient();
System.out.println("default policy: " + defaults.followRedirects());
HttpResponse<String> r1 = defaults.send(request, BodyHandlers.ofString());
System.out.println(" status " + r1.statusCode() + ", Location: " + r1.headers().firstValue("location").orElse("-"));
HttpClient never = HttpClient.newBuilder().followRedirects(Redirect.NEVER).build();
HttpResponse<String> r2 = never.send(request, BodyHandlers.ofString());
URI next = r2.uri().resolve(r2.headers().firstValue("location").orElseThrow()); // a relative Location
System.out.println("NEVER: status " + r2.statusCode() + ", next URI path " + next.getPath() + "?" + next.getQuery());
HttpClient normal = HttpClient.newBuilder().followRedirects(Redirect.NORMAL).build();
HttpResponse<String> r3 = normal.send(request, BodyHandlers.ofString());
System.out.println("NORMAL: status " + r3.statusCode() + ", final path " + r3.uri().getPath() + ", body " + r3.body());
System.out.println(" previous response: " + r3.previousResponse().map(p -> p.statusCode() + " " + p.uri().getPath()).orElse("none"));
server.stop(0);
}
}default policy: NEVER
status 302, Location: /cdn/sha256-ab12?expires=1760000000&signature=c0ffee
NEVER: status 302, next URI path /cdn/sha256-ab12?expires=1760000000&signature=c0ffee
NORMAL: status 200, final path /cdn/sha256-ab12, body GGUF...
previous response: 302 /org/model/resolve/main/model.ggufThe output lines show how the three policies differ:
default policy: NEVER: a client fromHttpClient.newHttpClient()does not follow redirects. A download with default settings returns the 302 response, whose body is empty, which is a common cause of zero-byte downloads.- With
NEVER, the program reads theLocationheader itself. The value here is relative (/cdn/...), sor.uri().resolve(...)turns it into an absolute URI on the same server. The query string carries the expiry time and signature of a signed URL; the storage server checks them instead of a token. - With
NORMAL, the client follows the redirect, and the final status is 200.r3.uri()is the URI of the final response, andpreviousResponse()holds the 302 response that led there.
Redirect.NORMAL follows every redirect except one from https to http. That one is skipped because it would send the request over an unencrypted connection. Redirect.ALWAYS follows that redirect too.
NORMAL also limits the number of attempts per request. The limit is the jdk.httpclient.redirects.retrylimit system property, 5 by default. Against a test server that redirected to itself, JDK 21 and JDK 25 sent 5 requests and then returned the last 302 response instead of throwing an exception. A caller therefore still has to check the status.
A downloader that wants to log the storage host, or to control which host receives its Authorization header (the header that carries the access token), uses NEVER and follows the Location itself. huggingface_hub, the Hub's Python library, does this for HEAD requests (lesson 26, Example 7).
Example 3: connect and request timeouts
A server that accepts the connection and then sends nothing would leave a download waiting forever. A timeout is a limit on how long the program waits before it gives up. HttpClient has two. The connect timeout limits the wait for the connection to be set up. The request timeout limits the wait for the response.
The question is exactly what each of them covers. The program sends requests with a 1-second request timeout to two servers. One is slow to send the response headers (the status line and header fields that come before the body). The other sends the headers at once and then delivers its body slowly. A third request goes to a port where nothing listens.
import com.sun.net.httpserver.HttpServer;
import java.net.ConnectException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse.BodyHandlers;
import java.net.http.HttpTimeoutException;
import java.time.Duration;
import java.util.concurrent.Executors;
public class Timeouts {
static void pause(long ms) { try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } }
public static void main(String[] args) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor()); // slow handlers must not block each other
server.createContext("/slow-headers", ex -> { pause(4000); ex.sendResponseHeaders(200, -1); ex.close(); });
server.createContext("/slow-body", ex -> {
ex.sendResponseHeaders(200, 5);
try (var out = ex.getResponseBody()) { for (int i = 0; i < 5; i++) { out.write('x'); out.flush(); pause(600); } }
});
server.start();
String base = "http://127.0.0.1:" + server.getAddress().getPort();
HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build();
for (String path : new String[] {"/slow-headers", "/slow-body"}) {
HttpRequest request = HttpRequest.newBuilder(URI.create(base + path)).timeout(Duration.ofSeconds(1)).build();
try {
String body = client.send(request, BodyHandlers.ofString()).body();
System.out.println(path + ": completed, body " + body + " (the body took about 3 s, longer than the 1 s timeout)");
} catch (HttpTimeoutException e) {
System.out.println(path + ": " + e.getClass().getSimpleName() + ": " + e.getMessage());
}
}
int closedPort;
try (ServerSocket s = new ServerSocket(0, 1, InetAddress.getLoopbackAddress())) { closedPort = s.getLocalPort(); }
try {
client.send(HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + closedPort + "/")).build(), BodyHandlers.discarding());
} catch (ConnectException e) {
System.out.println("closed port: " + e.getClass().getSimpleName() + " (refused at once, no timeout involved)");
}
server.stop(0);
}
}/slow-headers: HttpTimeoutException: request timed out
/slow-body: completed, body xxxxx (the body took about 3 s, longer than the 1 s timeout)
closed port: ConnectException (refused at once, no timeout involved)HttpServer runs handlers on the thread that accepts connections unless it is given an executor; setExecutor(Executors.newVirtualThreadPerTaskExecutor()) runs each request on its own virtual thread (Virtual Threads), so a sleeping handler does not block the next request.
The output lines show three different outcomes:
/slow-headerswaits 4 seconds before sending the status line. The request timeout of 1 second expires first, andsendthrowsHttpTimeoutException: request timed out./slow-bodysends its headers at once and then one byte every 600 ms, about 3 seconds for 5 bytes. The request completes even though it took longer than the 1-second timeout. The request timeout covers the wait for the response headers, not the time to read the body; the program behaves the same on JDK 21 and JDK 25.- A request to a port where nothing listens fails at once with
ConnectException. The connect timeout (2 seconds here, set withconnectTimeouton the client) applies only when the connection attempt gets no answer at all, for example when a firewall drops packets; a refused connection is an answer.
For a multi-gigabyte body, a limit on the total time does not fit, because a slow but working connection may need an hour. A downloader uses a stall detector instead. If no byte arrives for a set time, say 60 seconds, it aborts the transfer and retries. With ofInputStream (Example 4), a separate scheduled task can close the stream when the byte count stops growing. Closing the stream makes the blocked read call fail with an IOException. A test on JDK 21 and JDK 25, with a server that sent one byte and then paused, showed a read blocked in one thread failing with IOException: closed 300 ms after another thread closed the stream.
Example 4: streaming with ofInputStream and a progress callback
To show progress and to compute a hash while downloading, the program has to see the data as it arrives, a buffer at a time. A buffer is a small array of bytes, here 64 KiB. The question is how to read a body as a stream. BodyHandlers.ofInputStream() returns the response as soon as the headers arrive and gives the body as an InputStream, which the program then reads in a loop.
The program downloads 1,000,000 bytes. A method called copy reads buffers, writes them to a file, and reports the running total to a callback. The callback prints a line each time another 25% has arrived.
import com.sun.net.httpserver.HttpServer;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import java.util.function.LongConsumer;
public class StreamProgress {
/** Copies in to out and reports the running byte count after every buffer. */
static long copy(InputStream in, OutputStream out, LongConsumer progress) throws Exception {
byte[] buffer = new byte[64 * 1024];
long done = 0;
for (int n; (n = in.read(buffer)) >= 0; ) {
out.write(buffer, 0, n);
done += n;
progress.accept(done);
}
return done;
}
public static void main(String[] args) throws Exception {
byte[] data = new byte[1_000_000];
new Random(2).nextBytes(data);
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/model.gguf", ex -> {
ex.sendResponseHeaders(200, data.length);
try (var out = ex.getResponseBody()) { out.write(data); }
});
server.start();
HttpResponse<InputStream> response = HttpClient.newHttpClient().send(
HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf")).build(),
BodyHandlers.ofInputStream()); // returns once the headers have arrived
long total = response.headers().firstValueAsLong("content-length").orElse(-1);
System.out.println("status " + response.statusCode() + ", total " + total + " bytes");
Path target = Files.createTempFile("model", ".gguf");
int[] lastQuarter = {0};
long copied;
try (InputStream in = response.body(); OutputStream out = Files.newOutputStream(target)) {
copied = copy(in, out, done -> {
int quarter = (int) (done * 4 / total); // 0..4: report each 25% step once
if (quarter > lastQuarter[0]) {
lastQuarter[0] = quarter;
System.out.println(" " + quarter * 25 + "%");
}
});
}
System.out.println("copied " + copied + " bytes, file " + Files.size(target) + " bytes");
server.stop(0);
}
}status 200, total 1000000 bytes
25%
50%
75%
100%
copied 1000000 bytes, file 1000000 bytessend returns once the status line and the headers are in. At that point the program knows the status (200) and the length from content-length (1,000,000 bytes), and it has read none of the body yet. The copy method then reads up to 64 KiB at a time, writes the bytes to the file, and calls the LongConsumer (a function that takes one long) with the running total.
The callback turns the running total into a quarter from 0 to 4. For example, 600,000 bytes of 1,000,000 gives 600000 * 4 / 1000000 = 2, so the download is at the 50% step. The callback prints only when the quarter grows, so the output has exactly one line per 25% step. The number of read calls between those lines depends on how the network delivers the data, which is why the program does not print byte counts at each step: they would differ between runs. The last line confirms that 1,000,000 bytes were copied and the file has that size.
The stream must be closed, here by the try-with-resources block. Closing it before the end tells the client to abandon the rest of the body and release the connection.
Example 5: known length against chunked transfer
Example 4 computed percentages from the Content-Length header, which tells the client how many bytes the body has. The question is what happens when the server does not send it. A server that does not know the length in advance, for example because it generates the data while sending, uses chunked transfer encoding. The body is sent as a series of chunks, each with its own size, and a zero-length chunk marks the end.
The program starts a server with two addresses. One sends a fixed length, and the other uses chunked encoding. For each address the client prints the two relevant headers and the number of bytes received.
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
public class KnownLength {
public static void main(String[] args) throws Exception {
byte[] data = new byte[300_000];
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/fixed", ex -> { // length known before sending
ex.sendResponseHeaders(200, data.length);
try (var out = ex.getResponseBody()) { out.write(data); }
});
server.createContext("/chunked", ex -> { // 0 means: length unknown, use chunked encoding
ex.sendResponseHeaders(200, 0);
try (var out = ex.getResponseBody()) { for (int i = 0; i < 3; i++) out.write(data, 0, 100_000); }
});
server.start();
String base = "http://127.0.0.1:" + server.getAddress().getPort();
HttpClient client = HttpClient.newHttpClient();
for (String path : new String[] {"/fixed", "/chunked"}) {
HttpResponse<byte[]> r = client.send(HttpRequest.newBuilder(URI.create(base + path)).build(), BodyHandlers.ofByteArray());
System.out.printf("%-8s content-length=%s transfer-encoding=%s received=%d%n", path,
r.headers().firstValue("content-length").orElse("(none)"),
r.headers().firstValue("transfer-encoding").orElse("(none)"), r.body().length);
}
server.stop(0);
}
}/fixed content-length=300000 transfer-encoding=(none) received=300000
/chunked content-length=(none) transfer-encoding=chunked received=300000In com.sun.net.httpserver, the second argument of sendResponseHeaders selects how the body is framed. A positive number sends Content-Length with that value, 0 selects chunked encoding, and -1 means no body.
The two output lines show the difference:
/fixedhascontent-length=300000and notransfer-encoding. The client knows the total before the first byte arrives and can show a percentage and a time estimate./chunkedhas nocontent-lengthandtransfer-encoding=chunked. The client receives the same 300,000 bytes, but it learns the total only when the last chunk arrives.
For a chunked download, a progress display can show the bytes received and the rate, but not a percentage (Example 14 handles both cases). For Hub downloads, the expected size is also known from the tree listing, a Hub API answer that lists the files of a repository (the fields size and lfs.size). So loop pull uses that number when the header is missing, and it compares the number with the bytes received at the end.
Example 6: Range requests, 206 Partial Content, and Content-Range
Resuming a download needs one basic ability: asking the server for only part of a file. A range request does this with the header Range: bytes=first-last, where first and last are byte positions counted from 0. For example, bytes=0-99 asks for the first 100 bytes.
The question is how a server answers and how a client checks the answer. A server that supports ranges replies 206 Partial Content, with a Content-Range header that states which bytes the body holds and how large the whole file is.
The program starts a server that supports single ranges. Its file holds the byte values 0, 1, 2, and so on, so each byte reveals its own position. The client sends four range requests, including two unusual ones, and checks each answer.
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.util.Arrays;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class RangeRequests {
static final Pattern RANGE = Pattern.compile("bytes=(\\d+)-(\\d*)");
static final Pattern CONTENT_RANGE = Pattern.compile("bytes (\\d+)-(\\d+)/(\\d+|\\*)");
public static void main(String[] args) throws Exception {
byte[] data = new byte[1000];
for (int i = 0; i < data.length; i++) data[i] = (byte) i;
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/model.gguf", ex -> { // a server that supports single byte ranges
ex.getResponseHeaders().add("Accept-Ranges", "bytes");
String range = ex.getRequestHeaders().getFirst("Range");
Matcher m = range == null ? null : RANGE.matcher(range);
if (m == null || !m.matches()) {
ex.sendResponseHeaders(200, data.length);
try (var out = ex.getResponseBody()) { out.write(data); }
return;
}
long start = Long.parseLong(m.group(1));
long end = m.group(2).isEmpty() ? data.length - 1 : Math.min(Long.parseLong(m.group(2)), data.length - 1);
if (start >= data.length) {
ex.getResponseHeaders().add("Content-Range", "bytes */" + data.length);
ex.sendResponseHeaders(416, -1);
ex.close();
return;
}
ex.getResponseHeaders().add("Content-Range", "bytes " + start + "-" + end + "/" + data.length);
ex.sendResponseHeaders(206, end - start + 1);
try (var out = ex.getResponseBody()) { out.write(data, (int) start, (int) (end - start + 1)); }
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf");
HttpClient client = HttpClient.newHttpClient();
for (String range : new String[] {"bytes=0-99", "bytes=990-", "bytes=500-2000", "bytes=1000-"}) {
HttpResponse<byte[]> r = client.send(HttpRequest.newBuilder(uri).header("Range", range).build(), BodyHandlers.ofByteArray());
String cr = r.headers().firstValue("content-range").orElse("(none)");
System.out.printf("%-15s -> %d, Content-Range: %-17s body %4d bytes", range, r.statusCode(), cr, r.body().length);
Matcher m = CONTENT_RANGE.matcher(cr);
if (r.statusCode() == 206 && m.matches()) {
long first = Long.parseLong(m.group(1));
System.out.printf(", first=%d last=%d total=%s, starts with byte %d, matches: %b", first, Long.parseLong(m.group(2)),
m.group(3), r.body()[0] & 0xFF,
Arrays.equals(r.body(), Arrays.copyOfRange(data, (int) first, (int) first + r.body().length)));
}
System.out.println();
}
server.stop(0);
}
}bytes=0-99 -> 206, Content-Range: bytes 0-99/1000 body 100 bytes, first=0 last=99 total=1000, starts with byte 0, matches: true
bytes=990- -> 206, Content-Range: bytes 990-999/1000 body 10 bytes, first=990 last=999 total=1000, starts with byte 222, matches: true
bytes=500-2000 -> 206, Content-Range: bytes 500-999/1000 body 500 bytes, first=500 last=999 total=1000, starts with byte 244, matches: true
bytes=1000- -> 416, Content-Range: bytes */1000 body 0 bytesThe server's file holds the byte values 0, 1, 2, ... 255, 0, 1, ... so each byte's value is its position modulo 256, and the value of a received byte shows where it came from.
The four requests show the normal case and the edge cases:
bytes=0-99returns 206 withContent-Range: bytes 0-99/1000and 100 bytes. Both ends are inclusive, so0-99is 100 bytes, and the number after the slash is the full size of the file.bytes=990-has no end, which means "to the end of the file": bytes 990 to 999, 10 bytes. The first byte has the value 222 because 990 modulo 256 is 222.bytes=500-2000asks for more than exists. The server shortens the range to the end of the file and says so inContent-Range: bytes 500-999/1000. A client must use theContent-Rangeit receives, not theRangeit sent.bytes=1000-starts at the file's size. No byte satisfies it, so the server answers416 Range Not SatisfiablewithContent-Range: bytes */1000, which still tells the client the size.
The Accept-Ranges: bytes header announces that the server supports ranges. The client parses Content-Range with the pattern bytes (\d+)-(\d+)/(\d+|\*) and checks that the body matches the data at the stated offset, which is the matches: true on each 206 line. These are the rules of RFC 9110, the standard that defines HTTP semantics (section 14).
Example 7: resuming from a .part file, and a server that ignores Range
An interrupted download leaves a partial file on disk. Starting over wastes the part that already arrived, which can be gigabytes. The question is how to continue instead. If the partial file has N bytes, the client asks for Range: bytes=N- (everything from byte N to the end) and appends the answer to the file.
There is one catch. A server is allowed to ignore Range and send the whole file with status 200. A client that appended that answer would corrupt the file, so it has to look at the status code and handle both answers.
The program creates a partial file with the first 80,000 bytes and resumes it against two servers. One supports ranges, and the other ignores them.
import com.sun.net.httpserver.HttpServer;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.Random;
import static java.nio.file.StandardOpenOption.*;
public class ResumePart {
static byte[] data = new byte[200_000];
/** Continues the download in part; returns a description of what happened. */
static String resume(HttpClient client, URI uri, Path part) throws Exception {
long have = Files.exists(part) ? Files.size(part) : 0;
HttpRequest request = HttpRequest.newBuilder(uri).header("Range", "bytes=" + have + "-").build();
HttpResponse<InputStream> r = client.send(request, BodyHandlers.ofInputStream());
try (InputStream in = r.body()) {
if (r.statusCode() == 206) {
String cr = r.headers().firstValue("content-range").orElseThrow();
if (!cr.startsWith("bytes " + have + "-")) throw new IllegalStateException("unexpected " + cr);
try (OutputStream out = Files.newOutputStream(part, APPEND)) { in.transferTo(out); }
return "206, appended from byte " + have;
}
if (r.statusCode() == 200) { // Range ignored: the body is the whole file
try (OutputStream out = Files.newOutputStream(part, CREATE, TRUNCATE_EXISTING, WRITE)) { in.transferTo(out); }
return "200, Range ignored, rewrote the file from byte 0";
}
throw new IllegalStateException("status " + r.statusCode());
}
}
public static void main(String[] args) throws Exception {
new Random(3).nextBytes(data);
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/ranged", ex -> {
String range = ex.getRequestHeaders().getFirst("Range"); // only "bytes=N-" is supported here
int start = range == null ? 0 : Integer.parseInt(range.substring(6, range.length() - 1));
if (start > 0) ex.getResponseHeaders().add("Content-Range", "bytes " + start + "-" + (data.length - 1) + "/" + data.length);
ex.sendResponseHeaders(start > 0 ? 206 : 200, data.length - start);
try (var out = ex.getResponseBody()) { out.write(data, start, data.length - start); }
});
server.createContext("/plain", ex -> { // ignores Range, as it may
ex.sendResponseHeaders(200, data.length);
try (var out = ex.getResponseBody()) { out.write(data); }
});
server.start();
String base = "http://127.0.0.1:" + server.getAddress().getPort();
HttpClient client = HttpClient.newHttpClient();
for (String path : new String[] {"/ranged", "/plain"}) {
Path part = Files.createTempDirectory("dl").resolve("model.gguf.part");
Files.write(part, Arrays.copyOf(data, 80_000)); // an earlier attempt stopped at 80,000 bytes
String what = resume(client, URI.create(base + path), part);
System.out.println(path + ": " + what + "; size " + Files.size(part) + ", identical: " + Arrays.equals(Files.readAllBytes(part), data));
}
server.stop(0);
}
}/ranged: 206, appended from byte 80000; size 200000, identical: true
/plain: 200, Range ignored, rewrote the file from byte 0; size 200000, identical: trueThe partial file is named model.gguf.part, a common convention: the final name appears only when the download is complete (Example 12). Both runs start with a partial file holding the first 80,000 bytes.
The two output lines show the two possible answers:
/rangedanswers 206 withContent-Range: bytes 80000-199999/200000. The client checks that the range starts where its file ends, opens the file withAPPEND, and writes the 120,000 new bytes. The result is 200,000 bytes, identical to the server's file./plainignoresRangeand answers 200 with the full 200,000 bytes. Appending them would produce a 280,000-byte file with the first 80,000 bytes twice. The client sees the status 200 and opens the file withTRUNCATE_EXISTING, so the full body replaces the partial content.
The Content-Range check guards against a third case: a server that answers 206 with a range other than the one requested. Appending such a body would put bytes at the wrong position, so the client stops with an error instead.
Example 8: ETag and If-Range, so a resumed file never mixes two versions
A model repository can change between two download attempts. For example, the file model.gguf on the main branch may be replaced by a new quantization. A client that resumes with a plain range request would then glue the first part of the old file to the rest of the new file. The result has the right size but is useless, and nothing in the download notices.
The question is how a resumed download avoids this. An ETag is an identifier of one version of a resource, which the server sends in the ETag header. Whenever the content changes, the ETag changes. The client saves the ETag with the partial file. When it resumes, it sends the saved value back in an If-Range header, which means "give me this range only if the file is still this version".
The program runs three cases, each starting with 60,000 bytes of version 1 on disk. In the first the file is unchanged, in the second it was replaced and the client sends If-Range, and in the third it was replaced and the client does not send If-Range.
import com.sun.net.httpserver.HttpServer;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.Random;
import static java.nio.file.StandardOpenOption.*;
public class IfRange {
static volatile byte[] current; // the file the server holds now
static volatile String currentEtag;
static byte[] version(int seed) { byte[] b = new byte[100_000]; new Random(seed).nextBytes(b); return b; }
/** Resumes part; sends If-Range only when useIfRange is true. */
static String resume(HttpClient client, URI uri, Path part, String savedEtag, boolean useIfRange) throws Exception {
long have = Files.size(part);
HttpRequest.Builder b = HttpRequest.newBuilder(uri).header("Range", "bytes=" + have + "-");
if (useIfRange) b.header("If-Range", savedEtag);
HttpResponse<InputStream> r = client.send(b.build(), BodyHandlers.ofInputStream());
boolean partial = r.statusCode() == 206;
try (InputStream in = r.body(); OutputStream out = partial ? Files.newOutputStream(part, APPEND)
: Files.newOutputStream(part, TRUNCATE_EXISTING, WRITE)) {
in.transferTo(out);
}
return r.statusCode() + " ETag " + r.headers().firstValue("etag").orElse("-");
}
public static void main(String[] args) throws Exception {
byte[] v1 = version(1), v2 = version(2);
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/model.gguf", ex -> {
byte[] data = current;
String range = ex.getRequestHeaders().getFirst("Range"), ifRange = ex.getRequestHeaders().getFirst("If-Range");
boolean honorRange = range != null && (ifRange == null || ifRange.equals(currentEtag)); // RFC 9110 If-Range rule
int start = honorRange ? Integer.parseInt(range.substring(6, range.length() - 1)) : 0;
ex.getResponseHeaders().add("ETag", currentEtag);
if (honorRange) ex.getResponseHeaders().add("Content-Range", "bytes " + start + "-" + (data.length - 1) + "/" + data.length);
ex.sendResponseHeaders(honorRange ? 206 : 200, data.length - start);
try (var out = ex.getResponseBody()) { out.write(data, start, data.length - start); }
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf");
HttpClient client = HttpClient.newHttpClient();
String[][] cases = {{"file unchanged, If-Range", "v1", "true"}, {"file replaced, If-Range", "v2", "true"},
{"file replaced, no If-Range", "v2", "false"}};
for (String[] c : cases) {
current = c[1].equals("v1") ? v1 : v2;
currentEtag = "\"" + c[1] + "-etag\"";
Path part = Files.createTempFile("model", ".part");
Files.write(part, Arrays.copyOf(v1, 60_000)); // 60,000 bytes of v1, ETag "v1-etag" saved with them
String status = resume(client, uri, part, "\"v1-etag\"", Boolean.parseBoolean(c[2]));
byte[] got = Files.readAllBytes(part);
String result = Arrays.equals(got, v1) ? "complete v1" : Arrays.equals(got, v2) ? "complete v2" : "MIXED v1 and v2";
System.out.printf("%-27s -> %s -> %s%n", c[0], status, result);
}
server.stop(0);
}
}file unchanged, If-Range -> 206 ETag "v1-etag" -> complete v1
file replaced, If-Range -> 200 ETag "v2-etag" -> complete v2
file replaced, no If-Range -> 206 ETag "v2-etag" -> MIXED v1 and v2The header If-Range: "v1-etag" means: "send the range only if the file is still the version with this ETag, and otherwise send the whole current file". The server implements that rule in one line (honorRange).
The three cases give these results:
- File unchanged, If-Range. The server's ETag is still
"v1-etag", so it honors the range and answers 206. The appended bytes come from the same version, and the file is a complete copy of v1. - File replaced, If-Range. The server now holds v2 with ETag
"v2-etag". The ETags differ, so it ignores the range and answers 200 with the whole v2 file. The client truncates and writes it, and the result is a complete v2. - File replaced, no If-Range. Without the header, the server honors the range of the new file. The client appends bytes 60,000 to 99,999 of v2 to bytes 0 to 59,999 of v1. The result has the right size and is neither version:
MIXED v1 and v2.
The third case is the reason for If-Range. Without it, the file looks complete, and only a hash check (the next example) would catch the error, after the whole download. With it, the server settles the question with one comparison.
The Hub offers a second protection. The X-Linked-Etag of an LFS file (a large file stored outside normal git storage) is its SHA-256 (lesson 26, Example 6). A resolve URL can also name a commit hash instead of main. That pins the version, which makes the problem impossible for that download.
Example 9: SHA-256 while streaming with DigestInputStream
Even when every request succeeds, a downloaded file can be wrong, for example after a disk error or a faulty proxy. The Hub's tree listing gives the SHA-256 of every LFS file (lfs.oid), so the client can check. A hash function such as SHA-256 reads any input and produces a fixed-size fingerprint (32 bytes, shown as 64 hex characters). Any change to the input changes the fingerprint completely.
The question is how to check a download against the expected fingerprint without reading the multi-gigabyte file a second time. A DigestInputStream wraps another stream and updates a MessageDigest (the Java class that computes the hash) with every byte that passes through it.
The program downloads two versions of the same 500,000 bytes. One is correct. In the other, a single bit is flipped. It compares each download's hash with the expected one.
import com.sun.net.httpserver.HttpServer;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.Random;
public class StreamingHash {
static String sha256(byte[] b) throws Exception { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(b)); }
public static void main(String[] args) throws Exception {
byte[] data = new byte[500_000];
new Random(4).nextBytes(data);
String expected = sha256(data); // what the Hub's tree listing reports as lfs.oid
byte[] corrupted = data.clone();
corrupted[123_456] ^= 1; // one flipped bit
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", ex -> {
byte[] body = ex.getRequestURI().getPath().equals("/good") ? data : corrupted;
ex.sendResponseHeaders(200, body.length);
try (var out = ex.getResponseBody()) { out.write(body); }
});
server.start();
HttpClient client = HttpClient.newHttpClient();
System.out.printf("%-8s %s%n", "expected", expected);
for (String path : new String[] {"/good", "/bad"}) {
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + path);
MessageDigest sha = MessageDigest.getInstance("SHA-256");
Path target = Files.createTempFile("model", ".part");
try (InputStream in = new DigestInputStream(client.send(HttpRequest.newBuilder(uri).build(), BodyHandlers.ofInputStream()).body(), sha);
OutputStream out = Files.newOutputStream(target)) {
in.transferTo(out); // every byte passes through the digest once
}
String actual = HexFormat.of().formatHex(sha.digest());
boolean ok = actual.equals(expected);
if (!ok) Files.delete(target); // never keep a file that failed the check
System.out.printf("%-8s %s -> %s%n", path, actual, ok ? "OK" : "MISMATCH, file deleted");
}
server.stop(0);
}
}expected baa83b4415d47a00894156bbe38c47d02aac6d98a6dae91d037c700a58710f94
/good baa83b4415d47a00894156bbe38c47d02aac6d98a6dae91d037c700a58710f94 -> OK
/bad 2e42c5dbbddc112b24e329536e5c8896eb65a4f56f2d684ffb364a8dac8e711f -> MISMATCH, file deletedThe program computes the expected hash from the server's data. In real use, that is the value a client reads from the tree listing.
The output lines show the check working:
/goodproduces exactly the expected value,baa83b44..., so the download is accepted./badserves the same 500,000 bytes with a single bit flipped at offset 123,456. The hash is completely different (2e42c5db...), not slightly different, so even one wrong bit is detected. The program deletes the file, because a file that failed the check must never be used.
DigestInputStream wraps the response stream, so the hash is computed as the bytes are read and written, with no second pass over the file. transferTo copies the stream to the file through an internal buffer until the end.
Resuming adds a complication. The hash must cover the whole file, including the bytes that were already on disk before the resume. A resuming downloader can either feed the .part file through the digest before it continues, or hash the complete file once at the end. The Downloader in Example 15 does the second. It costs one extra read of the file and keeps the code short.
Example 10: a parallel ranged download on virtual threads
A single HTTP connection is sometimes slower than the network link, for example when a server limits the speed of each connection. Several connections at once can then download faster. The question is how to download several parts of one file at the same time and put them together correctly.
The program splits a file into 4 ranges (Example 6) and fetches each range on its own virtual thread. A virtual thread is a lightweight thread of the JVM that is cheap to create, which suits tasks that mostly wait for the network. Each thread writes its bytes at the correct position in one shared FileChannel, which is a Java object for reading and writing a file at any position.
import com.sun.net.httpserver.HttpServer;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicInteger;
import static java.nio.file.StandardOpenOption.*;
public class ParallelChunks {
record Chunk(int index, long start, long end) {} // end is inclusive, as in a Range header
/** Downloads one chunk and writes it at its own position in the shared channel. */
static long fetch(HttpClient client, URI uri, Chunk c, FileChannel ch) throws Exception {
HttpRequest request = HttpRequest.newBuilder(uri).header("Range", "bytes=" + c.start() + "-" + c.end()).build();
HttpResponse<InputStream> r = client.send(request, BodyHandlers.ofInputStream());
if (r.statusCode() != 206) throw new IllegalStateException("chunk " + c.index() + ": status " + r.statusCode());
long position = c.start();
byte[] buf = new byte[64 * 1024];
try (InputStream in = r.body()) {
for (int n; (n = in.read(buf)) >= 0; ) {
ByteBuffer bb = ByteBuffer.wrap(buf, 0, n);
while (bb.hasRemaining()) position += ch.write(bb, position); // positional write: no shared file pointer
}
}
return position - c.start();
}
static String sha256(byte[] b) throws Exception { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(b)); }
public static void main(String[] args) throws Exception {
byte[] data = new byte[1_000_003]; // not a multiple of the chunk count
new Random(5).nextBytes(data);
AtomicInteger rangeRequests = new AtomicInteger();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
server.createContext("/model.gguf", ex -> {
String[] r = ex.getRequestHeaders().getFirst("Range").substring(6).split("-");
int start = Integer.parseInt(r[0]), end = Integer.parseInt(r[1]);
rangeRequests.incrementAndGet();
ex.getResponseHeaders().add("Content-Range", "bytes " + start + "-" + end + "/" + data.length);
ex.sendResponseHeaders(206, end - start + 1);
try (var out = ex.getResponseBody()) { out.write(data, start, end - start + 1); }
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf");
long size = data.length;
int n = 4;
long chunkSize = (size + n - 1) / n; // ceiling division
List<Chunk> chunks = new ArrayList<>();
for (int i = 0; i < n; i++) chunks.add(new Chunk(i, i * chunkSize, Math.min(size, (i + 1) * chunkSize) - 1));
Path target = Files.createTempFile("model", ".part");
HttpClient client = HttpClient.newHttpClient();
List<Future<Long>> results = new ArrayList<>();
try (FileChannel ch = FileChannel.open(target, WRITE);
var pool = Executors.newVirtualThreadPerTaskExecutor()) {
for (Chunk c : chunks) results.add(pool.submit(() -> fetch(client, uri, c, ch)));
for (int i = 0; i < n; i++) // get() waits; printed in chunk order
System.out.printf("chunk %d: bytes %7d-%7d, wrote %d%n", i, chunks.get(i).start(), chunks.get(i).end(), results.get(i).get());
}
boolean same = sha256(Files.readAllBytes(target)).equals(sha256(data));
System.out.println("range requests " + rangeRequests.get() + ", file " + Files.size(target) + " bytes, SHA-256 matches: " + same);
server.stop(0);
}
}chunk 0: bytes 0- 250000, wrote 250001
chunk 1: bytes 250001- 500001, wrote 250001
chunk 2: bytes 500002- 750002, wrote 250001
chunk 3: bytes 750003-1000002, wrote 250000
range requests 4, file 1000003 bytes, SHA-256 matches: trueThe plan splits 1,000,003 bytes into 4 chunks. The size is deliberately not a multiple of 4, so the program has to handle a remainder. It uses a ceiling division, which rounds up: chunkSize = (1000003 + 3) / 4 = 250001. The first three chunks hold 250,001 bytes each, and the last chunk holds the rest, 250,000. Each Chunk record has an inclusive end because that is the form the Range header uses, so chunk 0 is bytes=0-250000.
Each chunk is written with FileChannel.write(buffer, position). This is a positional write: it writes at the given offset and does not move the channel's shared position. Four threads can therefore write into the same channel at the same time without coordinating, because their byte ranges do not overlap. A write call may write fewer bytes than the buffer holds, so the inner while loop repeats until the buffer is empty.
The output lines show the result:
- Each chunk line comes from
results.get(i).get(), which waits for chunkiand returns how many bytes it wrote. The program prints in chunk order after waiting, so the order is the same on every run, whatever order the threads finish in. range requests 4confirms one request per chunk, and the SHA-256 of the assembled file matches the original.
The executor is created with Executors.newVirtualThreadPerTaskExecutor() in a try-with-resources block, and close() waits for all tasks. Virtual threads fit here because each thread spends most of its time waiting for the network.
Parallel downloads are not always faster. They help when the limit is per connection, and they add load on the server. A resume after a crash also needs to know which chunks finished. A single sequential .part file records that by its size alone, but four chunks written at once do not. loop pull therefore uses one connection by default and offers chunks as an option.
Example 11: retries with exponential backoff and seeded jitter
Some failures are temporary. A 503 comes from an overloaded server, a 429 means "too many requests, slow down", and a dropped connection may work on the next try. Other failures are permanent, because a 404 will be a 404 again. The question is which failures to retry and how long to wait between tries.
The program retries temporary failures with growing waits and stops at once on permanent ones. A scripted server answers the three downloads with a fixed list of statuses. To keep the run fast, the program does not really sleep. It records the waits instead.
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.util.ArrayDeque;
import java.util.Deque;
import java.util.List;
import java.util.Random;
import java.util.Set;
public class Retries {
interface Sleeper { void sleep(long millis) throws InterruptedException; }
static final Set<Integer> RETRYABLE = Set.of(408, 429, 500, 502, 503, 504);
static final long BASE_MS = 500, CAP_MS = 8_000;
static final int MAX_ATTEMPTS = 5;
/** Delay before retry number `retry` (1-based): exponential with "equal jitter", or the server's Retry-After. */
static long delay(int retry, HttpResponse<?> r, Random random) {
if (r != null) {
var retryAfter = r.headers().firstValue("retry-after");
if (retryAfter.isPresent() && retryAfter.get().matches("\\d+")) return Long.parseLong(retryAfter.get()) * 1000;
}
long exp = Math.min(CAP_MS, BASE_MS << (retry - 1)); // 500, 1000, 2000, 4000, 8000, 8000, ...
return exp / 2 + random.nextLong(exp / 2 + 1); // between exp/2 and exp
}
static String get(HttpClient client, URI uri, Sleeper sleeper, Random random) throws Exception {
for (int attempt = 1; ; attempt++) {
HttpResponse<String> r = null;
String outcome;
try {
r = client.send(HttpRequest.newBuilder(uri).build(), BodyHandlers.ofString());
if (r.statusCode() == 200) return "attempt " + attempt + ": 200 " + r.body();
if (!RETRYABLE.contains(r.statusCode())) return "attempt " + attempt + ": " + r.statusCode() + ", not retryable, giving up";
outcome = String.valueOf(r.statusCode());
} catch (IOException e) { // connection refused, reset, truncated body
outcome = e.getClass().getSimpleName();
}
if (attempt == MAX_ATTEMPTS) return "attempt " + attempt + ": " + outcome + ", attempts exhausted";
long wait = delay(attempt, r, random);
System.out.println(" attempt " + attempt + ": " + outcome + ", waiting " + wait + " ms");
sleeper.sleep(wait);
}
}
public static void main(String[] args) throws Exception {
Deque<Integer> script = new ArrayDeque<>(List.of(503, 502, 429, 200, 404, 503, 503, 503, 503, 503));
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", ex -> {
int status = script.removeFirst();
if (status == 429) ex.getResponseHeaders().add("Retry-After", "3");
byte[] body = status == 200 ? "ok".getBytes() : new byte[0];
ex.sendResponseHeaders(status, body.length == 0 ? -1 : body.length);
try (var out = ex.getResponseBody()) { out.write(body); }
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf");
HttpClient client = HttpClient.newHttpClient();
Random random = new Random(42);
long[] slept = {0};
Sleeper fake = ms -> slept[0] += ms; // records the waits instead of sleeping
for (String name : new String[] {"download 1", "download 2", "download 3"}) {
System.out.println(name + ":");
System.out.println(" " + get(client, uri, fake, random));
}
System.out.println("total waiting " + slept[0] + " ms");
server.stop(0);
}
}download 1:
attempt 1: 503, waiting 488 ms
attempt 2: 502, waiting 930 ms
attempt 3: 429, waiting 3000 ms
attempt 4: 200 ok
download 2:
attempt 1: 404, not retryable, giving up
download 3:
attempt 1: 503, waiting 341 ms
attempt 2: 503, waiting 753 ms
attempt 3: 503, waiting 1783 ms
attempt 4: 503, waiting 2504 ms
attempt 5: 503, attempts exhausted
total waiting 9799 msExponential backoff doubles the wait after each failure: 500 ms, 1 s, 2 s, 4 s, and so on, capped at 8 s. The growing wait gives an overloaded server time to recover. Jitter is a random part of the wait. Many clients that failed at the same moment would otherwise all retry at the same moment, and jitter spreads their retries out.
This program uses "equal jitter": half of the exponential value plus a random amount of up to the other half. For the first retry, the exponential value is 500 ms, so the wait is between 250 and 500 ms. The Random is seeded with 42, so the waits are the same on every run. A real client uses an unseeded random generator.
The three downloads show three different endings:
- Download 1 gets 503, then 502, then 429, then 200. The first wait is 488 ms, between 250 and 500. The second is 930 ms, between 500 and 1,000. The 429 response carries
Retry-After: 3, so the third wait is exactly 3,000 ms: when the server says how long to wait, its value replaces the computed one. - Download 2 gets 404, which is not in the retryable set, and gives up after one attempt with no wait.
- Download 3 gets 503 five times. The waits grow (341, 753, 1,783, 2,504 ms) and the fifth failure ends the loop with
attempts exhausted.
The retryable set is 408 (request timeout), 429, and the 5xx statuses 500, 502, 503, and 504; an IOException such as a reset connection is retried too. 400, 401, 403, and 404 describe the request, and repeating the same request gives the same answer. huggingface_hub uses a similar rule with different numbers (lesson 26, Example 14: waits from 1 to 8 seconds, plus 1 second added to Retry-After).
The Sleeper interface is the reason the example runs in milliseconds. The program passes a fake that records the waits, while a real client passes Thread::sleep. Passing the clock or the sleeper in from outside, instead of calling it directly, is also what makes such code testable.
Example 12: atomic move into place, and cleanup on failure
A download can fail in two ways that call for opposite cleanup. If the connection is lost, the bytes received so far are correct, and keeping them lets the next attempt resume. If the finished file has the wrong hash, the bytes are wrong, and keeping them would only cause trouble. The question is what to keep and what to delete.
A second problem is that other programs must never mistake a half-written file for a finished model. The solution is to write to a temporary name, model.gguf.part, and to rename it to model.gguf only after the hash matches.
The program tries three downloads, each in an empty directory: one where the connection drops, one that delivers corrupted bytes, and one that succeeds. After each, it lists what is in the directory.
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.Random;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class AtomicPlace {
static class ChecksumException extends IOException { ChecksumException(String m) { super(m); } }
/** Downloads into name.part and renames it to name only after the hash matches. */
static void download(HttpClient client, URI uri, Path target, String expectedSha) throws Exception {
Path part = target.resolveSibling(target.getFileName() + ".part"); // same directory, so the rename is atomic
MessageDigest sha = MessageDigest.getInstance("SHA-256");
try (InputStream in = new DigestInputStream(client.send(HttpRequest.newBuilder(uri).build(), BodyHandlers.ofInputStream()).body(), sha);
OutputStream out = Files.newOutputStream(part)) {
in.transferTo(out);
} // an IOException here leaves .part for a resume
if (!HexFormat.of().formatHex(sha.digest()).equals(expectedSha)) {
Files.delete(part); // wrong content: resuming would not help
throw new ChecksumException("SHA-256 mismatch for " + target.getFileName());
}
Files.move(part, target, StandardCopyOption.ATOMIC_MOVE);
}
static String listing(Path dir) throws IOException {
try (Stream<Path> s = Files.list(dir)) {
return s.map(p -> p.getFileName() + (p.toString().endsWith(".part") ? " (partial)" : " (" + p.toFile().length() + " bytes)"))
.sorted().collect(Collectors.joining(", "));
}
}
public static void main(String[] args) throws Exception {
byte[] data = new byte[50_000];
new Random(6).nextBytes(data);
String sha = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(data));
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", ex -> {
String path = ex.getRequestURI().getPath();
ex.sendResponseHeaders(200, data.length);
OutputStream out = ex.getResponseBody();
if (path.equals("/drop")) { out.write(data, 0, 20_000); out.flush(); throw new RuntimeException("connection lost"); }
byte[] body = path.equals("/corrupt") ? new byte[data.length] : data;
out.write(body);
out.close();
});
server.start();
String base = "http://127.0.0.1:" + server.getAddress().getPort();
HttpClient client = HttpClient.newHttpClient();
for (String path : new String[] {"/drop", "/corrupt", "/ok"}) {
Path dir = Files.createTempDirectory("models");
try {
download(client, URI.create(base + path), dir.resolve("model.gguf"), sha);
System.out.printf("%-9s success; directory: [%s]%n", path, listing(dir));
} catch (IOException e) {
System.out.printf("%-9s %s; directory: [%s]%n", path, e.getClass().getSimpleName(), listing(dir));
}
}
server.stop(0);
}
}/drop IOException; directory: [model.gguf.part (partial)]
/corrupt ChecksumException; directory: []
/ok success; directory: [model.gguf (50000 bytes)]The three runs end like this:
/dropsends 20,000 of 50,000 bytes and then the connection breaks. The client gets anIOExceptionwhile reading. The directory holdsmodel.gguf.partwith the bytes received so far, which a resume (Example 7) continues. Nomodel.ggufexists, so no other program mistakes the partial file for a model./corruptsends 50,000 zero bytes. The transfer succeeds, but the hash does not match, and the method deletes the.partfile and throwsChecksumException. Resuming would not help, because the bytes on disk are wrong, so the directory is left empty./okdownloads, verifies, and renames. The directory holds onlymodel.ggufwith 50,000 bytes.
Files.move(part, target, ATOMIC_MOVE) asks the file system to rename the file in one step. Atomic means that no other program can see the operation half done. Other programs see either no model.gguf or the complete file, never a half-written one, even if the process is killed during the move. Atomic renames work only within one file system, which is why the .part file is created in the same directory as the target. After a crash, the leftover .part file is the only trace, and the next loop pull resumes it.
ChecksumException extends IOException, so both failures take the same catch in this program. The two exceptions are still different classes on purpose. The Downloader in Example 15 retries on network errors, and it must not retry on a checksum error.
Example 13: a disk space check with FileStore.getUsableSpace
A download that fills the disk fails near the end, after all the time was spent, and it can also break other programs that need free space. The question is how to check before starting. Files.getFileStore(dir) returns the file system that holds a directory, and getUsableSpace() reports how many bytes this program can still write there.
The program defines a check that compares the usable space with the needed size plus a safety margin. Free space differs on every machine, so the program does not print it. It prints facts that are the same everywhere.
import java.nio.file.FileStore;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Locale;
public class DiskSpace {
/** Throws when the file system holding dir has less than needed bytes plus a margin. */
static void requireSpace(Path dir, long needed, long margin) throws Exception {
FileStore store = Files.getFileStore(dir);
long usable = store.getUsableSpace();
if (usable < needed + margin)
throw new IllegalStateException(String.format(Locale.ROOT, "not enough space: need %.2f GiB plus %.2f GiB margin",
needed / 1073741824.0, margin / 1073741824.0));
}
public static void main(String[] args) throws Exception {
Path models = Files.createTempDirectory("models");
long gib = 1L << 30;
FileStore store = Files.getFileStore(models);
System.out.println("usable <= unallocated <= total: "
+ (store.getUsableSpace() <= store.getUnallocatedSpace() && store.getUnallocatedSpace() <= store.getTotalSpace()));
long[] sizes = {4_096, 1L << 60}; // 4 KiB and 1 EiB: the result is the same on any disk
for (long size : sizes) {
try {
requireSpace(models, size, gib / 2);
System.out.println(size + " bytes: enough space");
} catch (IllegalStateException e) {
System.out.println(size + " bytes: " + e.getMessage());
}
}
}
}usable <= unallocated <= total: true
4096 bytes: enough space
1152921504606846976 bytes: not enough space: need 1073741824.00 GiB plus 0.50 GiB marginThe program prints checks rather than the free space itself, because the numbers differ on every machine.
usable <= unallocated <= total: trueshows how the three values thatFileStorereports relate to each other. Total space is the size of the file system. Unallocated space is what no file uses. Usable space is what this program may use, which can be lower: on Linux ext4 file systems, a share of blocks is reserved for the administrator, and quotas can limit a user further.- 4 KiB fits on any disk, and 1 EiB (2^60 bytes) fits on none, so these two lines are the same everywhere. The message gives the required size in GiB, 1,073,741,824 GiB, and the margin.
The margin, half a GiB here, leaves room for small files that the download writes next to the model, such as the saved ETag of Example 15 and the registry files of the workshop lesson. It also leaves room for other programs. A 7B Q4_K_M file of 4.36 GiB (a 7-billion-number model in a 4-bit-style compressed format) needs 4.86 GiB free with this rule.
Free space can change between the check and the end of the download. The check therefore reduces the chance of a full disk without removing it, and the write still fails with an IOException if the disk fills.
Example 14: a terminal progress bar with \r, percentage, and rate
A user who waits for a 4 GB download wants to see how far it is, how fast it goes, and how long it will take. The question is how to draw one line that updates in place, and how to test it without waiting for a real download.
The carriage return character, written \r, moves the cursor back to the start of the current line. The next text then overwrites the old text, so one line can show changing numbers. The speed and the time estimate need a clock, so the program receives the clock as a parameter. In the test, the clock is a fake that the program moves forward by hand.
The program simulates a 4.36 GiB download in twelve steps with chosen time gaps and byte counts, including two slow seconds. It prints every frame on its own line so that each one can be read.
import java.util.Locale;
import java.util.function.LongSupplier;
public class ProgressBar {
private final long total; // -1 when the length is unknown
private final LongSupplier clockNanos; // System::nanoTime in a real program
private final long start;
private long lastDraw;
private boolean drawn;
ProgressBar(long total, LongSupplier clockNanos) {
this.total = total;
this.clockNanos = clockNanos;
this.start = clockNanos.getAsLong();
}
static String size(double bytes) {
return bytes >= 1 << 30 ? String.format(Locale.ROOT, "%.2f GiB", bytes / (1 << 30))
: String.format(Locale.ROOT, "%.1f MiB", bytes / (1 << 20));
}
/** Returns the text to print, or null when less than 100 ms passed since the last frame. */
String update(long done) {
long now = clockNanos.getAsLong();
boolean finished = total >= 0 && done >= total;
if (!finished && drawn && now - lastDraw < 100_000_000L) return null;
lastDraw = now;
drawn = true;
double seconds = Math.max(1e-9, (now - start) / 1e9);
double rate = done / seconds;
if (total < 0) return String.format(Locale.ROOT, "\r%s %s/s", size(done), size(rate));
int width = 20, filled = (int) (width * done / total);
long eta = rate > 0 ? Math.round((total - done) / rate) : 0;
return String.format(Locale.ROOT, "\r[%s%s] %3d%% %s / %s %s/s ETA %d:%02d%s",
"#".repeat(filled), ".".repeat(width - filled), done * 100 / total, size(done), size(total),
size(rate), eta / 60, eta % 60, finished ? "\n" : "");
}
public static void main(String[] args) {
long[] fakeNow = {0};
ProgressBar bar = new ProgressBar(4_683_073_984L, () -> fakeNow[0]);
long done = 0;
long[] stepMs = {50, 60, 400, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000}; // time between updates
long[] bytesPerMs = {600_000, 600_000, 600_000, 600_000, 150_000, 150_000, 600_000, 600_000, 600_000, 600_000, 600_000, 600_000};
for (int i = 0; i < stepMs.length && done < 4_683_073_984L; i++) {
fakeNow[0] += stepMs[i] * 1_000_000;
done = Math.min(4_683_073_984L, done + stepMs[i] * bytesPerMs[i]); // 600,000 bytes/ms is about 572 MiB/s
String frame = bar.update(done);
System.out.println(frame == null ? "(skipped: less than 100 ms since the last frame)"
: frame.replace("\r", "\\r").replace("\n", "\\n"));
}
ProgressBar unknown = new ProgressBar(-1, () -> fakeNow[0]);
fakeNow[0] += 2_000_000_000L;
System.out.println(unknown.update(300L << 20).replace("\r", "\\r"));
}
}\r[....................] 0% 28.6 MiB / 4.36 GiB 572.2 MiB/s ETA 0:08
(skipped: less than 100 ms since the last frame)
\r[#...................] 6% 291.8 MiB / 4.36 GiB 572.2 MiB/s ETA 0:07
\r[###.................] 19% 864.0 MiB / 4.36 GiB 572.2 MiB/s ETA 0:06
\r[####................] 22% 1007.1 MiB / 4.36 GiB 401.2 MiB/s ETA 0:09
\r[#####...............] 25% 1.12 GiB / 4.36 GiB 327.7 MiB/s ETA 0:10
\r[#######.............] 38% 1.68 GiB / 4.36 GiB 381.9 MiB/s ETA 0:07
\r[##########..........] 51% 2.24 GiB / 4.36 GiB 416.4 MiB/s ETA 0:05
\r[############........] 64% 2.80 GiB / 4.36 GiB 440.4 MiB/s ETA 0:04
\r[###############.....] 77% 3.36 GiB / 4.36 GiB 457.9 MiB/s ETA 0:02
\r[#################...] 89% 3.92 GiB / 4.36 GiB 471.3 MiB/s ETA 0:01
\r[####################] 100% 4.36 GiB / 4.36 GiB 469.6 MiB/s ETA 0:00\n
\r300.0 MiB 150.0 MiB/sThe program prints each frame on its own line with \r shown as the two characters \r, so every frame is visible. A terminal shows only the latest frame on one line.
The rate is all bytes so far divided by all seconds so far, and the estimate (ETA, estimated time of arrival) is the remaining bytes divided by that rate. The frames, with the fake clock and byte counts from the two arrays, show how the numbers behave:
- The first update comes at 50 ms with 28.6 MiB: 0%, 572.2 MiB/s. The first frame is always drawn.
- The second update, 60 ms later, is skipped. Frames are drawn at most every 100 ms, because redrawing after every 64 KiB buffer would write thousands of lines per second to the terminal.
- At 19%, the rate is still 572.2 MiB/s, and the estimate is 6 seconds: (total minus done) divided by the rate.
- Two slow seconds follow (150,000 bytes per ms instead of 600,000). The rate, computed as all bytes so far divided by all time so far, drops to 401.2 and then 327.7 MiB/s, and the estimate grows to 10 seconds. The size switches from GiB to MiB below 1 GiB, which is why one frame says
1007.1 MiB. - When the speed recovers, the average rate climbs again, and the estimate falls.
- The last frame is 100% and ends with
\n, so the next output starts on a new line. A finished download is always drawn, even within 100 ms of the previous frame.
The last line is the form for an unknown length (Example 5): bytes and rate only, 300 MiB in 2 seconds at 150 MiB/s.
The average since the start reacts slowly to changes, which keeps the estimate from jumping around. A tool that wants a faster reaction uses the rate over the last few seconds instead.
Progress lines also belong in the right place. Writing them to standard error, or only when standard error is a terminal (System.console() != null), keeps \r characters out of log files. loop applies the same detection as for colors (Subcommands, Configuration, and Packaging, Example 7).
Example 15: a Downloader that combines resume, retries, verification, and atomic placement
The last question is how the techniques of this lesson fit together in one class that loop pull can call. The method Downloader.download(uri, target, size, sha256, progress) returns only when target holds the verified file. On the way it checks disk space, resumes, retries, and verifies. It needs the size and the SHA-256 value of the file, which loop pull reads from the Hub's tree listing.
The test server follows a script. It drops the first connection at 40%, answers the second request with 503, and serves the rest of the file on the third request. The program prints what the server sees, the download's progress, and each retry, so the whole story can be followed in the output.
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.time.Duration;
import java.util.HexFormat;
import java.util.Optional;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
import static java.nio.file.StandardOpenOption.*;
public class DownloaderDemo {
interface Progress { void update(long done, long total); }
interface Sleeper { void sleep(long millis) throws InterruptedException; }
/** Resumable, verified download into target: .part file, Range + If-Range, retries, SHA-256, atomic rename. */
static final class Downloader {
static final Set<Integer> RETRYABLE = Set.of(408, 429, 500, 502, 503, 504);
private final HttpClient http = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(10)).build();
private final Sleeper sleeper;
private final Random random;
private final int maxAttempts;
Downloader(Sleeper sleeper, Random random, int maxAttempts) { this.sleeper = sleeper; this.random = random; this.maxAttempts = maxAttempts; }
void download(URI uri, Path target, long size, String sha256, Progress progress) throws Exception {
if (Files.exists(target)) return; // already in place
Path part = target.resolveSibling(target.getFileName() + ".part");
Path etag = target.resolveSibling(target.getFileName() + ".part.etag");
long free = Files.getFileStore(target.getParent()).getUsableSpace();
if (free < size) throw new IOException("not enough disk space for " + target.getFileName());
for (int attempt = 1; ; attempt++) {
try {
fetch(uri, part, etag, size, progress);
break;
} catch (IOException e) {
if (attempt == maxAttempts) throw e;
long exp = Math.min(8_000, 500L << (attempt - 1));
long wait = exp / 2 + random.nextLong(exp / 2 + 1);
System.out.println(" attempt " + attempt + " failed (" + e.getMessage() + "), retrying in " + wait + " ms");
sleeper.sleep(wait);
}
}
MessageDigest sha = MessageDigest.getInstance("SHA-256");
try (InputStream in = new DigestInputStream(Files.newInputStream(part), sha)) { in.transferTo(OutputStream.nullOutputStream()); }
if (!HexFormat.of().formatHex(sha.digest()).equals(sha256)) {
Files.delete(part);
Files.deleteIfExists(etag);
throw new IOException("SHA-256 mismatch, partial file removed");
}
Files.move(part, target, StandardCopyOption.ATOMIC_MOVE);
Files.deleteIfExists(etag);
}
private void fetch(URI uri, Path part, Path etag, long size, Progress progress) throws Exception {
long have = Files.exists(part) ? Files.size(part) : 0;
if (have == size) return;
HttpRequest.Builder b = HttpRequest.newBuilder(uri);
if (have > 0) b.header("Range", "bytes=" + have + "-");
if (have > 0 && Files.exists(etag)) b.header("If-Range", Files.readString(etag));
HttpResponse<InputStream> r = http.send(b.build(), BodyHandlers.ofInputStream());
try (InputStream body = r.body()) {
int s = r.statusCode();
if (RETRYABLE.contains(s)) throw new IOException("status " + s);
if (s != 200 && s != 206) throw new IllegalStateException("status " + s + " is not retryable");
if (s == 206 && !r.headers().firstValue("content-range").orElse("").startsWith("bytes " + have + "-"))
throw new IllegalStateException("unexpected Content-Range");
if (s == 200) have = 0; // full body: start over
Optional<String> tag = r.headers().firstValue("etag");
if (tag.isPresent()) Files.writeString(etag, tag.get()); // remembered for If-Range
try (OutputStream out = Files.newOutputStream(part, CREATE, WRITE, s == 206 ? APPEND : TRUNCATE_EXISTING)) {
byte[] buf = new byte[64 * 1024];
for (int n; (n = body.read(buf)) >= 0; ) { out.write(buf, 0, n); have += n; progress.update(have, size); }
}
}
if (have != size) throw new IOException("incomplete: " + have + " of " + size + " bytes");
}
}
public static void main(String[] args) throws Exception {
byte[] data = new byte[1_000_000];
new Random(7).nextBytes(data);
String sha = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(data));
AtomicInteger requests = new AtomicInteger();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/model.gguf", ex -> {
int n = requests.incrementAndGet();
String range = ex.getRequestHeaders().getFirst("Range"), ifRange = ex.getRequestHeaders().getFirst("If-Range");
System.out.println(" server: request " + n + ", Range " + range + ", If-Range " + ifRange);
ex.getResponseHeaders().add("ETag", "\"v1\"");
if (n == 2) { ex.sendResponseHeaders(503, -1); ex.close(); return; }
int start = range != null && "\"v1\"".equals(ifRange) ? Integer.parseInt(range.substring(6, range.length() - 1)) : 0;
if (start > 0) ex.getResponseHeaders().add("Content-Range", "bytes " + start + "-" + (data.length - 1) + "/" + data.length);
ex.sendResponseHeaders(start > 0 ? 206 : 200, data.length - start);
OutputStream out = ex.getResponseBody();
if (n == 1) { out.write(data, 0, 400_000); out.flush(); throw new RuntimeException("simulated connection drop"); }
out.write(data, start, data.length - start);
out.close();
});
server.start();
URI uri = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/model.gguf");
Path dir = Files.createTempDirectory("models");
Path target = dir.resolve("model.gguf");
int[] lastQuarter = {0};
Progress progress = (done, total) -> {
int q = (int) (done * 4 / total);
if (q > lastQuarter[0]) { lastQuarter[0] = q; System.out.println(" progress " + q * 25 + "%"); }
};
Downloader downloader = new Downloader(ms -> { }, new Random(42), 5);
downloader.download(uri, target, data.length, sha, progress);
try (var files = Files.list(dir)) { System.out.println("done: " + files.map(p -> p.getFileName().toString()).sorted().toList()); }
System.out.println("identical: " + java.util.Arrays.equals(Files.readAllBytes(target), data) + ", requests " + requests.get());
downloader.download(uri, target, data.length, sha, progress);
System.out.println("second call: requests still " + requests.get());
server.stop(0);
}
} server: request 1, Range null, If-Range null
progress 25%
attempt 1 failed (closed), retrying in 488 ms
server: request 2, Range bytes=400000-, If-Range "v1"
attempt 2 failed (status 503), retrying in 930 ms
server: request 3, Range bytes=400000-, If-Range "v1"
progress 50%
progress 75%
progress 100%
done: [model.gguf]
identical: true, requests 3
second call: requests still 3The code has two methods. download is the outer sequence of steps, and fetch makes one attempt. These are the steps inside download:
- If the target exists, return at once. The second call at the end of the output makes no request, which is what makes
loop pullcheap to repeat. - Check disk space (Example 13).
- Call
fetchin a retry loop (Example 11).fetchthrowsIOExceptionfor retryable problems (network errors, retryable statuses, an incomplete body) andIllegalStateExceptionfor the others, which leave the loop at once. - Hash the complete
.partfile and compare (Example 9). On a mismatch, delete the partial file and its ETag file (Example 12). - Rename
.partto the target atomically and delete the ETag file.
Inside fetch, the size of the .part file decides what request to send: no Range for an empty file, Range: bytes=N- and If-Range with the saved ETag otherwise (Examples 7 and 8). A 200 answer restarts the file, a 206 answer is appended after checking Content-Range, and the server's ETag is saved for the next attempt. After the body, a size check catches a stream that ended early.
The output follows the scenario, line by line:
server: request 1, Range null, If-Range null: the first request has an empty.partfile.progress 25%: the client received the first 400,000 bytes, passing 25%.attempt 1 failed (closed), retrying in 488 ms: the connection dropped. The JDK reports a body stream that ends beforeContent-Lengthwith anIOExceptionwhose message isclosed.server: request 2, Range bytes=400000-, If-Range "v1": the retry asks for the rest, with the ETag saved from the first response.attempt 2 failed (status 503), retrying in 930 ms: a retryable status.server: request 3 ...gets 206, and progress continues from 50% to 100%. The 25% line is not repeated, because the callback tracks the highest quarter already printed.done: [model.gguf]: after verification and the rename, the directory holds only the final file; the.partand.part.etagfiles are gone.identical: true, requests 3, and a seconddownloadcall makes no request.
The Sleeper is a no-op here, so the waits are printed but not slept, as in Example 11. The workshop lesson, Workshop: loop pull, list, show, rm, and fit, uses this class to download GGUF files from a fake Hub.
Interview points
BodyHandlers.ofFile(path)writes any body to the file, including a 404 page, and it does not truncate an existing file. A downloader must check the status code and download to a temporary name.HttpClientdoes not follow redirects by default, because the default policy isRedirect.NEVER.NORMALfollows every redirect except one fromhttpstohttp.- The request timeout covers the wait for the response headers, not the time to read the body. A stall detector on the stream catches slow bodies.
Range: bytes=N-resumes a download. The client must check for status 206 and for aContent-Rangethat starts at N. Status 200 means the server ignored the range and sent the whole file.If-Rangewith the saved ETag makes the server send the whole new file when the version changed, so a resumed file never mixes two versions.- Reading through a
DigestInputStreamcomputes the SHA-256 without a second read of the file. A resumed download must include the bytes already on disk in the hash. - Parallel range downloads write with the positional method
FileChannel.write(buffer, position). They help when the server limits the speed of each connection. - A client should retry 408, 429, 5xx, and network errors with exponential backoff, jitter, and the server's
Retry-Aftervalue. It should never retry other 4xx errors such as 404. - A downloader writes to
name.partin the target directory and renames withATOMIC_MOVEafter verification. It keeps the.partfile after a network error and deletes it after a hash mismatch. - A downloader checks
FileStore.getUsableSpace()with a margin before starting. The check does not replace handling a full disk during the download. - Passing the clock and the sleeper into the code from outside lets tests of the progress and retry logic run without waiting.
Summary
- In
sendResponseHeaders, a positive length sendsContent-Length, 0 selects chunked encoding, and -1 sends no body. - A chunked response has the header
transfer-encoding: chunkedand nocontent-length. ofInputStreamreturns after the headers arrive, and the program reads the body as it comes in.Content-Range: bytes 500-999/1000means 500 bytes starting at offset 500 of a 1,000-byte file. Both ends of a range are inclusive.- Status 416 with
Content-Range: bytes */1000answers a range that starts at or past the end of the file. - A server may ignore
Rangeand answer 200 with the whole file. - Without
If-Range, resuming after the file changed produced a file of the right size that matched neither version. - Flipping one bit in 500,000 bytes changes the whole SHA-256 value.
- Splitting 1,000,003 bytes into 4 chunks gives chunks of 250,001, 250,001, 250,001, and 250,000 bytes.
- With equal jitter and a 500 ms base, the first two waits fall in the ranges 250 to 500 ms and 500 to 1,000 ms.
Retry-After: 3sets the wait to 3,000 ms and replaces the computed wait.Files.move(..., ATOMIC_MOVE)renames a file in one step within one file system.- Usable space is at most the unallocated space, which is at most the total space.
- A progress bar redraws at most every 100 ms and always draws the final 100% frame.
- The
Downloadersurvived a dropped connection and a 503 response and produced a verified file in 3 requests.