repo-sync

A universal Linux package repository synchronization tool. It mirrors remote repositories into a local directory tree, preserving the upstream layout so the result can be served directly to package managers.

Supported repository types:

rpm — yum/dnf repositories (repodata/repomd.xml), including plain-text mirrorlist and metalink URLs with failover between mirrors.
deb — apt repositories, both standard (dists/<suite> with a shared pool/) and flat layouts, including Acquire-By-Hash population.
arch — pacman repositories (<name>.db), including detached package signatures and companion metadata (.files, .db.tar.gz, .links.tar.gz).
apk — Alpine Linux repositories (APKINDEX.tar.gz with packages beside it).

Usage

repo-sync <type> [flags] <url> [<url> ...] <destination-directory>

The last argument is always the destination directory; every argument before it is a repository or mirrorlist URL, synchronized one after the other.

A configuration file is optional: without one the built-in defaults apply. When one is present its crawler section supplies the defaults for every command, so workers, prune_grace, request_timeout, and user_agent are shared by the sync commands and the server. The global flags --config-path, --log-level, --user-agent, and --request-timeout override the configuration for any command.

# Mirror one repository. The URL path is copied below the destination, so
# this produces ./mirror/repos/CentOS/7/EA4/.
repo-sync rpm https://example.com/repos/CentOS/7/EA4/ ./mirror

# Trim the first two path components: ./mirror/7/EA4/.
repo-sync rpm --trim 2 https://example.com/repos/CentOS/7/EA4/ ./mirror

# No path copying at all: the repository lands directly in ./mirror.
repo-sync rpm --flat https://example.com/repos/CentOS/7/EA4/ ./mirror

# Multiple repositories in one run.
repo-sync rpm https://example.com/repos/a/ https://example.com/repos/b/ ./mirror

# Crawl directory listings for repositories, at most 4 levels deep.
repo-sync rpm --discover --depth 4 https://example.com/repos/ ./mirror

# A pacman repository. The database name is discovered from the directory
# listing, or by probing URL path segments when listings are disabled.
repo-sync arch https://example.com/archlinux/core/os/x86_64/ ./mirror

# An Alpine repository is one arch directory; discovery syncs every arch
# of a release/repo tree in one run.
repo-sync apk https://dl-cdn.alpinelinux.org/alpine/v3.24/community/x86_64/ ./mirror
repo-sync apk --discover --depth 1 https://dl-cdn.alpinelinux.org/alpine/v3.24/community/ ./mirror

# A Fedora-style metalink URL; mirrors are used in preference order. The
# mirrors' paths differ, so --flat keeps the destination stable.
repo-sync rpm --flat 'https://mirrors.fedoraproject.org/metalink?repo=epel-9&arch=x86_64' ./mirror/epel9

# An apt suite. Pool files resolve against the archive root, producing
# ./mirror/debian/dists/bookworm/ and ./mirror/debian/pool/.
repo-sync deb https://deb.example.com/debian/dists/bookworm ./mirror

# Limit an apt mirror to specific components and architectures. Include
# "source" as an architecture to keep source indexes.
repo-sync deb --component main --arch amd64,source https://deb.example.com/debian/dists/bookworm ./mirror

Trace files

A public mirror is expected to say something about itself: who runs it, where it is, and when it last synchronized. Debian archives established a convention for this — a file at project/trace/<host> inside the archive — and downstream mirrors and mirror checkers read it. --trace publishes one into every repository synchronized, and the mirror server publishes one into every repository it crawls when the trace section is enabled.

repo-sync rpm --trace --trace-maintainer 'Jane <jane@example.com>' \
  https://example.com/repos/CentOS/7/EA4/ ./mirror

Mon Jul 27 20:31:58 UTC 2026
Date: Mon, 27 Jul 2026 20:31:58 +0000
Date-Started: Mon, 27 Jul 2026 20:29:14 +0000
Creator: repo-sync 0.1.0
Running on host: mirror.example.com
Maintainer: Jane <jane@example.com>
Repository type: rpm
Upstream-mirror: https://example.com/repos/CentOS/7/EA4
Total bytes received: 5726208
Total time spent syncing: 164
Average rate: 34915 B/s

Upstream-mirror records the mirror actually fetched from, which for a mirrorlist or metalink is the one that answered rather than the list's own URL — the reason a trace is written here rather than by whatever schedules the sync, which cannot know that.

The file is placed in the repository directory, or for apt at the archive root, where the convention puts it. It is registered as part of the repository, so --prune never removes it even when the repository and the trace share a directory. --dry-run reports what would change without writing one.

Everything but the dates and transfer figures comes from configuration, and each field has a matching flag: --trace-host (defaults to the system hostname), --trace-maintainer, --trace-sponsor, --trace-country, --trace-location, and --trace-throughput. Setting them in the trace section of a config file avoids repeating them on every run; --trace and --no-trace then switch tracing on or off per run. Fields left unset are omitted from the file rather than written empty.

Upstream's own traces are mirrored alongside, so project/trace/ names every mirror the content passed through and a reader can follow the chain back to the archive it originated from. A trace upstream publishes under this mirror's own host name is skipped: that file describes a different mirror, and the one written here has to win. Upstreams that publish no traces, or that do not serve a directory index for them, are not an error; nothing is copied and the run continues.

Traces work the same on the mirror server, whose crawls are concurrent: each crawl accounts for its own transfer rather than reading the shared counters, so the totals in each trace describe that crawl alone. Every completed crawl republishes the repository's trace, so the timestamps track how current the cached copy is.

Mirror server

repo-sync server runs a caching mirror in front of upstream repositories. Repositories are discovered from client requests and then kept synchronized in the background, so package managers can be pointed straight at it.

repo-sync server --config-path ./config.yaml

Without --config-path the config is read from ./config.yaml, ~/.config/repo-sync/config.yaml, or /etc/repo-sync/config.yaml. See config.example.yaml for a documented configuration covering the listener, domains, mounts, and crawler tuning. The server needs at least one domain and one mount; the sync commands do not.

To run it as a systemd service, install /etc/repo-sync/config.yaml and:

repo-sync service install
repo-sync service start

service also accepts stop, restart, status, and uninstall. The installed unit runs repo-sync server as a notify service, restarts on failure, and reloads its configuration on systemctl reload repo-sync.

Building

make

The build stamps the binary with the contents of the VERSION file plus the git commit and build date, shown by repo-sync --version. A plain go build also works but reports the version as dev.

Testing

make test

The suite is hermetic: fixture repositories for every supported format (rpm, deb, arch, apk) are generated in temp directories and served over local HTTP, so no network access is required.

7.4 KiB Raw Blame History