179 lines
7.4 KiB
Markdown
179 lines
7.4 KiB
Markdown
# repo-sync
|
|
|
|
A universal Linux package repository synchronization tool. It mirrors remote
|
|
repositories into a local directory tree, preserving the upstream layout so
|
|
the result can be served directly to package managers.
|
|
|
|
Supported repository types:
|
|
|
|
- **rpm** — yum/dnf repositories (`repodata/repomd.xml`), including
|
|
plain-text mirrorlist and metalink URLs with failover between mirrors.
|
|
- **deb** — apt repositories, both standard (`dists/<suite>` with a shared
|
|
`pool/`) and flat layouts, including `Acquire-By-Hash` population.
|
|
- **arch** — pacman repositories (`<name>.db`), including detached package
|
|
signatures and companion metadata (`.files`, `.db.tar.gz`, `.links.tar.gz`).
|
|
- **apk** — Alpine Linux repositories (`APKINDEX.tar.gz` with packages
|
|
beside it).
|
|
|
|
## Usage
|
|
|
|
```
|
|
repo-sync <type> [flags] <url> [<url> ...] <destination-directory>
|
|
```
|
|
|
|
The last argument is always the destination directory; every argument before
|
|
it is a repository or mirrorlist URL, synchronized one after the other.
|
|
|
|
A configuration file is optional: without one the built-in defaults apply.
|
|
When one is present its `crawler` section supplies the defaults for every
|
|
command, so `workers`, `prune_grace`, `request_timeout`, and `user_agent`
|
|
are shared by the sync commands and the server. The global flags
|
|
`--config-path`, `--log-level`, `--user-agent`, and `--request-timeout`
|
|
override the configuration for any command.
|
|
|
|
```sh
|
|
# Mirror one repository. The URL path is copied below the destination, so
|
|
# this produces ./mirror/repos/CentOS/7/EA4/.
|
|
repo-sync rpm https://example.com/repos/CentOS/7/EA4/ ./mirror
|
|
|
|
# Trim the first two path components: ./mirror/7/EA4/.
|
|
repo-sync rpm --trim 2 https://example.com/repos/CentOS/7/EA4/ ./mirror
|
|
|
|
# No path copying at all: the repository lands directly in ./mirror.
|
|
repo-sync rpm --flat https://example.com/repos/CentOS/7/EA4/ ./mirror
|
|
|
|
# Multiple repositories in one run.
|
|
repo-sync rpm https://example.com/repos/a/ https://example.com/repos/b/ ./mirror
|
|
|
|
# Crawl directory listings for repositories, at most 4 levels deep.
|
|
repo-sync rpm --discover --depth 4 https://example.com/repos/ ./mirror
|
|
|
|
# A pacman repository. The database name is discovered from the directory
|
|
# listing, or by probing URL path segments when listings are disabled.
|
|
repo-sync arch https://example.com/archlinux/core/os/x86_64/ ./mirror
|
|
|
|
# An Alpine repository is one arch directory; discovery syncs every arch
|
|
# of a release/repo tree in one run.
|
|
repo-sync apk https://dl-cdn.alpinelinux.org/alpine/v3.24/community/x86_64/ ./mirror
|
|
repo-sync apk --discover --depth 1 https://dl-cdn.alpinelinux.org/alpine/v3.24/community/ ./mirror
|
|
|
|
# A Fedora-style metalink URL; mirrors are used in preference order. The
|
|
# mirrors' paths differ, so --flat keeps the destination stable.
|
|
repo-sync rpm --flat 'https://mirrors.fedoraproject.org/metalink?repo=epel-9&arch=x86_64' ./mirror/epel9
|
|
|
|
# An apt suite. Pool files resolve against the archive root, producing
|
|
# ./mirror/debian/dists/bookworm/ and ./mirror/debian/pool/.
|
|
repo-sync deb https://deb.example.com/debian/dists/bookworm ./mirror
|
|
|
|
# Limit an apt mirror to specific components and architectures. Include
|
|
# "source" as an architecture to keep source indexes.
|
|
repo-sync deb --component main --arch amd64,source https://deb.example.com/debian/dists/bookworm ./mirror
|
|
```
|
|
|
|
## Trace files
|
|
|
|
A public mirror is expected to say something about itself: who runs it,
|
|
where it is, and when it last synchronized. Debian archives established a
|
|
convention for this — a file at `project/trace/<host>` inside the archive —
|
|
and downstream mirrors and mirror checkers read it. `--trace` publishes one
|
|
into every repository synchronized, and the mirror server publishes one
|
|
into every repository it crawls when the `trace` section is enabled.
|
|
|
|
```sh
|
|
repo-sync rpm --trace --trace-maintainer 'Jane <jane@example.com>' \
|
|
https://example.com/repos/CentOS/7/EA4/ ./mirror
|
|
```
|
|
|
|
```
|
|
Mon Jul 27 20:31:58 UTC 2026
|
|
Date: Mon, 27 Jul 2026 20:31:58 +0000
|
|
Date-Started: Mon, 27 Jul 2026 20:29:14 +0000
|
|
Creator: repo-sync 0.1.0
|
|
Running on host: mirror.example.com
|
|
Maintainer: Jane <jane@example.com>
|
|
Repository type: rpm
|
|
Upstream-mirror: https://example.com/repos/CentOS/7/EA4
|
|
Total bytes received: 5726208
|
|
Total time spent syncing: 164
|
|
Average rate: 34915 B/s
|
|
```
|
|
|
|
`Upstream-mirror` records the mirror actually fetched from, which for a
|
|
mirrorlist or metalink is the one that answered rather than the list's own
|
|
URL — the reason a trace is written here rather than by whatever schedules
|
|
the sync, which cannot know that.
|
|
|
|
The file is placed in the repository directory, or for apt at the archive
|
|
root, where the convention puts it. It is registered as part of the
|
|
repository, so `--prune` never removes it even when the repository and the
|
|
trace share a directory. `--dry-run` reports what would change without
|
|
writing one.
|
|
|
|
Everything but the dates and transfer figures comes from configuration,
|
|
and each field has a matching flag: `--trace-host` (defaults to the system
|
|
hostname), `--trace-maintainer`, `--trace-sponsor`, `--trace-country`,
|
|
`--trace-location`, and `--trace-throughput`. Setting them in the `trace`
|
|
section of a config file avoids repeating them on every run; `--trace` and
|
|
`--no-trace` then switch tracing on or off per run. Fields left unset are
|
|
omitted from the file rather than written empty.
|
|
|
|
Upstream's own traces are mirrored alongside, so `project/trace/` names
|
|
every mirror the content passed through and a reader can follow the chain
|
|
back to the archive it originated from. A trace upstream publishes under
|
|
this mirror's own host name is skipped: that file describes a different
|
|
mirror, and the one written here has to win. Upstreams that publish no
|
|
traces, or that do not serve a directory index for them, are not an error;
|
|
nothing is copied and the run continues.
|
|
|
|
Traces work the same on the mirror server, whose crawls are concurrent:
|
|
each crawl accounts for its own transfer rather than reading the shared
|
|
counters, so the totals in each trace describe that crawl alone. Every
|
|
completed crawl republishes the repository's trace, so the timestamps
|
|
track how current the cached copy is.
|
|
|
|
## Mirror server
|
|
|
|
`repo-sync server` runs a caching mirror in front of upstream repositories.
|
|
Repositories are discovered from client requests and then kept synchronized
|
|
in the background, so package managers can be pointed straight at it.
|
|
|
|
```sh
|
|
repo-sync server --config-path ./config.yaml
|
|
```
|
|
|
|
Without `--config-path` the config is read from `./config.yaml`,
|
|
`~/.config/repo-sync/config.yaml`, or `/etc/repo-sync/config.yaml`. See
|
|
[config.example.yaml](config.example.yaml) for a documented configuration
|
|
covering the listener, domains, mounts, and crawler tuning. The server
|
|
needs at least one domain and one mount; the sync commands do not.
|
|
|
|
To run it as a systemd service, install `/etc/repo-sync/config.yaml` and:
|
|
|
|
```sh
|
|
repo-sync service install
|
|
repo-sync service start
|
|
```
|
|
|
|
`service` also accepts `stop`, `restart`, `status`, and `uninstall`. The
|
|
installed unit runs `repo-sync server` as a notify service, restarts on
|
|
failure, and reloads its configuration on `systemctl reload repo-sync`.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make
|
|
```
|
|
|
|
The build stamps the binary with the contents of the `VERSION` file plus the
|
|
git commit and build date, shown by `repo-sync --version`. A plain
|
|
`go build` also works but reports the version as `dev`.
|
|
|
|
## Testing
|
|
|
|
```sh
|
|
make test
|
|
```
|
|
|
|
The suite is hermetic: fixture repositories for every supported format
|
|
(rpm, deb, arch, apk) are generated in temp directories and served over
|
|
local HTTP, so no network access is required.
|