diff --git a/LICENSE b/LICENSE index b7c498f..83d4a90 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2023 Mr. Gecko's Media (James Coleman). http://mrgeckosmedia.com/ +Copyright (c) 2026 Mr. Gecko's Media (James Coleman). http://mrgeckosmedia.com/ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -16,4 +16,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. \ No newline at end of file +SOFTWARE. diff --git a/README.md b/README.md index 3ac86db..a771337 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,78 @@ # mirror-sync + A tool to mirror repositories for Linux and other similar tools. This tool is designed to help follow upstream mirror instructions, and implement the features they expect from a downstream official mirror. It also includes features to help keep you in the loop in case of situations that need manual intervention. -## Configuration -It is suggested that you mirror using a sub user account, this tool prevents execution as root to protect you. Once you have an user account dedicated to mirror activities, you can make the log directory, configure logrotate, and add a configuration file to define configurations. +Companion tool: [mirror-file-generator](docs/mirror-file-generator.md) generates index pages and mirror info files from the same configuration file. + +## Contents + +- [Quick start](#quick-start) +- [How the configuration file works](#how-the-configuration-file-works) +- [Global settings](#global-settings) +- [Module settings](#module-settings) +- [Change counts in hooks](#change-counts-in-hooks) +- [Sync methods](#sync-methods) +- [Trace files](#trace-files) +- [CLI options](#cli-options) +- [Requirements](#requirements) + +## Quick start + +Mirror using a dedicated user account rather than root; the script refuses to run as root. + +**1. Create the log directory** -### Making log directory ```bash mkdir -p /var/log/mirror-sync/ chown mirror: /var/log/mirror-sync/ ``` -### Configuration for logrotate +**2. Write `/etc/mirror-sync.conf`** + +This is a complete, working configuration. Copy it and change the values: + +```bash +# --- Global settings, shared by every module --- +MAILTO="admin@example.com" +INFO_MAINTAINER="Your Name " +INFO_COUNTRY="US" +INFO_LOCATION="Texas" + +# --- List every module you want to sync, separated by spaces --- +MODULES="epel debian" + +# --- Settings for the "epel" module. Every line starts with "epel_" --- +epel_sync_method="rsync" +epel_repo="/home/mirror/http/epel" +epel_timestamp="/home/mirror/timestamp/epel" +epel_source="rsync://rsync.example.org/epel/" +epel_type="rpm" + +# --- Settings for the "debian" module. Same settings, "debian_" prefix --- +debian_sync_method="rsync" +debian_repo="/home/mirror/http/debian" +debian_timestamp="/home/mirror/timestamp/debian" +debian_source="rsync://rsync.example.org/debian/" +debian_type="deb" +``` + +**3. Sync a module** + +One module per invocation: + +```bash +mirror-sync epel +``` + +**4. Schedule it** + +``` +0 */6 * * * /usr/local/bin/mirror-sync epel +30 */6 * * * /usr/local/bin/mirror-sync debian +``` + +**5. Rotate the logs** + ``` /var/log/mirror-sync/*.log { rotate 7 @@ -24,311 +86,179 @@ chown mirror: /var/log/mirror-sync/ } ``` -### Configuring mirror-sync -The configuration file is in `/etc/mirror-sync.conf` and is formatted in bash. +## How the configuration file works -## Main configurations -### MODULES -The available modules separated by space. Each module is a separate repository to sync, and this list allows the script to know how to find their configs. +The configuration file is `/etc/mirror-sync.conf` and is **formatted in bash**. It is required; the script exits if it is missing. -### TRACEHOST -The hostname to show in trace project files, it defaults to the FQDN hostname of the server. +Set `MIRROR_SYNC_CONF` to read a different file, which is what testing a change or running a mirror entirely out of a home directory needs, as neither can write to `/etc`. Both `mirror-sync` and `mirror-file-generator` read the variable and share the file. -### mirror_hostname -The hostname of this mirror server, it defaults to the FQDN hostname of the server. If you have a public domain for your mirror, you may wish to adjust this configurtion to that. - -### PIDPATH -If you wish to override where pid files are stored to prevent duplicate module syncs, the default is `/tmp` and the directory must have write access for the mirror user. - -### LOGPATH -If you wish to override where logs are stored, the default is `/var/log/mirror-sync` and the directory must have write access for the mirror user. - -### sync_timeout -Timeout before a sync is cancelled, defaults to `timeout 1d` which should work for most mirrors. - -### max_errors -How many errors before an email is sent regarding the issue. This allows you to ignore anomalies. - -### upstream_max_age -If the upstream last modified date is older than the defined number of seconds, the upstream check will skip syncing. Default is 5 hours. - -### upstream_timestamp_min -If an upstream check is configured, this defines the minimum age in seconds of the last successful sync before the next sync will skip the upstream check. Default is 24 hours. - -### QFM_PATH -Path to where quick-fedora-mirror is located and configurations are saved. If you already have QFM installed, but want configurations stored separately. You can use the `QFM_BIN` configuration to set the QFM binary path. - -### QFM_BIN -The binary path for quick-fedora-mirror. If you override `QFM_PATH`, you will likely also have to override this path. Default: ```bash -QFM_BIN="$QFM_PATH/quick-fedora-mirror" +MIRROR_SYNC_CONF=~/mirror-sync.conf mirror-sync epel ``` -### JIGDO_FILE_BIN -If you installed jigdo outside of the home directory, you need to manually configure the `jigdo-file` binary path here. +Two rules cover everything: -### JIGDO_MIRROR_BIN -If you installed jigdo outside of the home directory, you need to manually configure the `jigdo-mirror` binary path here. +**Rule 1 — `MODULES` lists your modules.** Each module is one repository to sync. The script only knows about modules named here. -### jigdoConf -If you use jigdo to build ISO images, this is the base configuration file name. The jigdo hook saves configurations in `${jigdoConf:?}.${arch}.${s}` format. - -### MAILTO -The email address of which to mail errors to. - -### INFO_MAINTAINER -The maintainer of this repository, should be defined in `name ` format. - -### INFO_SPONSOR -If this repo is sponsored, you may define the sponsors here. - -### INFO_COUNTRY -The country of which this server resides. - -### INFO_LOCATION -The region of which this server resides (state/providence). - -### INFO_THROUGHPUT -How fast are the pipes to your repository. - -### INFO_TRIGGER -How did the sync occur, cron job or manually via ssh? This is auto detected and you do not need to define this configuration. - -### dusum_human_readable_total_file -Path to save a grand total of each disk usage sum in human readable form. - -### dusum_kbytes_total_file -Path to save a grand total of each disk usage sum in kilobytes. - -## Module specific configurations -Each module is configured via configurations prefixed by the module name. The one configuration used by all modules is the `_sync_method` configuration which defines what sync method to use. Each sync method has different configurations available. The default sync method is rsync. - -Each repo has at bare minimum the following configurations: - -- sync_method - rsync, git, aws, s3cmd, ftp, wget, or qfm. -- repo - The destination directory of the repository. -- timestamp - Path to a file to store the last successful sync unix time stamp. Can be used by a monitoring system to confirm each repo is syncing successfully. -- dusum - Path to a file to store disk usage summary results of the repository directory. -- post_successful_sync_hook - A hook to run at the very end of any sync method, after all cleanup has completed and the sync has been confirmed successful. Unlike the rsync/qfm `post_hook`, this runs after trace files, log cleanup, and mirror reporting are finished, and it is available for every sync method. - -### git -Synchronizes a git repository. To use this method, you need to have the git package installed. - -If the destination (`repo`) does not yet contain a git repository, it is cloned from the configured `source`. Otherwise it is updated in place. Bare repositories have no working tree, so they cannot be updated with `git pull`; instead the script clones them with `git clone --mirror` and updates them with `git remote update --prune`, which honors the repository's configured fetch refspec. - -Whether a repository is bare can be set explicitly with the `bare` configuration, otherwise existing repositories are auto-detected. - -Bare repositories are usually served to clients over "dumb" HTTP (a plain file server). After each successful sync (and after the initial clone) the script runs `git update-server-info` so the `info/refs` and `objects/info/packs` files clients need stay up to date. This is run every time rather than detected, as it is cheap; the packaged `post-update` hook cannot be relied on because it only fires on push, never on the fetch a mirror performs. - -#### source -The git URL to clone the repository from. Required when the destination does not already contain a git repository. - -#### bare -Set to `true` to treat the repository as bare. When cloning, this clones with `git clone --mirror`. When updating, this forces use of `git remote update --prune` (or a direct fetch when `hide_remote` is set). If unset, existing repositories are auto-detected. - -#### hide_remote -Set to `true` (requires `bare` and `source`) to keep the upstream URL out of the served repository so clients cannot probe the dumb-HTTP files to discover where the mirror pulls from. When enabled, the stored remote is removed from the repository's `config` and the `FETCH_HEAD` file (which records the URL on every fetch) is deleted after each sync. Because there is no stored remote to update, the sync fetches directly from `source` with a mirror refspec (`+refs/*:refs/*`) on each run. - -#### options -Extra options appended to the git command (`git clone` when cloning, `git pull` when updating a working-tree repository, or `git remote update --prune` / `git fetch` when updating a bare repository). - -#### Example ```bash -example_sync_method="git" -example_source="https://github.com/example/example.git" -example_repo="/home/mirror/http/example" -example_timestamp="/home/mirror/timestamp/example" +MODULES="epel debian alpine" ``` -#### Bare repository example +**Rule 2 — every module setting is the module name, an underscore, then the setting name.** Throughout this document settings are documented by their bare name (`repo`, `source`, `options`). What you actually write in the file is that name with the module's prefix: + +| Setting documented as | For module `epel`, you write | For module `debian`, you write | +| --- | --- | --- | +| `repo` | `epel_repo` | `debian_repo` | +| `sync_method` | `epel_sync_method` | `debian_sync_method` | +| `source` | `epel_source` | `debian_source` | +| `options` | `epel_options` | `debian_options` | + +There is no separate section or block syntax. A module is simply the set of variables sharing its prefix. + +### Things that commonly go wrong + +- **A module name must be a valid bash variable name** — letters, digits and underscores only, not starting with a digit. `my-repo` and `centos.9` cannot work, because the script builds variable names by joining the module name to the setting name. Use `my_repo` or `centos9`. +- **`MODULES` is easy to forget.** If you paste an example from this document without adding its module name to `MODULES`, the script reports `MODULES: parameter null or not set`, or `Unknown module`. +- **The prefix must match the name in `MODULES` exactly**, including case. A module listed as `epel` with settings written as `EPEL_repo` has no configuration, and the script reports `No configuration exists for epel`. +- **It is bash, so quoting matters.** Values containing spaces, globs or `#` need quotes: `epel_options="--exclude '.~tmp~'"`. +- **Booleans accept `true`, `false`, `1` or `0`** in any capitalization. Anything else is a hard error naming the setting, so `prune="yes"` stops the run rather than being quietly read as its opposite. + +## Global settings + +These are written without any prefix and apply to every module. + +### Core + +| Setting | Default | Description | +| --- | --- | --- | +| `MODULES` | *(required)* | The modules to sync, separated by spaces. | +| `MAILTO` | *(none)* | Email address errors are mailed to. | +| `PIDPATH` | `/tmp` | Where pid files are stored to prevent duplicate module syncs. Must be writable by the mirror user. | +| `LOGPATH` | `/var/log/mirror-sync` | Where logs are stored. Must be writable by the mirror user. | +| `sync_timeout` | `timeout 1d` | Timeout before a sync is cancelled. Works for most mirrors as-is. | +| `max_errors` | `3` | How many consecutive errors before an email is sent, so anomalies can be ignored. | +| `upstream_max_age` | `18000` (5 hours) | If the upstream last modified date is older than this many seconds, the upstream check skips syncing. | +| `upstream_timestamp_min` | `86400` (24 hours) | Minimum age in seconds of the last successful sync before the next sync will skip the upstream check. | +| `dusum_human_readable_total_file` | *(none)* | Path to save a grand total of each disk usage sum in human readable form. | +| `dusum_kbytes_total_file` | *(none)* | Path to save a grand total of each disk usage sum in kilobytes. | + +### Mirror identity + +These fill in the [trace files](#trace-files) and are set once for the whole server. + +| Setting | Default | Description | +| --- | --- | --- | +| `save_trace` | `true` | Whether to publish a trace file after each successful sync. Every method except git honors it. Override per module with `save_trace`. | +| `mirror_hostname` | FQDN hostname | The hostname of this mirror server. Set this to your public mirror domain if you have one. | +| `TRACEHOST` | FQDN hostname | The hostname to show in trace project files. | +| `INFO_MAINTAINER` | *(none)* | The maintainer of this repository, in `name ` format. | +| `INFO_SPONSOR` | *(none)* | The sponsors of this repo, if it is sponsored. | +| `INFO_COUNTRY` | *(none)* | The country this server resides in. | +| `INFO_LOCATION` | *(none)* | The region this server resides in (state/province). | +| `INFO_THROUGHPUT` | *(none)* | How fast the pipes to your repository are. | +| `INFO_TRIGGER` | auto detected | How the sync occurred, cron or ssh. Detected automatically; you do not need to set this. | + +### Tool paths + +Only needed when a tool is installed somewhere other than the default. + +| Setting | Default | Description | +| --- | --- | --- | +| `QFM_PATH` | `$HOME/quick-fedora-mirror` | Where quick-fedora-mirror is located and its configurations are saved. | +| `QFM_BIN` | `$QFM_PATH/quick-fedora-mirror` | The quick-fedora-mirror binary. If you override `QFM_PATH` you will likely need to override this too. | +| `S5CMD_BIN` | `$HOME/bin/s5cmd` | The s5cmd binary, which this tool auto installs. | +| `REPO_SYNC_BIN` | `$HOME/bin/repo-sync` | The repo-sync binary, which this tool auto installs. | +| `JIGDO_FILE_BIN` | `$HOME/bin/jigdo-file` | The `jigdo-file` binary, if jigdo is installed outside the home directory. | +| `JIGDO_MIRROR_BIN` | `$HOME/bin/jigdo-mirror` | The `jigdo-mirror` binary, if jigdo is installed outside the home directory. | +| `jigdoConf` | `$HOME/etc/jigdo/jigdo-mirror.conf` | Base configuration file name used when building ISO images. The jigdo hook saves configurations as `${jigdoConf}.${arch}.${s}`. | + +## Module settings + +Remember the prefix: a setting shown as `repo` is written `_repo`. + +Every module supports these, whatever its sync method: + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `repo` | **yes** | — | The destination directory of the repository. The script exits with `No configuration exists for ` if this is unset. | +| `sync_method` | no | `rsync` | One of `rsync`, `git`, `aws`, `s3cmd`, `s5cmd`, `repo-sync`, `ftp`, `wget`, `qfm`. | +| `timestamp` | no | *(none)* | Path to a file storing the last successful sync unix timestamp. Useful for monitoring that each repo is syncing. Required if you use `upstream_check`. | +| `dusum` | no | *(none)* | Path to a file storing the disk usage summary of the repository directory. | +| `save_trace` | no | global `save_trace` | Whether this module publishes a trace file. A module setting always beats the global one. Not used by git, which never writes one. | +| `post_successful_sync_hook` | no | *(none)* | A hook to run at the very end of any sync method, after all cleanup has completed and the sync has been confirmed successful. Unlike the rsync/qfm `post_hook`, this runs after trace files, log cleanup and mirror reporting are finished, and it is available for every sync method. See [change counts in hooks](#change-counts-in-hooks). | + +## Change counts in hooks + +Most syncs find nothing new. Every sync method counts what it changed and hands the result to the hooks, so a hook can stay idle on a run that brought nothing down: + ```bash -example_sync_method="git" -example_source="https://github.com/example/example.git" -example_repo="/home/mirror/git/example.git" -example_bare="true" -example_timestamp="/home/mirror/timestamp/example" -# Optional: keep the upstream URL out of the served files. -example_hide_remote="true" +epel_post_successful_sync_hook='[[ $SYNC_CHANGED ]] && systemctl reload nginx' ``` -### aws -Synchronize with an s3 bucket using aws cli. To use this, you need the aws cli package installed. +| Variable | Description | +| --- | --- | +| `SYNC_CHANGED` | `true` when the sync changed something, empty when it did not. Test it with `[[ $SYNC_CHANGED ]]`. | +| `SYNC_FILES_CHANGED` | Transferred plus deleted. | +| `SYNC_FILES_TRANSFERRED` | Files received from upstream, whether new or updated. | +| `SYNC_FILES_DELETED` | Files removed because upstream no longer publishes them. | +| `SYNC_REFS_UPDATED` | The git method only: refs that moved, appeared or were deleted. `0` everywhere else. | -#### aws_bucket -The bucket URL to sync with. +The variables are exported, so a hook that calls a script of its own gets them too. They are set before every hook that runs after a sync: `post_successful_sync_hook` for every method, and the rsync and qfm `post_hook`. -#### aws_access_key -The access key for the s3 bucket. +Each method counts what its own tool reports, so the counts are a statement of what that method moved rather than a number to compare between methods. `SYNC_CHANGED` is the dependable part. -#### aws_secret_key -The secret for the s3 bucket. +| Method | Counted from | +| --- | --- | +| `rsync`, `qfm` | The itemized change lines rsync prints, one per item it acted on. Both rsync stages count toward one total, as does each module qfm synchronized. Directories are left out, since one is restamped whenever anything inside it changes, and so are the trace files this script rewrites after every sync — without that, a mirror publishing a trace would report a change on every run. | +| `git` | The refs that differ from before the sync, and for a working tree the files that differ between the commit checked out before and the one checked out now. A bare mirror has no working tree, so its file counts stay `0` and `SYNC_REFS_UPDATED` carries the change. A first clone counts its whole tree. | +| `repo-sync` | Its run summary, which states outright whether the repository changed. | +| `aws`, `s3cmd`, `s5cmd` | The line the client prints for each object it downloads or removes. | +| `ftp` | The commands lftp logged, one per file it transferred or removed. Removed directories are not counted. | +| `wget` | The count wget reports having downloaded. wget never deletes, so `SYNC_FILES_DELETED` is always `0`, and a mirror run that re-fetches an unchanged index does not descend into it, so a change behind an unchanged index is not seen until the index changes. | -#### aws_endpoint_url -If you are using a third party S3 compatible service, you can enter their endpoint URL here. +## Sync methods -#### options -Extra options to append to `aws s3 sync`. +Pick the method that matches how upstream publishes: -#### Example -```bash -example_sync_method="aws" -example_repo="/home/mirror/http/example" -example_timestamp="/home/mirror/timestamp/example" -example_aws_bucket="s3://bucket/directory" -example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" -example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" -``` - -### s3cmd -Synchronize with an s3 bucket using s3cmd. To use this, you need the s3cmd package installed. - -#### aws_bucket -The bucket URL to sync with. - -#### aws_access_key -The access key for the s3 bucket. - -#### aws_secret_key -The secret for the s3 bucket. - -#### aws_endpoint_url -If you are using a third party S3 compatible service, you can enter their endpoint URL here in format of HOSTNAME:PORT. - -#### options -Extra options to append to `s3cmd`. - -#### Example -```bash -example_sync_method="s3cmd" -example_repo="/home/mirror/http/example" -example_timestamp="/home/mirror/timestamp/example" -example_aws_bucket="s3://bucket/directory" -example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" -example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" -``` - -Example of using third party bucket: -```bash -example_sync_method="s3cmd" -example_repo="/home/mirror/http/example" -example_timestamp="/home/mirror/timestamp/example" -example_aws_bucket="s3://bucket/directory" -example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" -example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" -example_options="--host='objects.example.com' --host-bucket='%(bucket).objects.example.com'" -``` - -### s5cmd -Synchronize with an s3 bucket using s5cmd. The s5cmd will auto install if not existing. - -#### aws_bucket -The bucket URL to sync with. You must end the bucket url with `*` for s5cmd to work. - -#### aws_access_key -The access key for the s3 bucket. - -#### aws_secret_key -The secret for the s3 bucket. - -#### aws_endpoint_url -If you are using a third party S3 compatible service, you can enter their endpoint URL here. - -#### options -Extra options to append to `s5cmd`. - -#### sync_options -Extra options to append to the `sync` command of s5cmd. - -#### Example -```bash -example_sync_method="s5cmd" -example_repo="/home/mirror/http/example" -example_timestamp="/home/mirror/timestamp/example" -example_aws_bucket="s3://bucket/directory/*" -example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" -example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" -``` - -### ftp -Synchronize both http and ftp sources to a repo. This sync method requires the lftp package to be installed. - -#### source -The source url to mirror from. - -#### options -Extra options to append to the mirror command of lftp. - -#### Example -```bash -example_sync_method="ftp" -example_repo="/home/mirror/http/example/" -example_timestamp="/home/mirror/timestamp/example" -example_source="https://repos.example.com/rhel/7/x86_64/stable" -``` - -### wget -Synchronizes using wget to a repository. To use this, you need the wget package installed. - -#### source -The source url to mirror from. - -#### options -The options passed to wget. Defaults to `--mirror --no-host-directories --no-parent`. - -#### Example -```bash -example_sync_method="wget" -example_repo="/home/mirror/http/example/" -example_timestamp="/home/mirror/timestamp/example" -example_source="https://repos.example.com/rhel/7/x86_64/stable" -example_options="--mirror --no-host-directories --no-parent --cut-dirs=4" -``` +| Method | Use when | +| --- | --- | +| [`rsync`](#rsync) | Upstream runs an rsync daemon. **The most common choice**, and the default. | +| [`qfm`](#qfm) | Upstream is a Fedora tier 1 mirror, or another rsync server following the Fedora module layout. | +| [`repo-sync`](#repo-sync) | Upstream publishes over http(s) only, and it is an rpm, deb, arch or apk repository. | +| [`git`](#git) | Mirroring a git repository. | +| [`aws`](#aws) / [`s3cmd`](#s3cmd) / [`s5cmd`](#s5cmd) | Upstream is an S3 bucket. | +| [`ftp`](#ftp) | Upstream is FTP, or http with no repository metadata. | +| [`wget`](#wget) | Simple recursive http(s) copy. | ### rsync -By far, the most common mirror method is to use rsync. It, while not perfect, is more efficent than using wget or ftp mirroring. You will need the rsync package installed for this to function. There is an extra CLI argument available for this sync method, `--force` which allows you to by-pass upstream checks and synchronize immediately. -#### pre_hook -A hook to run prior to the first stage sync. +By far the most common mirror method is rsync. It, while not perfect, is more efficient than using wget or ftp mirroring. You will need the rsync package installed. -#### source -The rsync server or ssh server URL. +This method supports an extra CLI argument, `--force`, which bypasses upstream checks and synchronizes immediately. -#### options -Synchronization options for the first rsync stage. +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `source` | **yes** | — | The rsync server or ssh server URL. | +| `options` | no | *(none)* | Synchronization options for the first rsync stage. | +| `options_stage2` | no | *(none)* | Options for a second rsync stage, if your repo needs one. The most basic value, if you just want to force stage 2 to occur, is `--exclude '.~tmp~'`. | +| `type` | no | *(none)* | What type of repo is being synced, for the trace file. One of `deb`, `rpm`, `iso`, `source`. | +| `arch_configurations` | no | *(none)* | Which architectures are synchronized to this mirror, for the trace file. | +| `upstream_check` | no | *(none)* | An http URL whose last modified date indicates whether upstream changed recently, to lower the impact on upstream mirrors. Needs `timestamp` set. See the global `upstream_timestamp_min` and `upstream_max_age`. | +| `time_file_check` | no | *(none)* | Name of a time file to check before syncing all files, to reduce load on upstream. | +| `rsync_password` | no | *(none)* | Password for authenticating with an rsync server. | +| `report_mirror` | no | *(none)* | Path to a `report_mirror` configuration, run after a successful sync if you need to report status back to Fedora. | +| `pre_hook` | no | *(none)* | A hook to run prior to the first stage sync. | +| `pre_stage2_hook` | no | *(none)* | A hook to run prior to the second stage sync. | +| `post_hook` | no | *(none)* | A hook to call after a successful sync. If you are using jigdo, the hook is `jigdo_hook`. See [change counts in hooks](#change-counts-in-hooks). | +| `jigdo_pkg_repo` | no | *(none)* | Path to the repo of packages, if you are using jigdo to build ISO images. | -#### options_stage2 -If your repo needs a 2 stage rsync, define some options here. The most basic option you can use, if you want to force stage 2 to occur, would be `--exclude '.~tmp~'`. +#### Examples -#### pre_stage2_hook -A hook to run prior to the second stage sync. +RPM based mirror: -#### upstream_check -An http URL to check the last modified date as a reference for if the upstream mirror was possibly modified recently. This option is mainly here to lower the impact on upstream mirrors so that mirroring happens less often. See `upstream_timestamp_min` and `upstream_max_age` for global configuration options of this check. - -#### time_file_check -Name of a time file to check if the upstream has updated before syncing all files to reduce load on upstream mirrors. - -#### report_mirror -If you have Fedora report mirror installed, and need to report back to Fedora about the status of your repository, you can provide this option a configuration path for the `report_mirror` utility to run the report after a successful sync. - -#### rsync_password -If you have an rsync password and need to authenticate with an rsync server, this is where you define the password. - -#### post_hook -Any hooks to call after a successful sync, define here. If you are using jigdo, the hook is `jigdo_hook`. - -#### jigdo_pkg_repo -If you are using jigdo to build ISO images, you need to define the path to the repo of packages. - -#### arch_configurations -Information for trace files on what architectures are synchronized to this mirror. - -#### type -For the trace file saving, this defines what type of repo is being synced. Options are deb, rpm, iso, or source. - -#### Example -Example for RPM based mirror: ```bash +MODULES="example" + example_repo="/home/mirror/http/example/" example_timestamp="/home/mirror/timestamp/example" example_source="rsync://rsync.example.org/module/" @@ -337,7 +267,9 @@ example_options_stage2="--exclude '.~tmp~'" example_type="rpm" ``` -Example for DEB based mirror: +
+DEB based mirror + ```bash example_repo="/home/mirror/http/example/" example_timestamp="/home/mirror/timestamp/example" @@ -347,7 +279,11 @@ example_options_stage2="--exclude '.~tmp~'" example_type="deb" ``` -Example with jigdo: +
+ +
+Mirror with jigdo ISO building + ```bash example_repo="/home/mirror/http/example/" example_timestamp="/home/mirror/timestamp/example" @@ -359,84 +295,380 @@ example_options_stage2="--exclude '.~tmp~'" example_type="iso" ``` +
+ ### qfm -Quick Fedora Mirror is a tool to help Fedora mirrors distribute changes faster and save on resources when trying to discover what needs to be synced. To use this method, you must have both the rsync and zsh package installed. This tool automatically downloads QFM if you do not already have it installed. -This tool requires that the upstream mirror has a module with sub modules designed for use with quick-fedora-mirror. You can use this tool with non-fedora mirrors, however they must follow the fedora module configurations. For fedora mirrors, you can utilize [tier 1 mirrors](https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering#Tier_1_mirrors). +Quick Fedora Mirror helps Fedora mirrors distribute changes faster and save resources discovering what needs to be synced. You must have both the rsync and zsh packages installed. This tool automatically downloads QFM if you do not already have it. + +The upstream mirror must offer a module with sub modules designed for quick-fedora-mirror. You can use this with non-Fedora mirrors, but they must follow the Fedora module layout. For Fedora, use a [tier 1 mirror](https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering#Tier_1_mirrors). + +List the modules available on an rsync server with: -You can list modules available on an rsync server with: ```bash rsync --list-only rsync://SERVER ``` -And to check a module out, you can list the files with: +And list the files in a module with: + ```bash rsync --list-only rsync://SERVER/MODULE ``` -#### repo -For the repo config, QFM requires the directory to be `$DOCROOT` which it'll then copy modules into. This is different from all other sync methods. - -#### pre_hook -A hook to run prior to running QFM. - -#### source -The source rsync server, without any modules appended. - -#### master_module -The main rsync module under which the fedora sub module directories exist. Defaults to `fedora-buffet`. - -#### module_mapping -If you are using this with a non-fedora mirror, you can define your own custom sub module mapping. - -#### mirror_manager_mapping -The names for custom module mapping. - -#### modules -The sub modules to sync. It is recommended that you only do one sub module, the modules available by default are fedora-alt, fedora-archive, fedora-enchilada, fedora-epel, and fedora-secondary. - -#### options -Extra options to pass to quick-fedora-mirror. - -#### filterexp -If you wish to filter out particular directories/files, define regular expression here. - -#### rsync_options -Extra options to pass to rsync during sync. - -#### report_mirror -If you have Fedora report mirror installed, and need to report back to Fedora about the status of your repository, you can provide this option a configuration path for the `report_mirror` utility to run the report after a successful sync. - -#### rsync_password -If you have an rsync password and need to authenticate with an rsync server, this is where you define the password. - -#### post_hook -Any hooks to call after a successful sync, define here. - -#### arch_configurations -Information for trace files on what architectures are synchronized to this mirror. - -#### type -For the trace file saving, this defines what type of repo is being synced. Options are deb, rpm, iso, or source. +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `repo` | **yes** | — | **Differs from other methods**: QFM requires this to be the `$DOCROOT`, into which it copies modules. | +| `source` | **yes** | — | The source rsync server, without any modules appended. | +| `modules` | **yes** | — | The sub modules to sync. Only one sub module is recommended. Defaults available upstream are `fedora-alt`, `fedora-archive`, `fedora-enchilada`, `fedora-epel`, `fedora-secondary`. | +| `master_module` | no | `fedora-buffet` | The main rsync module under which the Fedora sub module directories exist. | +| `module_mapping` | no | *(none)* | A custom sub module mapping, for use with non-Fedora mirrors. | +| `mirror_manager_mapping` | no | *(none)* | The names for a custom module mapping. | +| `filterexp` | no | *(none)* | Regular expression filtering out particular directories/files. | +| `options` | no | *(none)* | Extra options to pass to quick-fedora-mirror. | +| `rsync_options` | no | *(none)* | Extra options to pass to rsync during sync. | +| `type` | no | *(none)* | What type of repo is being synced, for the trace file. One of `deb`, `rpm`, `iso`, `source`. | +| `arch_configurations` | no | *(none)* | Which architectures are synchronized to this mirror, for the trace file. | +| `rsync_password` | no | *(none)* | Password for authenticating with an rsync server. | +| `report_mirror` | no | *(none)* | Path to a `report_mirror` configuration, run after a successful sync. | +| `pre_hook` | no | *(none)* | A hook to run prior to running QFM. | +| `post_hook` | no | *(none)* | A hook to call after a successful sync. See [change counts in hooks](#change-counts-in-hooks). | #### Example + ```bash -example_sync_method=qfm -example_repo='/home/mirror/http/' -example_timestamp='/home/mirror/timestamp/example' -example_source='rsync://mirrors.example.com' -example_modules=fedora-enchilada -example_report_mirror='/home/mirror/report_mirror.conf' -example_type=rpm +MODULES="example" + +example_sync_method="qfm" +example_repo="/home/mirror/http/" +example_timestamp="/home/mirror/timestamp/example" +example_source="rsync://mirrors.example.com" +example_modules="fedora-enchilada" +example_report_mirror="/home/mirror/report_mirror.conf" +example_type="rpm" ``` -## CLI Options -There are not that many cli options available, usage is as follows: -``` -[--help|--update-support-utilities|--version] {module} [--force] +### repo-sync + +Synchronize a Linux package repository over http(s) using [repo-sync](https://github.com/grmrgecko/repo-sync). The repo-sync binary auto installs if not present. + +Unlike `wget` or `ftp`, which copy whatever files they find, repo-sync reads the repository's own metadata (`repodata/repomd.xml`, `dists//Release`, `.db`, `APKINDEX.tar.gz`) and downloads only the files that metadata references, verifying their checksums. The upstream layout is preserved so the destination can be served directly to package managers. This makes it a good fit for upstreams that publish over http only, with no rsync daemon available. + +Each repository is identified by its own metadata, so no format has to be stated: an upstream publishing both yum and apt repositories is mirrored by one module in one run. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `source` | **yes** | — | The repository or mirrorlist URL to mirror from. Multiple URLs may be given separated by spaces, and are synchronized one after another into the same destination. | +| `type` | no | *(every format)* | Limit the run to these repository formats: `rpm`, `deb`, `arch`, `apk`. Several may be listed, separated by spaces or commas. A module limited to a single format synchronizes every URL as that format rather than identifying it, which is what a mirrorlist or metalink URL needs. | +| `prune` | no | `true` | Delete files upstream no longer publishes, matching the delete behavior of other methods. Set `false` for repositories where old packages should stay available. | +| `save_trace` | no | global `save_trace` | Traces follow the global setting as with every method, but repo-sync publishes them itself. See below. | +| `options` | no | *(none)* | Extra options passed to `repo-sync` ahead of the sub command, for options applying to every command: `--config-path`, `--log-level`, `--user-agent`, `--request-timeout`. | +| `sync_options` | no | *(none)* | Extra options passed to `repo-sync sync`. See the list below. | + +**`type` values** + +- `rpm` — yum/dnf repositories, including plain-text mirrorlist and metalink URLs with failover between mirrors. +- `deb` — apt repositories, both standard (`dists/` with a shared `pool/`) and flat layouts, including `Acquire-By-Hash` population. +- `arch` — pacman repositories, including detached package signatures and companion metadata. +- `apk` — Alpine Linux repositories. + +**Common `sync_options`** + +- `--trim N` — drop this many leading URL path components when building the destination path. +- `--flat` — place the repository directly in the destination without copying the URL path. Useful with metalink URLs, where each mirror's path differs. +- `--discover` — treat each URL as a directory index and crawl it for repositories. +- `--depth N` — maximum directory depth crawled in discovery mode. Defaults to 5. +- `--exclude GLOB` — keep part of a discovery crawl out of the mirror. Matched against the entry name wherever it appears in the tree, or against the path below the crawled URL when the pattern contains a slash. Repeatable, and an excluded directory is never listed, so nothing below it is mirrored either. +- `--include-file GLOB` — mirror loose files the crawl passes that no repository's metadata lists, such as signing keys or packages published outside a repository. Matched against the file name, repeatable, and only files in directories that are not repositories are considered. These are revalidated against their modification time rather than a checksum, and `--prune` never removes them. +- `--workers N` — number of concurrent download workers. +- `--verify` — re-verify checksums of files that already exist locally. +- `--prune-grace` — keep files that left the repository for this long before pruning them, for example `72h`. +- `--dry-run` — report what would be downloaded or pruned without changing the repository. +- `--component` and `--arch` — limit an apt mirror to specific components and architectures. Include `source` as an architecture to keep source indexes. Applies to `deb` repositories only. + +**Trace files** + +Traces land inside each repository synchronized, at `project/trace/`, or at the archive root for apt. The details recorded come from `mirror_hostname` and the `INFO_*` settings, so they are configured once globally and passed through to repo-sync rather than repeated in a repo-sync config file. Unset values are omitted from the trace, and the setting is passed either way so a repo-sync config file cannot contradict it. + +Because repo-sync writes the trace, it records the mirror it actually fetched from, which for a mirrorlist or metalink is the mirror that answered rather than the configured URL. + +#### Examples + +By default the URL path is copied below the destination, so this produces `/home/mirror/http/example/repos/CentOS/7/EA4/`. The repository identifies itself, so no `type` is needed: + +```bash +MODULES="example" + +example_sync_method="repo-sync" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +example_source="https://repos.example.com/repos/CentOS/7/EA4/" ``` -## Requirements list +
+Metalink URL, where mirrors' paths differ + +Nothing can be identified through a metalink, so the `type` is stated, and `--flat` keeps the destination stable regardless of which mirror answers: + +```bash +example_sync_method="repo-sync" +example_type="rpm" +example_repo="/home/mirror/http/epel9" +example_timestamp="/home/mirror/timestamp/epel9" +example_source="https://mirrors.fedoraproject.org/metalink?repo=epel-9&arch=x86_64" +example_sync_options="--flat" +``` + +
+ +
+apt suite limited to one component and architecture + +Mirroring two suites in one run. A trace is published at `/home/mirror/http/debian/project/trace/` without configuring anything, as tracing is on by default: + +```bash +example_sync_method="repo-sync" +example_repo="/home/mirror/http/debian" +example_timestamp="/home/mirror/timestamp/debian" +example_source="https://deb.example.com/debian/dists/bookworm https://deb.example.com/debian/dists/trixie" +example_sync_options="--component main --arch amd64,source" +``` + +
+ +
+Alpine repository, discovering every architecture + +Discovers a whole release/repo tree in one run and keeps old packages: + +```bash +example_sync_method="repo-sync" +example_repo="/home/mirror/http/alpine" +example_timestamp="/home/mirror/timestamp/alpine" +example_source="https://dl-cdn.alpinelinux.org/alpine/v3.24/community/" +example_sync_options="--discover --depth 1" +example_prune="false" +``` + +
+ +
+A vendor archive of mixed formats, crawled in one run + +Mirrors the yum and apt repositories one upstream publishes, skipping the distributions this mirror does not serve, and keeping the signing keys and loose packages published outside the repositories: + +```bash +example_sync_method="repo-sync" +example_repo="/home/mirror/http/mysql" +example_timestamp="/home/mirror/timestamp/mysql" +example_source="http://repo.mysql.com/" +example_sync_options="--discover --depth 6 --exclude sles --exclude 'fc*' --exclude docker --include-file '*.rpm' --include-file '*.deb' --include-file 'RPM-GPG-KEY-mysql*'" +``` + +
+ +### git + +Synchronizes a git repository. You need the git package installed. + +If the destination (`repo`) does not yet contain a git repository, it is cloned from the configured `source`. Otherwise it is updated in place. Bare repositories have no working tree, so they cannot be updated with `git pull`; instead the script clones them with `git clone --mirror` and updates them with `git remote update --prune`, which honors the repository's configured fetch refspec. + +Bare repositories are usually served to clients over "dumb" HTTP (a plain file server). After each successful sync (and after the initial clone) the script runs `git update-server-info` so the `info/refs` and `objects/info/packs` files clients need stay up to date. This runs every time rather than being detected, as it is cheap; the packaged `post-update` hook cannot be relied on because it only fires on push, never on the fetch a mirror performs. + +This is the one method that never publishes a trace file, so `save_trace` does not apply to it. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `source` | **yes**, when cloning | — | The git URL to clone from. Required when the destination does not already contain a git repository. | +| `bare` | no | auto-detected | Set `true` to treat the repository as bare, which clones with `git clone --mirror` and updates with `git remote update --prune` (or a direct fetch when `hide_remote` is set). If unset, existing repositories are auto-detected. `false` forces treatment as a working tree. | +| `hide_remote` | no | `false` | Set `true` (requires `bare` and `source`) to keep the upstream URL out of the served repository. See below. | +| `options` | no | *(none)* | Extra options appended to the git command: `git clone` when cloning, `git pull` when updating a working tree, or `git remote update --prune` / `git fetch` when updating a bare repository. | + +**`hide_remote`** keeps the upstream URL out of the served repository so clients cannot probe the dumb-HTTP files to discover where the mirror pulls from. When enabled, the stored remote is removed from the repository's `config` and the `FETCH_HEAD` file (which records the URL on every fetch) is deleted after each sync. Because there is no stored remote to update, the sync fetches directly from `source` with a mirror refspec (`+refs/*:refs/*`) on each run. + +#### Examples + +Working tree: + +```bash +MODULES="example" + +example_sync_method="git" +example_source="https://github.com/example/example.git" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +``` + +
+Bare repository + +```bash +example_sync_method="git" +example_source="https://github.com/example/example.git" +example_repo="/home/mirror/git/example.git" +example_bare="true" +example_timestamp="/home/mirror/timestamp/example" +# Optional: keep the upstream URL out of the served files. +example_hide_remote="true" +``` + +
+ +### aws + +Synchronize with an S3 bucket using the aws cli. You need the aws cli package installed. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `aws_bucket` | **yes** | — | The bucket URL to sync with. | +| `aws_access_key` | no | *(none)* | The access key for the S3 bucket. | +| `aws_secret_key` | no | *(none)* | The secret for the S3 bucket. | +| `aws_endpoint_url` | no | *(none)* | Endpoint URL, if you are using a third party S3 compatible service. | +| `options` | no | *(none)* | Extra options to append to `aws s3 sync`. | + +#### Example + +```bash +MODULES="example" + +example_sync_method="aws" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +example_aws_bucket="s3://bucket/directory" +example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" +example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" +``` + +### s3cmd + +Synchronize with an S3 bucket using s3cmd. You need the s3cmd package installed. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `aws_bucket` | **yes** | — | The bucket URL to sync with. | +| `aws_access_key` | no | *(none)* | The access key for the S3 bucket. | +| `aws_secret_key` | no | *(none)* | The secret for the S3 bucket. | +| `aws_endpoint_url` | no | *(none)* | Endpoint URL for a third party S3 compatible service, in `HOSTNAME:PORT` format. | +| `options` | no | *(none)* | Extra options to append to `s3cmd`. | + +#### Examples + +```bash +MODULES="example" + +example_sync_method="s3cmd" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +example_aws_bucket="s3://bucket/directory" +example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" +example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" +``` + +
+Third party bucket + +```bash +example_sync_method="s3cmd" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +example_aws_bucket="s3://bucket/directory" +example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" +example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" +example_options="--host='objects.example.com' --host-bucket='%(bucket).objects.example.com'" +``` + +
+ +### s5cmd + +Synchronize with an S3 bucket using s5cmd. The s5cmd binary auto installs if not present. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `aws_bucket` | **yes** | — | The bucket URL to sync with. **Must end with `*`** for s5cmd to work. | +| `aws_access_key` | no | *(none)* | The access key for the S3 bucket. | +| `aws_secret_key` | no | *(none)* | The secret for the S3 bucket. | +| `aws_endpoint_url` | no | *(none)* | Endpoint URL, if you are using a third party S3 compatible service. | +| `options` | no | *(none)* | Extra options to append to `s5cmd`. | +| `sync_options` | no | *(none)* | Extra options to append to the `sync` command of s5cmd. | + +#### Example + +```bash +MODULES="example" + +example_sync_method="s5cmd" +example_repo="/home/mirror/http/example" +example_timestamp="/home/mirror/timestamp/example" +example_aws_bucket="s3://bucket/directory/*" +example_aws_access_key="RANDOM_KEY_FROM_PROVIDER" +example_aws_secret_key="RANDOM_SECRET_FROM_PROVIDER" +``` + +### ftp + +Synchronize both http and ftp sources to a repo. Requires the lftp package. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `source` | **yes** | — | The source URL to mirror from. | +| `options` | no | *(none)* | Extra options to append to the mirror command of lftp. | + +#### Example + +```bash +MODULES="example" + +example_sync_method="ftp" +example_repo="/home/mirror/http/example/" +example_timestamp="/home/mirror/timestamp/example" +example_source="https://repos.example.com/rhel/7/x86_64/stable" +``` + +### wget + +Synchronizes using wget. You need the wget package installed. + +| Setting | Required | Default | Description | +| --- | --- | --- | --- | +| `source` | **yes** | — | The source URL to mirror from. | +| `options` | no | `--mirror --no-host-directories --no-parent` | The options passed to wget. Setting this replaces the default entirely. | + +#### Example + +```bash +MODULES="example" + +example_sync_method="wget" +example_repo="/home/mirror/http/example/" +example_timestamp="/home/mirror/timestamp/example" +example_source="https://repos.example.com/rhel/7/x86_64/stable" +example_options="--mirror --no-host-directories --no-parent --cut-dirs=4" +``` + +## Trace files + +Every sync method except git publishes a trace file at `/project/trace/` after a successful sync, following the convention Debian archives established. Downstream mirrors and mirror checkers read it to learn who runs a mirror, where it is, and when it last synchronized. Alongside it, `_hierarchy` lists every mirror the content passed through and `_traces` lists the trace files present. + +The contents come from the [`INFO_*` settings](#mirror-identity), `mirror_hostname` and `TRACEHOST`, so they are set once and apply to every module. The `Architectures` line is filled in for rsync and qfm modules that set `type`, and `Upstream-mirror` names the host the module syncs from. + +Traces are on by default. Turn them off globally with `save_trace="false"`, or per module with `_save_trace="false"`; a module setting always wins over the global one. + +Methods that delete files upstream no longer has are told to leave this mirror's own trace alone, since it exists only locally and would otherwise be treated as removed. Two methods differ from the rest: + +- **git** publishes no trace at all, and `save_trace` does not apply to it. A trace is an ordinary file inside the repository: in a checked-out working tree it is untracked and leaves the tree permanently dirty, and in a bare repository it means dropping a directory that is not part of the object store into the repository root. Naming the upstream would also work against `hide_remote`. +- **repo-sync** writes its own traces rather than having this script write them, so each records the mirror it actually fetched from, which for a mirrorlist or metalink is the one that answered. The `INFO_*` values are passed through to it, and `save_trace` switches it on or off the same way. + +## CLI options + +``` +mirror-sync [--help|--update-support-utilities|--version] {module} [--force] +``` + +One module per invocation. `--force` applies to the rsync method only, where it bypasses the upstream check. Running with no arguments, or with an unknown module, prints the help along with the list of modules found in your configuration. + +`MIRROR_SYNC_CONF` in the environment names the configuration file to read, in place of `/etc/mirror-sync.conf`. + +## Requirements - bash - zsh @@ -448,196 +680,27 @@ There are not that many cli options available, usage is as follows: - wget - curl - rsync -- jigdo - this tool auto installs. -- quick-fedora-mirror - this tool auto installs. +- jq +- tar +- jigdo — this tool auto installs. +- quick-fedora-mirror — this tool auto installs. +- s5cmd — this tool auto installs. +- repo-sync — this tool auto installs. ### Install on RPM based servers + ```bash -yum install bash zsh sendmail git awscli s3cmd lftp wget curl rsync +yum install bash zsh sendmail git awscli s3cmd lftp wget curl rsync jq tar ``` ### Install on DEB based servers + ```bash -apt install bash zsh sendmail git awscli s3cmd lftp wget curl rsync +apt install bash zsh sendmail git awscli s3cmd lftp wget curl rsync jq tar ``` -### Install on Arch +### Install on Arch + ```bash -yay -S bash zsh sendmail git aws-cli-git s3cmd lftp wget curl rsync +yay -S bash zsh sendmail git aws-cli-git s3cmd lftp wget curl rsync jq tar ``` - -# mirror-file-generator -A tool to generate common mirror info files at the mirror document root. - -## Configuration of modules -This tool utilizes the same config file as mirror-sync, and shares the following configurations. - -* repo - Used to verify a module is the same repo under the mirror. -* sync_method - Used to determine if qfm mirror. -* timestamp - Used for sync time. -* dusum - Used for disk usage summary. - -The tool also adds the following repo-specific configurations: - -### section -What section to associate the repo with. - -### repo_title -A title for the repo to show instead of the directory name. - -### repo_icon -The repo icon, will default to tux if not defined. The icon can be defined as an http(s) link, file path, a file stored in the template directory, or png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script will automatically make a copy or download the icon to the image folder. - -### repo_icon_dark -The dark-mode variant of the repo icon. Accepts the same sources as `repo_icon`. When defined, browsers that detect a dark OS theme will display this icon instead. If omitted, the light icon is used in all themes. - -### repo_description -A description to show at the bottom of the repo card. - -### repo_skip -This repo should not be put in generated files. - -### disable_size_calc -Should be set to a 1 if you do not want a size to be calculated. - -### timestamp_file_stat -If you do not have a timestamp file with the UNIX timestamp of the last sync, but there is a file or folder that is updated when changes are made. You can specify the path to that file or folder here and the script will stat it to determine the last sync time. - -## Configuration of custom modules -If you have a repo that is not synced via the mirror-sync, but want to customize its look on the generated index.html. You can define a list of custom modules with the `CUSTOM_MODULES` variable, then define any of the following configurations. - -* repo -* timestamp -* dusum -* section -* repo_title -* repo_icon -* repo_icon_dark -* repo_description -* repo_skip -* disable_size_calc -* timestamp_file_stat - -All of the above configurations behave the same way a regular module behaves. - -### Example -```bash -CUSTOM_MODULES="example example2" - -example_repo='/home/mirror/http/' -example_section="official" -example_repo_title="Test repo" -example_repo_icon="terminal.png" -example_repo_description="Test, this is a test." - -example2_repo='/home/mirror/windows/' -example2_repo_icon="windows.png" -``` - -## Mirrors -You can define multiple mirrors for this tool to generate files for. Each mirror can have their own templates and repos, and are configured similar to how modules are configured. As such, it is worth maybe pre-pending `mirror_` to your mirror name. - -### path -The path to the mirror under which repos are stored. - -### title -A title for the mirror, defaults to the name if unset. - -### logo -The logo, will default to tux if not defined. The logo can be defined as an http(s) link, file path, a file stored in the template directory, or png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script will automatically make a copy or download the icon to the image folder. - -### logo_dark -The dark-mode variant of the mirror logo. Accepts the same sources as `logo`. When defined, browsers that detect a dark OS theme will display this logo instead. If omitted, the light logo is used in all themes. - -### description -A description to place below the logo that can be HTML formatted. - -### provider_site -A site for the global footer generation. - -### provider_name -A name for the global footer generation. - -### Example -```bash -MIRRORS="mirror_example" - -mirror_example_path="/home/mirror/mirror_docroot" -mirror_example_title="My company" -mirror_example_logo="http://example.com/logo.png" -mirror_example_logo_dark="http://example.com/logo-dark.png" -mirror_example_description="A public mirror provided by this cool company." -mirror_example_provider_site="http://www.example.com/" -mirror_example_provider_name="Company" -``` - -## Sections -You can define multiple sections for the index.html with `SECTIONS` variable, it defaults to `official unofficial`. You can then set a default section with `section_default`, which defaults to `unofficial`. A title is auto generated as `{SECTION} Mirrors`, which you can customize with a variable named `section_{SECTION}_title`. - -## Templates -Where templates are stored is configured by `template_dir` which defaults to `/usr/local/share/mirror-file-generator/templates`. Default files should be stored under the `default` sub directory, and any customizations to individual mirrors should be saved under a sub directory with that mirror's name. You can add icons/logos into these template directories as well. - -Default templates: -* header.html - The main index header. -* section.html - Template for a section. -* repo.html - The repo card template. -* footer.html - The footer of the index. -* footer.txt - Template for the global footer file. - -The default `header.html` template includes a `@media (prefers-color-scheme: dark)` CSS block that automatically switches the page to a dark theme when the OS reports a dark preference. The logo and repo icon templates use the HTML `` element so that a dark variant image is served to dark-mode browsers when `logo_dark` or `repo_icon_dark` is configured. - -## Configurations of general defaults. - -### index_generate -Whether or not to generate the index.html file. - -* 1 Enabled -* 0 Disabled - -### index_file_name -If your index file name is different, you can adjust here. - -### footer_generate -Whether or not to generate a footer file that can be configured as the mirror's global footer. - -* 1 Enabled -* 0 Disabled - -### footer_file_name -Alternative file name for the footer file. - -### dir_sizes_generate -Whether or not to generate directory sizes file. - -* 1 Enabled -* 0 Disabled - -### dir_sizes_file_name -Alternative file name for directory sizes file. - -### dir_sizes_unknown_path -Path to store directory size summaries for unknown repos. - -### dir_sizes_human_readable -Should make human readable or output in kbytes. - -* 1 Human readable -* 0 Kbytes - -### icons_dir_name -Where to store logos and icons. - -### icons_default_source -The default URL to pull icons from, defaults to [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). - -### icons_default_img -A default file to use if icon or logo defined either isn't defined or isn't accessible. - -### icons_local_repo -Local path to a cloned copy of the dashboard-icons git repository. When this directory exists, the script serves icons from it instead of fetching them over HTTP, which avoids per-icon network requests. Defaults to `$HOME/dashboard-icons`. - -### icons_repo_url -Git URL used to clone the dashboard-icons repository into `icons_local_repo` if it does not already exist. Set to an empty string to disable automatic cloning. Defaults to `https://github.com/homarr-labs/dashboard-icons.git`. - -### icons_repo_refresh -How often (in seconds) the local dashboard-icons clone is pulled for updates. Defaults to `604800` (7 days). \ No newline at end of file diff --git a/docs/mirror-file-generator.md b/docs/mirror-file-generator.md new file mode 100644 index 0000000..8afae12 --- /dev/null +++ b/docs/mirror-file-generator.md @@ -0,0 +1,130 @@ +# mirror-file-generator + +A tool to generate common mirror info files at the mirror document root. + +It reads the same `/etc/mirror-sync.conf` file as [mirror-sync](../README.md), and follows the same two configuration rules: `MODULES` lists your modules, and every module setting is written as `_`. See [How the configuration file works](../README.md#how-the-configuration-file-works) if you have not read it yet. `MIRROR_SYNC_CONF` in the environment names a different file to read, the same as it does for mirror-sync. + +## Contents + +- [Module settings](#module-settings) +- [Custom modules](#custom-modules) +- [Mirrors](#mirrors) +- [Sections](#sections) +- [Templates](#templates) +- [General defaults](#general-defaults) + +## Module settings + +These settings are shared with mirror-sync and are read straight from its configuration: + +| Setting | Description | +| --- | --- | +| `repo` | Used to verify a module is the same repo under the mirror. | +| `sync_method` | Used to determine if the module is a qfm mirror. | +| `timestamp` | Used for sync time. | +| `dusum` | Used for the disk usage summary. | + +These settings are added by this tool: + +| Setting | Default | Description | +| --- | --- | --- | +| `section` | `section_default` | What section to associate the repo with. | +| `repo_title` | directory name | A title for the repo to show instead of the directory name. | +| `repo_icon` | tux | The repo icon. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script automatically copies or downloads the icon into the image folder. | +| `repo_icon_dark` | *(none)* | The dark-mode variant of the repo icon, accepting the same sources as `repo_icon`. When set, browsers detecting a dark OS theme display this instead. If omitted, the light icon is used in all themes. | +| `repo_description` | *(none)* | A description to show at the bottom of the repo card. | +| `repo_skip` | *(unset)* | This repo should not be put in generated files. | +| `disable_size_calc` | *(unset)* | Set to `1` if you do not want a size to be calculated. | +| `timestamp_file_stat` | *(none)* | Path to a file or folder that is updated when changes are made, stat'd to determine the last sync time. Use this when you have no timestamp file containing a UNIX timestamp. | + +## Custom modules + +If you have a repo that is not synced via mirror-sync, but you still want to customize how it looks on the generated `index.html`, list it in `CUSTOM_MODULES` and then configure it with any of the settings below. + +`CUSTOM_MODULES` is separate from `MODULES` — a repo goes in one list or the other, never both. + +Available settings, all behaving exactly as they do for a regular module: + +`repo`, `timestamp`, `dusum`, `section`, `repo_title`, `repo_icon`, `repo_icon_dark`, `repo_description`, `repo_skip`, `disable_size_calc`, `timestamp_file_stat` + +### Example + +```bash +CUSTOM_MODULES="example example2" + +example_repo="/home/mirror/http/" +example_section="official" +example_repo_title="Test repo" +example_repo_icon="terminal.png" +example_repo_description="Test, this is a test." + +example2_repo="/home/mirror/windows/" +example2_repo_icon="windows.png" +``` + +## Mirrors + +You can define multiple mirrors for this tool to generate files for. Each mirror can have its own templates and repos, and they are configured the same way modules are — listed in a variable, then configured by prefix. Because mirror names share the same flat namespace, it is worth prefixing your mirror names with `mirror_`. + +| Setting | Default | Description | +| --- | --- | --- | +| `path` | *(required)* | The path to the mirror under which repos are stored. | +| `title` | the mirror name | A title for the mirror. | +| `logo` | tux | The logo. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script automatically copies or downloads it into the image folder. | +| `logo_dark` | *(none)* | The dark-mode variant of the mirror logo, accepting the same sources as `logo`. When set, browsers detecting a dark OS theme display this instead. If omitted, the light logo is used in all themes. | +| `description` | *(none)* | A description to place below the logo. May be HTML formatted. | +| `provider_site` | *(none)* | A site for the global footer generation. | +| `provider_name` | *(none)* | A name for the global footer generation. | + +### Example + +```bash +MIRRORS="mirror_example" + +mirror_example_path="/home/mirror/mirror_docroot" +mirror_example_title="My company" +mirror_example_logo="http://example.com/logo.png" +mirror_example_logo_dark="http://example.com/logo-dark.png" +mirror_example_description="A public mirror provided by this cool company." +mirror_example_provider_site="http://www.example.com/" +mirror_example_provider_name="Company" +``` + +## Sections + +Define the sections used in `index.html` with the `SECTIONS` variable, which defaults to `official unofficial`. Set a default section with `section_default`, which defaults to `unofficial`. A title is auto generated as `{SECTION} Mirrors`, which you can customize with a variable named `section_{SECTION}_title`. + +## Templates + +Where templates are stored is configured by `template_dir`, which defaults to `/usr/local/share/mirror-file-generator/templates`. Default files are stored under the `default` sub directory, and customizations for individual mirrors go under a sub directory named after that mirror. You can add icons and logos into these template directories as well. + +Default templates: + +| File | Purpose | +| --- | --- | +| `header.html` | The main index header. | +| `section.html` | Template for a section. | +| `repo.html` | The repo card template. | +| `footer.html` | The footer of the index. | +| `footer.txt` | Template for the global footer file. | + +The default `header.html` includes a `@media (prefers-color-scheme: dark)` CSS block that automatically switches the page to a dark theme when the OS reports a dark preference. The logo and repo icon templates use the HTML `` element, so a dark variant image is served to dark-mode browsers when `logo_dark` or `repo_icon_dark` is configured. + +## General defaults + +| Setting | Default | Description | +| --- | --- | --- | +| `index_generate` | `1` | Whether to generate the `index.html` file. `1` enabled, `0` disabled. | +| `index_file_name` | `index.html` | Alternative name for the index file. | +| `footer_generate` | `1` | Whether to generate a footer file that can be configured as the mirror's global footer. `1` enabled, `0` disabled. | +| `footer_file_name` | `footer.txt` | Alternative file name for the footer file. | +| `dir_sizes_generate` | `1` | Whether to generate the directory sizes file. `1` enabled, `0` disabled. | +| `dir_sizes_file_name` | `DIRECTORY_SIZES.TXT` | Alternative file name for the directory sizes file. | +| `dir_sizes_unknown_path` | `$HOME/dusum/unknown_dirs` | Path to store directory size summaries for unknown repos. | +| `dir_sizes_human_readable` | `1` | `1` for human readable, `0` for kbytes. | +| `icons_dir_name` | `img` | Where to store logos and icons, relative to the mirror path. | +| `icons_default_source` | [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png) | The default URL to pull icons from. | +| `icons_default_img` | `tux.png` | A default file to use if the icon or logo is either undefined or inaccessible. | +| `icons_local_repo` | `$HOME/dashboard-icons` | Local path to a cloned copy of the dashboard-icons git repository. When this directory exists, icons are served from it instead of being fetched over HTTP, avoiding per-icon network requests. | +| `icons_repo_url` | `https://github.com/homarr-labs/dashboard-icons.git` | Git URL used to clone the dashboard-icons repository into `icons_local_repo` if it does not already exist. Set to an empty string to disable automatic cloning. | +| `icons_repo_refresh` | `604800` (7 days) | How often, in seconds, the local dashboard-icons clone is pulled for updates. | diff --git a/mirror-file-generator.sh b/mirror-file-generator.sh index a43f045..38ae17b 100644 --- a/mirror-file-generator.sh +++ b/mirror-file-generator.sh @@ -40,16 +40,32 @@ if (( EUID == 0 )); then exit 1 fi -# Load the required configuration file or quit. -if [[ -f /etc/mirror-sync.conf ]]; then +# Load the required configuration file or quit. The default is where a system +# installation keeps it; MIRROR_SYNC_CONF names another one, so the script can +# be pointed at a configuration of its own for testing or for a mirror run out +# of a home directory. +CONFIG_FILE="${MIRROR_SYNC_CONF:-/etc/mirror-sync.conf}" +if [[ -f $CONFIG_FILE ]]; then # shellcheck source=/dev/null - source /etc/mirror-sync.conf + source "$CONFIG_FILE" else - echo "No configuration file defined, please setup a proper configuration file." + echo "No configuration file at '${CONFIG_FILE}', please setup a proper configuration file." exit 1 fi -# Print the help for this command. +# Repos are collected into one list per section and only the sections named in +# SECTIONS are assembled into the index, so a default that is not one of them +# would drop every repo that does not name a section of its own. +# We are purposely adding quotes to match the space. +# shellcheck disable=SC2076 +if ((index_generate)) && [[ ! " $SECTIONS " =~ " $section_default " ]]; then + echo "section_default '${section_default}' is not one of SECTIONS (${SECTIONS})." + exit 1 +fi + +# Print the help for this command. An exit status may be given so that printing +# help because of a mistake (an unknown mirror) exits non-zero, letting cron or +# a calling script tell a misconfiguration apart from a run that asked for help. print_help() { echo "Mirror File Generator (${VERSION})" echo @@ -62,7 +78,7 @@ print_help() { done echo echo "Note: Defaults to generating files for all mirrors." - exit + exit "${1:-0}" } # Output message in log format and to logger. @@ -111,6 +127,7 @@ image_copy() { extension="${extension%\?*}" local save_path="$path/$icons_dir_name/$file_name.$extension" local http_code + local t_file # Determine if the saved file needs to be updated. local needs_update=0 @@ -126,6 +143,14 @@ image_copy() { if (( $(stat --format='%Y' "$file") != $(stat --format='%Y' "$save_path") )); then needs_update=1 fi + else + # A name that resolves to a template file is copied the same way a local + # file is, so it goes stale the same way and is compared by mtime too. + t_file=$(template_file "$file") + if [[ -f $t_file ]] \ + && (( $(stat --format='%Y' "$t_file") != $(stat --format='%Y' "$save_path") )); then + needs_update=1 + fi fi if (( needs_update )); then @@ -134,11 +159,15 @@ image_copy() { # If failure, and is not the default image, attempt to grab the default file. # --fail suppresses writing the response body on HTTP errors; rm -f cleans up # any partial file so the fallback recursion isn't blocked by the needs_update check. + # The fallback must be skipped when we are already fetching the default + # image, otherwise a download failure recurses into itself forever. if ! http_code=$(curl -sf --write-out "%{http_code}" -o "$save_path" "$file") \ - || ( ((http_code!=200)) && [[ "$file" != "$icons_default_img" ]] \ - && [[ "$file" != "$icons_default_source/$icons_default_img" ]] ); then + || ((http_code!=200)); then rm -f "$save_path" - image_copy "$icons_default_img" "$file_name" + if [[ "$file" != "$icons_default_img" ]] \ + && [[ "$file" != "$icons_default_source/$icons_default_img" ]]; then + image_copy "$icons_default_img" "$file_name" + fi return fi # If the file exists, copy it preserving mtime. @@ -146,7 +175,6 @@ image_copy() { cp -p "$file" "$save_path" else # Check to see if a template file exists with the file name. - local t_file t_file=$(template_file "$file") # If the file exists, copy it preserving mtime. if [[ -f $t_file ]]; then @@ -155,6 +183,23 @@ image_copy() { # If nothing else exists, try grabbing from the default source. image_copy "$icons_default_source/$file" "$file_name" return + else + # Nothing produced an image. Fall back to the default one, just + # as a failed download does, so a name that the icon source does + # not carry renders as the default rather than as a broken + # image. This matters most when the icon source is the local + # dashboard-icons clone, where a missing name is simply an + # absent file rather than a failed request. Skipped when this + # call is already placing the default, as there would then be + # nothing left to fall back to. + if [[ "$file" != "$icons_default_img" ]] \ + && [[ "$file" != "$icons_default_source/$icons_default_img" ]]; then + image_copy "$icons_default_img" "$file_name" + return + fi + # Warn on stderr, as stdout is this function's return value. + log "Warning: no image found for '$file_name', tried '$file'" >&2 + return fi fi fi @@ -162,6 +207,25 @@ image_copy() { echo "$icons_dir_name/$file_name.$extension" } +# Pull the size of a directory out of a du summary file. The path is matched +# exactly (with any trailing slashes removed) rather than with grep, as a +# substring match would also hit sibling repos sharing a prefix (for example +# "almalinux" matching "almalinux-stage") and return multiple sizes. +# +# The path is taken as everything after the size column rather than as the +# second field, since a repo path containing a space spans several fields and +# would otherwise be compared as its first word alone and never match. +dusum_size() { + awk -v dir="$1" ' + { + p = $0 + sub(/^[[:space:]]*[0-9]+[[:space:]]+/, "", p) + sub(/\/+$/, "", p) + } + p == dir { print $1; exit } + ' "$2" +} + # Read the module's configuration. read_config() { eval timestamp="\${${MODULE}_timestamp:-}" @@ -190,7 +254,7 @@ while (( $# > 0 )); do ;; # If help is requested, print it. -h|h|help|--help) - print_help "$@" + print_help ;; # Print version. -v|--version) @@ -220,7 +284,7 @@ while (( $# > 0 )); do if [[ -z $foundMirror ]]; then echo "Unknown mirror '$mirror'" echo - print_help "$@" + print_help 1 fi # Add mirror to list. @@ -234,6 +298,7 @@ while (( $# > 0 )); do done # Redirect stdout to both stdout and log file. +mkdir -p "$(dirname "$LOGFILE")" exec 1> >(tee -a "$LOGFILE") # Redirect errors to stdout so they also are logged. exec 2>&1 @@ -370,6 +435,10 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do # Keep record of total kbytes of repo sizes. totalKBytes=0 + # Count the repos written into the index, so an index that ended up with no + # repos at all is not published over a good one. + repos_added=0 + # For each directory under the mirror, generate repo data. for dir in "$path"/*; do # Some repos may be built with symbolic links, so get the real path. @@ -448,7 +517,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do # If a directory usage summary exists and we're not skipping, parse the size. if [[ -n $dusum ]] && [[ -f $dusum ]] && ((${repo_skip:-0} == 0)); then - repo_size_kb=$(grep "$real_dir" "$dusum" | awk '{print $1}') + repo_size_kb=$(dusum_size "$real_dir" "$dusum") if [[ -n $repo_size_kb ]]; then totalKBytes=$((totalKBytes+repo_size_kb)) repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec) @@ -480,7 +549,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do repo_sync_time=$(date -d "@$(cat "$timestamp")" '+%c') fi if [[ -n ${dusum:-} ]] && [[ -f $dusum ]] && ((${repo_skip:-0} == 0)); then - repo_size_kb=$(grep "$real_dir" "$dusum" | awk '{print $1}') + repo_size_kb=$(dusum_size "$real_dir" "$dusum") if [[ -n $repo_size_kb ]]; then totalKBytes=$((totalKBytes+repo_size_kb)) repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec) @@ -541,10 +610,18 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do if [[ -z ${repo_size:-} ]] && ((${disable_size_calc:-0} == 0)); then unknown_path="$dir_sizes_unknown_path/$mirror/$dir_name" - # If we should update the directory usage sizes, do so. - # shellcheck disable=SC2076 - if ((update_unknown_dir_size)) \ - && [[ ! " ${repo_sizes_updated[*]} " =~ " ${real_dir} " ]]; then + # If we should update the directory usage sizes, do so. The + # already-updated check compares exact strings rather than a + # regex, as a path containing regex metacharacters (such as dots) + # could otherwise match a different repo and skip this update. + already_updated=0 + for updated_repo in "${repo_sizes_updated[@]}"; do + if [[ $updated_repo == "$real_dir" ]]; then + already_updated=1 + break + fi + done + if ((update_unknown_dir_size)) && ((already_updated == 0)); then # Add to list of repos with updated sizes. repo_sizes_updated+=("$real_dir") @@ -565,7 +642,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do # If the unknown repo size path exists, grab it. if [[ -f $unknown_path ]]; then - repo_size_kb=$(grep "$real_dir" "$unknown_path" | awk '{print $1}') + repo_size_kb=$(dusum_size "$real_dir" "$unknown_path") if [[ -n $repo_size_kb ]]; then totalKBytes=$((totalKBytes+repo_size_kb)) repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec) @@ -577,10 +654,21 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do repo_size=$(html_encode "${repo_size:-Unknown}") export repo_size - # If we're generating the index.html, do so. + # If we're generating the index.html, do so. A section that is not one + # of SECTIONS has no list to append to, so the repo would be written to + # a file that is never assembled into the index and is left behind in + # the mirror root. Report it and fall back to the default section, so a + # typo costs the repo its placement rather than its listing. if ((index_generate)); then section=${section:-$section_default} + # We are purposely adding quotes to match the space. + # shellcheck disable=SC2076 + if [[ ! " $SECTIONS " =~ " $section " ]]; then + log "Unknown section '$section' for $dir_name, using '$section_default'" + section=$section_default + fi envsubst < "$(template_file repo.html)" >> "$index_file_temp.$section" + repos_added=$((repos_added+1)) fi # If we're generating the repo size file, add to it. @@ -610,8 +698,11 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do # Add footer subsituting environment variables. envsubst < "$(template_file footer.html)" >> "$index_file_temp" - # Verify the index temp contains a repo before moving into place. - if grep -q "Last Sync:" "$index_file_temp"; then + # Verify the index temp contains a repo before moving into place. The + # count of repos written is used rather than grepping the built file for + # text from the stock template, so that a customized repo.html does not + # stop the index from ever being published. + if ((repos_added > 0)); then [[ -f $index_file_path ]] && rm -f "$index_file_path" mv "$index_file_temp" "$index_file_path" else diff --git a/mirror-sync.sh b/mirror-sync.sh index 6685394..fe8d568 100644 --- a/mirror-sync.sh +++ b/mirror-sync.sh @@ -1,11 +1,12 @@ #!/bin/bash # This script is designed to handle mirror syncing tasks from external mirrors. -# Each mirror is handled within a module which can be configured via the configuration file /etc/mirror-sync.conf. +# Each mirror is handled within a module which can be configured via the configuration file /etc/mirror-sync.conf, +# or another file named by the MIRROR_SYNC_CONF environment variable. PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:$HOME/.local/bin:$HOME/bin" # Variables for trace generation. PROGRAM="mirror-sync" -VERSION="20260722" +VERSION="20260727" TRACEHOST=$(hostname -f) mirror_hostname=$(hostname -f) DATE_STARTED=$(LC_ALL=POSIX LANG=POSIX date -u -R) @@ -31,6 +32,10 @@ sync_timeout="timeout 1d" upstream_max_age=18000 # Update anyway if last check was more than 24 hours ago. upstream_timestamp_min=86400 +# Publish a project trace file describing this mirror after each successful +# sync. Every sync method honors this, and an individual module can override it +# with ${MODULE}_save_trace. +save_trace="true" # quick-fedora-mirror tool config. QFM_GIT="https://pagure.io/quick-fedora-mirror.git" @@ -46,18 +51,25 @@ jigdoConf="$HOME/etc/jigdo/jigdo-mirror.conf" # Path to s5cmd bin. S5CMD_BIN="$HOME/bin/s5cmd" +# Path to repo-sync bin. +REPO_SYNC_BIN="$HOME/bin/repo-sync" + # Prevent run as root. if (( EUID == 0 )); then echo "Do not mirror as root." exit 1 fi -# Load the required configuration file or quit. -if [[ -f /etc/mirror-sync.conf ]]; then +# Load the required configuration file or quit. The default is where a system +# installation keeps it; MIRROR_SYNC_CONF names another one, so the script can +# be pointed at a configuration of its own for testing or for a mirror run out +# of a home directory. +CONFIG_FILE="${MIRROR_SYNC_CONF:-/etc/mirror-sync.conf}" +if [[ -f $CONFIG_FILE ]]; then # shellcheck source=/dev/null - source /etc/mirror-sync.conf + source "$CONFIG_FILE" else - echo "No configuration file defined, please setup a proper configuration file." + echo "No configuration file at '${CONFIG_FILE}', please setup a proper configuration file." exit 1 fi @@ -93,7 +105,10 @@ handle_interrupt() { } trap handle_interrupt INT TERM -# Print the help for this command. +# Print the help for this command. An exit status may be given so the paths +# that print help because of a mistake (an unknown module or option) exit +# non-zero, letting cron or a calling script tell a misconfiguration apart from +# a run that simply asked for help. print_help() { echo "Mirror Sync (${VERSION})" echo @@ -104,7 +119,7 @@ print_help() { for MODULE in ${MODULES:?}; do echo "$MODULE" done - exit + exit "${1:-0}" } # Send email to admins about error. @@ -260,11 +275,73 @@ s5cmd_install() { fi } +# Installs repo-sync for package repository syncing. +repo_sync_install() { + # Only install if not installed or update requested. + if ! [[ -f $REPO_SYNC_BIN ]] || [[ $1 == "-u" ]]; then + # Make sure we're in the home dir and the bin dir exists. + if ! cd "$HOME"; then + echo "Unable to access home dir." + exit 1 + fi + if [[ ! -d bin ]]; then + mkdir -p bin + fi + + # Releases are published per architecture, so map this machine's + # architecture onto the name used in the release asset. + machine=$(uname -m) + case "$machine" in + x86_64|amd64) release_arch="amd64" ;; + i386|i486|i586|i686) release_arch="386" ;; + aarch64|arm64) release_arch="arm64" ;; + armv6*|armv7*|armhf|arm) release_arch="armv6" ;; + ppc64le) release_arch="ppc64le" ;; + *) + echo "Unsupported architecture '${machine}' for repo-sync." + exit 1 + ;; + esac + + # Get the latest download URL from github. + download_url=$(curl -s https://api.github.com/repos/grmrgecko/repo-sync/releases/latest | jq --arg suffix "_linux_${release_arch}.tar.gz" '.assets[] | select(.name | endswith($suffix)) | .browser_download_url' -r) + if [[ -z $download_url ]]; then + echo "Unable to get download url for repo-sync." + exit 1 + fi + + # Download repo-sync + if ! wget "$download_url" -O repo-sync.tar.gz; then + echo "Unable to download repo-sync." + exit 1 + fi + + # Extract just the binary. The archive also ships a LICENSE and + # README.md, which would clobber files in the home directory. + if ! tar -xf repo-sync.tar.gz repo-sync; then + echo "Unable to extract repo-sync." + exit 1 + fi + if ! [[ -f repo-sync ]]; then + echo "Unable to extract repo-sync." + exit 1 + fi + + # Remove the tar.gz file. + rm -f repo-sync.tar.gz + + # Move repo-sync to the bin path. + chmod +x repo-sync + mv repo-sync "$REPO_SYNC_BIN" + fi +} + # Updates the mirror support utilties on server with upstream. update_support_utilities() { quick_fedora_mirror_install -u jigdo_install -u s5cmd_install -u + repo_sync_install -u } # Builds iso images from jigdo files. @@ -320,6 +397,64 @@ EOF done } +# Sum a "Field: count" statistic across every run recorded in a log. Both +# rsync's --stats and repo-sync's run summary report in this form, and one log +# can hold several runs, as quick-fedora-mirror calls rsync once per module. A +# log that recorded no run at all counts as none. +sum_log_stat() { + [[ -f $2 ]] || { echo 0; return; } + awk -F': ' -v field="$1" ' + $1==field {split($2, count, " "); total+=count[1]} + END {print total+0} + ' "$2" 2>/dev/null +} + +# Count what an rsync run changed, from the itemized change lines in one or +# more of its logs. rsync prints one line per item it acts on: "*deleting" for +# something it removed, and otherwise an eleven character summary of the change +# such as ">f+++++++++", followed by the item's path. quick-fedora-mirror +# formats those lines with a leading marker, which is skipped over. +# +# Only content counts. Directories are left out, as an unchanged directory is +# still restamped whenever something inside it changes, and so are the trace +# files this script maintains, which it rewrites after every sync no matter +# what upstream did. The two counts are printed as "transferred deleted". +count_rsync_changes() { + local _logs=() _log + for _log in "$@"; do + [[ -f $_log ]] && _logs+=("$_log") + done + if (( ${#_logs[@]} == 0 )); then + echo "0 0" + return + fi + awk ' + { + field = ($1 == "@") ? 2 : 1 + summary = $field + name = $(field+1) + if (name ~ /(^|\/)project\/trace\//) next + if (summary == "*deleting") { + if (name !~ /\/$/) deleted++ + next + } + if (summary ~ /^[<>ch.][fLDS]/) transferred++ + } + END {print transferred+0, deleted+0} + ' "${_logs[@]}" +} + +# Count how many times a message appears in a log. Sync tools that show +# progress rewrite the current line with a carriage return rather than ending +# it, so several messages can end up sharing one line; occurrences are counted +# rather than lines, and a message is only recognized where a line or a +# carriage return puts it at the start of the output. +count_sync_events() { + local _log=$1 _message=$2 _cr=$'\r' + [[ -f $_log ]] || { echo 0; return; } + grep -oE "(^|${_cr})${_message}" "$_log" 2>/dev/null | wc -l +} + # Pull a field from a trace file or rsync stats. extract_trace_field() { value=$(awk -F': ' "\$1==\"$1\" {print \$2; exit}" "$2" 2>/dev/null) @@ -390,7 +525,7 @@ build_trace_content() { echo "Architectures-Configuration: ${arch_configurations:-ALL}" echo "Upstream-mirror: ${RSYNC_HOST:-unknown}" - + # Total bytes synced per rsync stage. total=0 if [[ -f $LOGFILE_SYNC ]]; then @@ -430,15 +565,30 @@ build_trace_content() { echo "Total time spent in stage2 rsync: ${STATS_TOTAL_RSYNC_TIME2}" fi echo "Total time spent in rsync: ${total_time}" - if (( total_time != 0 )); then + # Only the rsync based methods report a byte count, so without one there is + # no rate to state; printing 0 B/s would claim the sync moved nothing. + if (( total > 0 && total_time != 0 )); then rate=$(( total / total_time )) echo "Average rate: ${rate} B/s" fi } +# Reduce an upstream location to the host it names, for the trace's +# Upstream-mirror field. Sources come in several shapes across the sync +# methods: rsync URLs and rsync daemon syntax (host::module), http(s) and ftp +# URLs, s3 bucket URLs, and space separated lists of URLs. Only the first entry +# of a list is reported, as the trace names a single upstream. +upstream_host() { + local _host="${1%% *}" # First entry of a space separated list. + _host="${_host#*://}" # Drop any scheme. + _host="${_host#*@}" # Drop any credentials. + _host="${_host%%/*}" # Drop the path. + _host="${_host%%:*}" # Drop the port, or the rsync daemon module. + echo "$_host" +} + # For modules that are repositories (with RPM, DEB, ISOs, or source code), # build a project trace file with information about the repo. -# Mainly works with rsync based modules. save_trace_file() { # Trace file/dir paths. TRACE_DIR="${repo}/project/trace" @@ -447,22 +597,31 @@ save_trace_file() { TRACE_MASTER_FILE="${TRACE_DIR}/master" TRACE_HIERARCHY="${TRACE_DIR}/_hierarchy" - # Parse the rsync host from the source. - RSYNC_HOST=${source:-} - RSYNC_HOST=${RSYNC_HOST/rsync:\/\//} - RSYNC_HOST=${RSYNC_HOST%%:*} - RSYNC_HOST=${RSYNC_HOST%%/*} + # Parse the upstream host. Modules define their upstream differently, so a + # method may set trace_upstream to whatever names its source; the rsync + # based methods leave it unset and their source is used. + RSYNC_HOST=$(upstream_host "${trace_upstream:-${source:-}}") # Build trace and save to file. build_trace_content > "${TRACE_FILE}.new" mv "${TRACE_FILE}.new" "$TRACE_FILE" - # Build heirarchy file. + # Build the hierarchy file, which lists every mirror the content passed + # through. The upstream entries come from the copy saved on the previous + # run; this mirror's own entry is dropped from that base before a fresh one + # is appended, so repeated syncs replace the entry rather than stacking a + # new copy of it on every run. + local _self _name _rest + _self=$(basename "$TRACE_FILE") { if [[ -e "${TRACE_HIERARCHY}.mirror" ]]; then - cat "${TRACE_HIERARCHY}.mirror" + while read -r _name _rest; do + [[ $_name ]] || continue + [[ $_name == "$_self" ]] && continue + echo "$_name $_rest" + done < "${TRACE_HIERARCHY}.mirror" fi - echo "$(basename "$TRACE_FILE") $mirror_hostname $TRACEHOST ${RSYNC_HOST:-unknown}" + echo "${_self} $mirror_hostname $TRACEHOST ${RSYNC_HOST:-unknown}" } > "${TRACE_HIERARCHY}.new" mv "${TRACE_HIERARCHY}.new" "$TRACE_HIERARCHY" cp "$TRACE_HIERARCHY" "${TRACE_HIERARCHY}.mirror" @@ -570,6 +729,45 @@ rebuild_dusum_totals() { fi } +# Counts describing what a sync changed. Every sync method publishes these +# before it runs any hook, so a hook can stay idle on a run that found nothing +# new: +# +# example_post_successful_sync_hook='[[ $SYNC_CHANGED ]] && do_something' +# +# They are exported so a hook that calls out to a script sees them as well. +# What counts as a file differs between the sync methods, and a few of them can +# only report what their tool chose to print, so the counts are each method's +# best statement of what it moved rather than a number to compare across +# methods. SYNC_CHANGED is the dependable part: it is set whenever a method saw +# any change at all. +reset_sync_counts() { + SYNC_FILES_TRANSFERRED=0 + SYNC_FILES_DELETED=0 + SYNC_FILES_CHANGED=0 + SYNC_REFS_UPDATED=0 + SYNC_CHANGED="" + export SYNC_FILES_TRANSFERRED SYNC_FILES_DELETED SYNC_FILES_CHANGED + export SYNC_REFS_UPDATED SYNC_CHANGED +} + +# Publish what a sync changed, and report it to the log. A third argument +# states whether the mirror changed for a method whose changes are not measured +# in files (git counts refs); without one, any counted file is a change. +set_sync_counts() { + SYNC_FILES_TRANSFERRED=${1:-0} + SYNC_FILES_DELETED=${2:-0} + SYNC_FILES_CHANGED=$((SYNC_FILES_TRANSFERRED+SYNC_FILES_DELETED)) + if (( $# >= 3 )); then + SYNC_CHANGED=$3 + elif (( SYNC_FILES_CHANGED > 0 )); then + SYNC_CHANGED="true" + else + SYNC_CHANGED="" + fi + echo "Files changed: ${SYNC_FILES_CHANGED} (${SYNC_FILES_TRANSFERRED} transferred, ${SYNC_FILES_DELETED} deleted)" +} + # Items to do post an successful sync. post_successful_sync() { # Save timestamp of last sync. @@ -589,7 +787,7 @@ post_successful_sync() { # If modules are defined, sum each module directory. if [[ $modules ]]; then for module in $modules; do - du -s "$docroot$(module_dir "$module")/" + du -s "${docroot%/}/$(module_dir "$module")/" done else # Standard repo sum. @@ -642,7 +840,8 @@ post_failed_sync() { module_config() { MODULE=$1 acquire_lock "$MODULE" - + reset_sync_counts + # Read the configuration for this module. eval repo="\$${MODULE}_repo" eval timestamp="\$${MODULE}_timestamp" @@ -654,9 +853,99 @@ module_config() { echo "No configuration exists for ${MODULE}" exit 1 fi + resolve_save_trace log_start_header } +# Normalize a boolean configuration in place. Only "true", "false", "1" and "0" +# are accepted, so a value that was meant to enable something but does not spell +# it one of those ways is reported instead of being quietly read as its +# opposite. An unset value takes the given default. The named variable is left +# holding "true" or the empty string, so callers can test it with [[ ]]. +# +# The variable is passed by name rather than by value because a value returned +# through a command substitution would be produced in a subshell, where a +# rejected value could not stop the script. +parse_bool() { + local _var=$1 _default=$2 _config=$3 _value _resolved + eval _value="\$${_var}" + _resolved="${_value:-$_default}" + case "${_resolved,,}" in + true|1) eval "${_var}=true" ;; + false|0) eval "${_var}=" ;; + *) + echo "Invalid value '${_value}' for ${_config}, expected true, false, 1 or 0." + exit 1 + ;; + esac +} + +# Resolve whether this module publishes a trace file. The module setting wins +# when present, otherwise the global save_trace applies, which defaults to on. +# Every sync method reads the result from save_trace_enabled. +resolve_save_trace() { + eval save_trace_enabled="\$${MODULE}_save_trace" + if [[ $save_trace_enabled ]]; then + parse_bool save_trace_enabled "true" "${MODULE}_save_trace" + else + save_trace_enabled="${save_trace:-}" + parse_bool save_trace_enabled "true" "save_trace" + fi +} + +# List every ref in the repository, so what a sync changed can be measured by +# comparing the listing against one taken beforehand. Tags and remote tracking +# refs are listed alongside branches, so an update that moved any of them is +# seen. +git_ref_state() { + git -C "${repo:?}" show-ref 2>/dev/null +} + +# Count the refs that differ between two listings. Refs are compared by name, +# so one that moved, one that appeared and one that was deleted each count +# once. Blank lines are ignored, which is what an empty listing (a repository +# that did not exist yet) reads as. +git_changed_refs() { + awk ' + NF<2 {next} + NR==FNR {before[$2]=$1; next} + {after[$2]=$1} + END { + for (ref in after) if (!(ref in before) || before[ref]!=after[ref]) changed++ + for (ref in before) if (!(ref in after)) changed++ + print changed+0 + } + ' <(echo "$1") <(echo "$2") +} + +# Publish what a git sync changed, given the ref listing and the checked out +# commit from before it ran. A bare mirror has no working tree whose files +# could be counted, so refs are all it reports. A working tree also counts the +# files the update brought in, by comparing the commit checked out before +# against the one checked out now; a fresh clone has no commit from before, so +# its whole tree is new. +git_sync_counts() { + local _refs_before=$1 _head_before=$2 _head_after _transferred=0 _deleted=0 _changed="" + + SYNC_REFS_UPDATED=$(git_changed_refs "$_refs_before" "$(git_ref_state)") + + if [[ ! $bare ]]; then + _head_after=$(git -C "${repo:?}" rev-parse HEAD 2>/dev/null) + if [[ $_head_before ]] && [[ $_head_after ]]; then + _transferred=$(git -C "${repo:?}" diff --name-only --diff-filter=d "$_head_before" "$_head_after" 2>/dev/null | wc -l) + _deleted=$(git -C "${repo:?}" diff --name-only --diff-filter=D "$_head_before" "$_head_after" 2>/dev/null | wc -l) + elif [[ $_head_after ]]; then + _transferred=$(git -C "${repo:?}" ls-files 2>/dev/null | wc -l) + fi + fi + + if (( SYNC_REFS_UPDATED > 0 || _transferred > 0 || _deleted > 0 )); then + _changed="true" + fi + set_sync_counts "$_transferred" "$_deleted" "$_changed" + echo "Refs updated: ${SYNC_REFS_UPDATED}" +} + # Maintenance to run after a successful sync (or clone) of a bare repository. # Bare repositories are typically served to clients over "dumb" HTTP (a plain # file server), which requires the info files kept up to date and, optionally, @@ -681,6 +970,13 @@ git_bare_post_sync() { } # Sync git based mirrors. +# +# This is the one method that publishes no trace file, so save_trace does not +# apply to it. A trace is an ordinary file inside the repository: in a checked +# out working tree it shows up as untracked and leaves the tree permanently +# dirty, and in a bare repository it means dropping a directory that is not part +# of the object store into the repository root. Neither is worth a trace, and +# naming the upstream would work against hide_remote besides. git_sync() { # Start the module. module_config "$1" @@ -690,19 +986,21 @@ git_sync() { eval bare="\$${MODULE}_bare" eval hide_remote="\$${MODULE}_hide_remote" - # Normalize the bare flag to either "true" or empty. - case "${bare,,}" in - 1|true|yes|bare) bare="true" ;; - *) bare="" ;; - esac + # An unset bare flag stays empty so the repository can be auto-detected + # below; only an explicit value is normalized here. Normalization maps + # false to the empty string, which an unset flag also has, so whether the + # flag was explicitly configured is remembered separately to keep an + # explicit false from being auto-detected away. + bare_configured="" + if [[ $bare ]]; then + parse_bool bare "false" "${MODULE}_bare" + bare_configured=1 + fi - # Normalize the hide_remote flag to either "true" or empty. Hiding the remote - # requires a configured source, since without a stored remote the URL to - # fetch from must come from the configuration on every sync. - case "${hide_remote,,}" in - 1|true|yes) hide_remote="true" ;; - *) hide_remote="" ;; - esac + # Hiding the remote requires a configured source, since without a stored + # remote the URL to fetch from must come from the configuration on every + # sync. + parse_bool hide_remote "false" "${MODULE}_hide_remote" if [[ $hide_remote ]] && [[ ! $source ]]; then echo "hide_remote requires a source to be configured." exit 1 @@ -716,6 +1014,14 @@ git_sync() { echo "No git repository at '${repo:?}' and no source defined to clone from." exit 1 fi + # There is no repository to auto-detect yet, so the bare flag is + # whatever was configured. Refuse the same combination the update path + # refuses rather than cloning a working tree and leaving the upstream + # URL in the served config, which is what hide_remote exists to prevent. + if [[ $hide_remote ]] && [[ ! $bare ]]; then + echo "hide_remote requires a bare repository." + exit 1 + fi echo "Cloning '${source}' into '${repo:?}'." if [[ $bare ]]; then eval git clone --mirror ${options:+$options} "'${source}'" "'${repo:?}'" @@ -724,6 +1030,9 @@ git_sync() { fi RT=$? if (( RT == 0 )); then + # Nothing existed here before the clone, so it is measured against + # an empty repository. + git_sync_counts "" "" [[ $bare ]] && git_bare_post_sync post_successful_sync else @@ -744,9 +1053,20 @@ git_sync() { # if set, otherwise auto-detect. Bare repositories have no working tree, so a # `git pull` is not possible; instead update by fetching the configured # refspec. - if [[ ! $bare ]] && [[ $(git rev-parse --is-bare-repository 2>/dev/null) == "true" ]]; then + if [[ ! $bare_configured ]] && [[ $(git rev-parse --is-bare-repository 2>/dev/null) == "true" ]]; then bare="true" fi + # Hiding the remote only happens in the bare update path, so requiring it + # here keeps a working tree from silently serving the upstream URL. + if [[ $hide_remote ]] && [[ ! $bare ]]; then + echo "hide_remote requires a bare repository." + exit 1 + fi + # Record what the repository holds now, so the sync can be measured + # against it once it has run. + refs_before=$(git_ref_state) + head_before=$(git -C "${repo:?}" rev-parse HEAD 2>/dev/null) + if [[ $bare ]]; then # When hiding the remote there is no stored remote to update, so fetch # directly from the source with a mirror refspec. Otherwise honor the @@ -761,6 +1081,9 @@ git_sync() { fi RT=$? if (( RT == 0 )); then + # Count before any post-sync maintenance runs, as hiding the remote + # drops the refs it stored and that is not a change from upstream. + git_sync_counts "$refs_before" "$head_before" [[ $bare ]] && git_bare_post_sync post_successful_sync else @@ -779,6 +1102,17 @@ read_aws_config() { eval AWS_SECRET_ACCESS_KEY="\$${MODULE}_aws_secret_key" export AWS_SECRET_ACCESS_KEY eval AWS_ENDPOINT_URL="\$${MODULE}_aws_endpoint_url" + + # The trace names the service the objects came from, which is the endpoint + # when one is configured and otherwise the bucket itself. + trace_upstream="${AWS_ENDPOINT_URL:-$bucket}" +} + +# The path of this mirror's own trace file, relative to the repository root. +# The sync methods that delete removed files have to be told to leave it alone, +# as it exists only locally and would otherwise be treated as removed upstream. +trace_relative_path() { + echo "project/trace/${mirror_hostname:?}" } # Sync AWS S3 bucket based mirrors. @@ -791,19 +1125,42 @@ aws_sync() { options="$options --endpoint-url='$AWS_ENDPOINT_URL'" fi + # Keep --delete from removing the trace file this module publishes. + if [[ $save_trace_enabled ]]; then + options="$options --exclude='$(trace_relative_path)'" + fi + + # Capture the sync's output so what it changed can be counted from it. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + # Run AWS client to sync the S3 bucket. + sync_started=$(date +%s) eval "$sync_timeout" aws s3 sync \ --no-follow-symlinks \ --delete \ "$options" \ - "'${bucket:?}'" "'${repo:?}'" + "'${bucket:?}'" "'${repo:?}'" | tee -a "$LOGFILE_SYNC" RT=${PIPESTATUS[0]} + sync_ended=$(date +%s) + + # The client reports each object it downloads or removes on its own line. + set_sync_counts \ + "$(count_sync_events "$LOGFILE_SYNC" 'download: ')" \ + "$(count_sync_events "$LOGFILE_SYNC" 'delete: ')" + rm -f "$LOGFILE_SYNC" + if (( RT == 0 )); then post_successful_sync else post_failed_sync fi + if [[ $save_trace_enabled ]]; then + save_trace_file + fi + run_post_successful_sync_hook log_end_header } @@ -814,25 +1171,56 @@ s3cmd_sync() { module_config "$1" read_aws_config + # s3cmd's --host takes a bare "hostname[:port]", while the aws and s5cmd + # methods read the same aws_endpoint_url setting as a full URL. Trim the + # scheme and path so one endpoint value works for all three; a value that + # was already written as a bare host passes through unchanged. if [[ -n $AWS_ENDPOINT_URL ]]; then - options="$options --host='$AWS_ENDPOINT_URL'" + s3cmd_host="${AWS_ENDPOINT_URL#*://}" + s3cmd_host="${s3cmd_host%%/*}" + options="$options --host='$s3cmd_host'" fi + # Keep --delete-removed from removing the trace file this module publishes. + if [[ $save_trace_enabled ]]; then + options="$options --exclude='$(trace_relative_path)'" + fi + + # Capture the sync's output so what it changed can be counted from it. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + # Run AWS client to sync the S3 bucket. + sync_started=$(date +%s) eval "$sync_timeout" s3cmd sync \ -v --progress \ --skip-existing \ --delete-removed \ --delete-after \ "$options" \ - "'${bucket:?}'" "'${repo:?}'" + "'${bucket:?}'" "'${repo:?}'" | tee -a "$LOGFILE_SYNC" RT=${PIPESTATUS[0]} + sync_ended=$(date +%s) + + # s3cmd reports each object it downloads or removes with the path quoted, + # which is also what tells those messages apart from the warnings it + # writes about deletions it refused to make. + set_sync_counts \ + "$(count_sync_events "$LOGFILE_SYNC" "download: '")" \ + "$(count_sync_events "$LOGFILE_SYNC" "delete: '")" + rm -f "$LOGFILE_SYNC" + if (( RT == 0 )); then post_successful_sync else post_failed_sync fi + if [[ $save_trace_enabled ]]; then + save_trace_file + fi + run_post_successful_sync_hook log_end_header } @@ -851,13 +1239,149 @@ s5cmd_sync() { options="$options --endpoint-url='$AWS_ENDPOINT_URL'" fi + # Keep --delete from removing the trace file this module publishes. + if [[ $save_trace_enabled ]]; then + sync_options="${sync_options:+$sync_options }--exclude='$(trace_relative_path)'" + fi + + # Capture the sync's output so what it changed can be counted from it. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + # Run AWS client to sync the S3 bucket. + sync_started=$(date +%s) eval "$sync_timeout" "$S5CMD_BIN" "$options" \ sync ${sync_options:+$sync_options} \ --no-follow-symlinks \ --delete \ - "'${bucket:?}'" "'${repo:?}'" + "'${bucket:?}'" "'${repo:?}'" | tee -a "$LOGFILE_SYNC" RT=${PIPESTATUS[0]} + sync_ended=$(date +%s) + + # s5cmd echoes the operation it performed for each object, so a sync + # reports one "cp" per object copied and one "rm" per object removed. + set_sync_counts \ + "$(count_sync_events "$LOGFILE_SYNC" 'cp ')" \ + "$(count_sync_events "$LOGFILE_SYNC" 'rm ')" + rm -f "$LOGFILE_SYNC" + + if (( RT == 0 )); then + post_successful_sync + else + post_failed_sync + fi + + if [[ $save_trace_enabled ]]; then + save_trace_file + fi + + run_post_successful_sync_hook + log_end_header +} + +# Sync Linux package repositories (rpm/deb/arch/apk) using repo-sync. +repo_sync_sync() { + # Install repo-sync if not already installed. + repo_sync_install + + # Start the module. + module_config "$1" + + # Read repo-sync specific configuration. + eval source="\$${MODULE}_source" + eval repo_type="\$${MODULE}_type" + eval prune="\$${MODULE}_prune" + eval sync_options="\$${MODULE}_sync_options" + + # repo-sync identifies each repository from its own metadata, so a type is + # only needed to limit which formats a run accepts, or to state the format + # outright for a mirrorlist or metalink URL, which cannot be identified. + # Several may be listed, separated by spaces or commas. + type_options="" + if [[ $repo_type ]]; then + read -r -a repo_sync_types <<< "${repo_type//,/ }" + for repo_sync_type in "${repo_sync_types[@]}"; do + case "${repo_sync_type,,}" in + rpm|deb|arch|apk) type_options+=" --type ${repo_sync_type,,}" ;; + *) + echo "Unknown type '${repo_sync_type}' for ${MODULE}, expected one of: rpm, deb, arch, apk." + exit 1 + ;; + esac + done + fi + + # Pruning removes files that upstream no longer publishes, matching the + # delete behavior of the other sync methods. Default it on, but allow it to + # be turned off for repositories that should keep old packages around. + parse_bool prune "true" "${MODULE}_prune" + + # A module may mirror several repositories in one run, so the source is a + # list of URLs. Quote each so the eval below does not glob or re-split them. + read -r -a repo_sync_urls <<< "${source:?}" + if (( ${#repo_sync_urls[@]} == 0 )); then + echo "No source URLs defined for ${MODULE}." + exit 1 + fi + quoted_urls="" + for repo_sync_url in "${repo_sync_urls[@]}"; do + quoted_urls+=" '${repo_sync_url}'" + done + + # repo-sync writes its own trace files, so rather than calling + # save_trace_file this module tells repo-sync what to record. The details + # are the same ones the other methods put in their trace files, passed + # through from the INFO_* configuration rather than configured a second time + # in repo-sync's own config file. Tracing is stated either way, so a + # repo-sync config file cannot contradict this module's setting. + # These values are shell quoted with printf rather than wrapped in single + # quotes, as a maintainer name may itself contain an apostrophe. + trace_options="--no-trace" + if [[ $save_trace_enabled ]]; then + trace_options="--trace --trace-host $(printf '%q' "${mirror_hostname:?}")" + for trace_field in \ + "maintainer:${INFO_MAINTAINER:-}" \ + "sponsor:${INFO_SPONSOR:-}" \ + "country:${INFO_COUNTRY:-}" \ + "location:${INFO_LOCATION:-}" \ + "throughput:${INFO_THROUGHPUT:-}" + do + trace_value="${trace_field#*:}" + if [[ $trace_value ]]; then + trace_options+=" --trace-${trace_field%%:*} $(printf '%q' "$trace_value")" + fi + done + fi + + # Capture the sync's output so its run summary can be read from it. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + + # Run repo-sync to synchronize the repositories. The destination is always + # the last argument. + eval "$sync_timeout" "$REPO_SYNC_BIN" "$options" \ + ${trace_options:+$trace_options} \ + sync ${type_options:+$type_options} ${sync_options:+$sync_options} \ + ${prune:+--prune} \ + "$quoted_urls" "'${repo:?}'" | tee -a "$LOGFILE_SYNC" + RT=${PIPESTATUS[0]} + + # repo-sync ends a run by printing a summary of what it moved. It states + # whether the repository changed itself, which is taken as given rather + # than inferred from the counts, as a dry run reports work it only + # planned. + repo_sync_changed="" + if grep -q '^Repository changed: true' "$LOGFILE_SYNC" 2>/dev/null; then + repo_sync_changed="true" + fi + set_sync_counts \ + "$(sum_log_stat "Number of files transferred" "$LOGFILE_SYNC")" \ + "$(sum_log_stat "Number of files pruned" "$LOGFILE_SYNC")" \ + "$repo_sync_changed" + rm -f "$LOGFILE_SYNC" + if (( RT == 0 )); then post_successful_sync else @@ -874,15 +1398,45 @@ ftp_sync() { module_config "$1" eval source="\$${MODULE}_source" - # Run AWS client to sync the S3 bucket. - $sync_timeout lftp <<< "mirror -v --delete --no-perms $options '${source:?}' '${repo:?}'" + # Keep --delete from removing the trace file this module publishes. lftp + # takes a glob here, matched against the path below the mirrored directory. + trace_exclude="" + if [[ $save_trace_enabled ]]; then + trace_exclude="--exclude-glob $(trace_relative_path) " + fi + + # lftp's mirror writes the commands it runs to its log, one per file it + # transfers or removes, which is what the changes are counted from. The + # verbose output it prints is meant for a person to read and is translated + # to the running locale, so it is not counted from. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + + # Mirror the source tree with lftp. + sync_started=$(date +%s) + $sync_timeout lftp <<< "mirror -v --log='${LOGFILE_SYNC}' --delete --no-perms ${trace_exclude}$options '${source:?}' '${repo:?}'" RT=${PIPESTATUS[0]} + sync_ended=$(date +%s) + + # Transfers are logged as a get command, in whichever form lftp chose for + # the transfer, and removals as an rm. Removed directories are logged as + # rmdir and are not counted, as they hold no content of their own. + set_sync_counts \ + "$(count_sync_events "$LOGFILE_SYNC" '(m|p)?get1? ')" \ + "$(count_sync_events "$LOGFILE_SYNC" 'rm ')" + rm -f "$LOGFILE_SYNC" + if (( RT == 0 )); then post_successful_sync else post_failed_sync fi + if [[ $save_trace_enabled ]]; then + save_trace_file + fi + run_post_successful_sync_hook log_end_header } @@ -897,6 +1451,17 @@ wget_sync() { options="--mirror --no-host-directories --no-parent" fi + # Capture the sync's output so what it changed can be counted from it. The + # counting happens out here rather than in the subshell below, which + # cannot hand a count back to the hooks. + LOGFILE_SYNC="${LOGFILE}.sync" + echo -n > "$LOGFILE_SYNC" + cleanup_files+=("$LOGFILE_SYNC") + + # The sync runs in a subshell, so it is timed from out here where the + # result is still readable once it has finished. wget never deletes, so + # unlike the other methods the trace needs no protection from this run. + sync_started=$(date +%s) ( trap - EXIT # Make sure the repo directory exists and we are in it. @@ -909,8 +1474,10 @@ wget_sync() { exit 1 fi - # Run wget with configured options. - eval "$sync_timeout" wget "$options" "'${source:?}'" + # Run wget with configured options. wget reports on its progress + # through stderr, which is what has to be captured to see what it + # retrieved. + eval "$sync_timeout" wget "$options" "'${source:?}'" 2>&1 | tee -a "$LOGFILE_SYNC" RT=${PIPESTATUS[0]} if (( RT == 0 )); then post_successful_sync @@ -919,10 +1486,25 @@ wget_sync() { fi ) RT=$? + sync_ended=$(date +%s) + + # wget closes a run by reporting what it downloaded. Mirroring only + # retrieves what changed upstream, so that count is the count of changed + # files; a run that found nothing new prints no such report at all, and + # wget never deletes. + set_sync_counts \ + "$(awk '$1=="Downloaded:" {total+=$2} END {print total+0}' "$LOGFILE_SYNC" 2>/dev/null)" \ + 0 + rm -f "$LOGFILE_SYNC" + if (( RT != 0 )); then exit "$RT" fi + if [[ $save_trace_enabled ]]; then + save_trace_file + fi + run_post_successful_sync_hook log_end_header } @@ -959,7 +1541,7 @@ rsync_sync() { *) echo "Unknown option $1" echo - print_help "$@" + print_help 1 ;; esac done @@ -994,10 +1576,14 @@ rsync_sync() { if [[ -z $last_modified ]]; then echo "Could not determine upstream last-modified, proceeding with sync." else - last_modified_unix=$(date -u +%s -d "$last_modified") - + # If the date cannot be parsed, proceed with the sync just + # as when the header is missing; an empty value would + # otherwise be read as the epoch and skip every sync until + # the timestamp minimum passes. + if ! last_modified_unix=$(date -u +%s -d "$last_modified" 2>/dev/null); then + echo "Could not parse upstream last-modified '${last_modified}', proceeding with sync." # If last modified is greater than our max age, it wasn't modified recently and we should not rsync. - if (( now-last_modified_unix > ${upstream_max_age:-0} )); then + elif (( now-last_modified_unix > ${upstream_max_age:-0} )); then echo "Skipping sync as upstream wasn't updated recently." exit 88 fi @@ -1142,6 +1728,12 @@ rsync_sync() { fi fi + # Count what the sync changed, before the logs it is counted from are + # cleaned up below. Both stages write into the same repository, so their + # logs are read as one. + read -r rsync_transferred rsync_deleted < <(count_rsync_changes "$LOGFILE_STAGE1" "${LOGFILE_STAGE2:-}") + set_sync_counts "$rsync_transferred" "$rsync_deleted" + # At this point we are successful. post_successful_sync @@ -1152,7 +1744,7 @@ rsync_sync() { fi # Save trace information. - if [[ $repo_type ]]; then + if [[ $save_trace_enabled ]]; then save_trace_file fi rm -f "$LOGFILE_STAGE1" @@ -1239,10 +1831,12 @@ EOF eval "$pre_hook" fi - # Create archive update file. - docroot=$repo + # Create archive update file. The repo path is normalized without its + # trailing slash so module paths can be built with an explicit separator + # regardless of how the configuration spelled it. + docroot=${repo%/} for module in $modules; do - _auip="$docroot$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" + _auip="${docroot}/$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" touch "$_auip" # Register for cleanup so an interrupt does not leave these behind. cleanup_files+=("$_auip") @@ -1264,11 +1858,17 @@ EOF # Check if run was successful. if ! grep -q '^total size is' "$LOGFILE_SYNC"; then for module in $modules; do - rm -f "$docroot$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" + rm -f "${docroot}/$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" done post_failed_sync fi + # Count what the sync changed, before the log it is counted from is + # cleaned up below. Every module quick-fedora-mirror synchronized reports + # into the one log, so the counts cover them all. + read -r qfm_transferred qfm_deleted < <(count_rsync_changes "$LOGFILE_SYNC") + set_sync_counts "$qfm_transferred" "$qfm_deleted" + # At this point we are successful. post_successful_sync @@ -1278,18 +1878,22 @@ EOF eval "$post_hook" fi - # Save trace information. - if [[ $repo_type ]]; then + # Save trace information. Each sub-module gets its own trace, so repo is + # pointed at each in turn and then restored, as the configured repo is what + # the post-successful-sync hook expects to be given. + if [[ $save_trace_enabled ]]; then + module_repo=$repo for module in $modules; do - repo="$docroot$(module_dir "$module")" + repo="${docroot}/$(module_dir "$module")" save_trace_file done + repo=$module_repo fi rm -f "$LOGFILE_SYNC" # Remove archive update file. for module in $modules; do - rm -f "$docroot$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" + rm -f "${docroot}/$(module_dir "$module")/Archive-Update-in-Progress-${mirror_hostname:?}" done # If report mirror configuration file provided, run report mirror. @@ -1303,9 +1907,9 @@ EOF log_end_header } -# If no arugments are provided, we can print help. +# If no arguments are provided, we can print help. if (( $# < 1 )); then - print_help "$@" + print_help fi # Parse arguments. @@ -1318,7 +1922,7 @@ while (( $# > 0 )); do ;; # If help is requested, print it. -h|h|help|--help) - print_help "$@" + print_help ;; # Print version. -v|--version) @@ -1338,6 +1942,8 @@ while (( $# > 0 )); do s3cmd_sync "$@" elif [[ "${sync_method:?}" == "s5cmd" ]]; then s5cmd_sync "$@" + elif [[ "${sync_method:?}" == "repo-sync" ]] || [[ "${sync_method:?}" == "repo_sync" ]]; then + repo_sync_sync "$@" elif [[ "${sync_method:?}" == "ftp" ]]; then ftp_sync "$@" elif [[ "${sync_method:?}" == "wget" ]]; then @@ -1353,7 +1959,7 @@ while (( $# > 0 )); do # No module was found, so give help. echo "Unknown module '$1'" echo - print_help "$@" + print_help 1 ;; esac done