GRMrGecko/mirror-sync
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Add repo-sync method, sync change counts, and configurable trace saving

Browse source 
mirror-sync:
- Add a repo-sync sync method for rpm/deb/arch/apk repositories, with
  installer, prune setting, an optional type filter, and trace fields passed
  through to it.
- Count what each sync changed and publish SYNC_FILES_* / SYNC_CHANGED to
  the post successful sync hook, so a hook can stay idle on a no-op run.
- Make trace saving configurable with save_trace (global and per module)
  and publish traces from every method, not just rsync. Exclude the local
  trace from deletes, and stop hierarchy entries stacking up per run.
- Add parse_bool so boolean settings reject values that would otherwise be
  read as their opposite; keep an explicit bare=false from being
  auto-detected away, and require hide_remote on the clone path too.
- MIRROR_SYNC_CONF names an alternate configuration file.
- Fix docroot joining when repo has a trailing slash, unparsable
  last-modified headers skipping every sync, s3cmd --host taking a full
  URL, and the average rate line for methods with no byte count.
- Exit non-zero when help is printed because of a mistake.

mirror-file-generator:
- Match du summary paths exactly instead of by substring, so sibling repos
  sharing a prefix no longer return several sizes.
- Fall back to the default icon when no image is found, and stop the
  fallback recursing into itself.
- Validate sections against SECTIONS, count repos written to decide whether
  to publish the index, and create the log directory if missing.
- MIRROR_SYNC_CONF support, and exit non-zero on an unknown mirror.

Docs: rewrite README around quick start and sync methods, and move the
file generator reference to docs/mirror-file-generator.md.
This commit is contained in:
James Coleman 2026-07-27 17:41:17 -05:00
parent ccb938e300
commit 68cb51bb6f
5 changed files with 1497 additions and 607 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
LICENSE
View file

@ -1,4 +1,4 @@
Copyright (c) 2023 Mr. Gecko's Media (James Coleman). http://mrgeckosmedia.com/
Copyright (c) 2026 Mr. Gecko's Media (James Coleman). http://mrgeckosmedia.com/
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal

1109
README.md
View file

File diff suppressed because it is too large Load diff

130
docs/mirror-file-generator.md Normal file
View file

@ -0,0 +1,130 @@
# mirror-file-generator
A tool to generate common mirror info files at the mirror document root.
It reads the same `/etc/mirror-sync.conf` file as [mirror-sync](../README.md), and follows the same two configuration rules: `MODULES` lists your modules, and every module setting is written as `<module>_<setting>`. See [How the configuration file works](../README.md#how-the-configuration-file-works) if you have not read it yet. `MIRROR_SYNC_CONF` in the environment names a different file to read, the same as it does for mirror-sync.
## Contents
- [Module settings](#module-settings)
- [Custom modules](#custom-modules)
- [Mirrors](#mirrors)
- [Sections](#sections)
- [Templates](#templates)
- [General defaults](#general-defaults)
## Module settings
These settings are shared with mirror-sync and are read straight from its configuration:
| Setting | Description |
| --- | --- |
| `repo` | Used to verify a module is the same repo under the mirror. |
| `sync_method` | Used to determine if the module is a qfm mirror. |
| `timestamp` | Used for sync time. |
| `dusum` | Used for the disk usage summary. |
These settings are added by this tool:
| Setting | Default | Description |
| --- | --- | --- |
| `section` | `section_default` | What section to associate the repo with. |
| `repo_title` | directory name | A title for the repo to show instead of the directory name. |
| `repo_icon` | tux | The repo icon. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script automatically copies or downloads the icon into the image folder. |
| `repo_icon_dark` | *(none)* | The dark-mode variant of the repo icon, accepting the same sources as `repo_icon`. When set, browsers detecting a dark OS theme display this instead. If omitted, the light icon is used in all themes. |
| `repo_description` | *(none)* | A description to show at the bottom of the repo card. |
| `repo_skip` | *(unset)* | This repo should not be put in generated files. |
| `disable_size_calc` | *(unset)* | Set to `1` if you do not want a size to be calculated. |
| `timestamp_file_stat` | *(none)* | Path to a file or folder that is updated when changes are made, stat'd to determine the last sync time. Use this when you have no timestamp file containing a UNIX timestamp. |
## Custom modules
If you have a repo that is not synced via mirror-sync, but you still want to customize how it looks on the generated `index.html`, list it in `CUSTOM_MODULES` and then configure it with any of the settings below.
`CUSTOM_MODULES` is separate from `MODULES` — a repo goes in one list or the other, never both.
Available settings, all behaving exactly as they do for a regular module:
`repo`, `timestamp`, `dusum`, `section`, `repo_title`, `repo_icon`, `repo_icon_dark`, `repo_description`, `repo_skip`, `disable_size_calc`, `timestamp_file_stat`
### Example
```bash
CUSTOM_MODULES="example example2"
example_repo="/home/mirror/http/"
example_section="official"
example_repo_title="Test repo"
example_repo_icon="terminal.png"
example_repo_description="Test, this is a test."
example2_repo="/home/mirror/windows/"
example2_repo_icon="windows.png"
```
## Mirrors
You can define multiple mirrors for this tool to generate files for. Each mirror can have its own templates and repos, and they are configured the same way modules are — listed in a variable, then configured by prefix. Because mirror names share the same flat namespace, it is worth prefixing your mirror names with `mirror_`.
| Setting | Default | Description |
| --- | --- | --- |
| `path` | *(required)* | The path to the mirror under which repos are stored. |
| `title` | the mirror name | A title for the mirror. |
| `logo` | tux | The logo. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png). The script automatically copies or downloads it into the image folder. |
| `logo_dark` | *(none)* | The dark-mode variant of the mirror logo, accepting the same sources as `logo`. When set, browsers detecting a dark OS theme display this instead. If omitted, the light logo is used in all themes. |
| `description` | *(none)* | A description to place below the logo. May be HTML formatted. |
| `provider_site` | *(none)* | A site for the global footer generation. |
| `provider_name` | *(none)* | A name for the global footer generation. |
### Example
```bash
MIRRORS="mirror_example"
mirror_example_path="/home/mirror/mirror_docroot"
mirror_example_title="My company"
mirror_example_logo="http://example.com/logo.png"
mirror_example_logo_dark="http://example.com/logo-dark.png"
mirror_example_description="A public mirror provided by this cool company."
mirror_example_provider_site="http://www.example.com/"
mirror_example_provider_name="Company"
```
## Sections
Define the sections used in `index.html` with the `SECTIONS` variable, which defaults to `official unofficial`. Set a default section with `section_default`, which defaults to `unofficial`. A title is auto generated as `{SECTION} Mirrors`, which you can customize with a variable named `section_{SECTION}_title`.
## Templates
Where templates are stored is configured by `template_dir`, which defaults to `/usr/local/share/mirror-file-generator/templates`. Default files are stored under the `default` sub directory, and customizations for individual mirrors go under a sub directory named after that mirror. You can add icons and logos into these template directories as well.
Default templates:
| File | Purpose |
| --- | --- |
| `header.html` | The main index header. |
| `section.html` | Template for a section. |
| `repo.html` | The repo card template. |
| `footer.html` | The footer of the index. |
| `footer.txt` | Template for the global footer file. |
The default `header.html` includes a `@media (prefers-color-scheme: dark)` CSS block that automatically switches the page to a dark theme when the OS reports a dark preference. The logo and repo icon templates use the HTML `<picture>` element, so a dark variant image is served to dark-mode browsers when `logo_dark` or `repo_icon_dark` is configured.
## General defaults
| Setting | Default | Description |
| --- | --- | --- |
| `index_generate` | `1` | Whether to generate the `index.html` file. `1` enabled, `0` disabled. |
| `index_file_name` | `index.html` | Alternative name for the index file. |
| `footer_generate` | `1` | Whether to generate a footer file that can be configured as the mirror's global footer. `1` enabled, `0` disabled. |
| `footer_file_name` | `footer.txt` | Alternative file name for the footer file. |
| `dir_sizes_generate` | `1` | Whether to generate the directory sizes file. `1` enabled, `0` disabled. |
| `dir_sizes_file_name` | `DIRECTORY_SIZES.TXT` | Alternative file name for the directory sizes file. |
| `dir_sizes_unknown_path` | `$HOME/dusum/unknown_dirs` | Path to store directory size summaries for unknown repos. |
| `dir_sizes_human_readable` | `1` | `1` for human readable, `0` for kbytes. |
| `icons_dir_name` | `img` | Where to store logos and icons, relative to the mirror path. |
| `icons_default_source` | [Dashboard Icons](https://github.com/homarr-labs/dashboard-icons/tree/main/png) | The default URL to pull icons from. |
| `icons_default_img` | `tux.png` | A default file to use if the icon or logo is either undefined or inaccessible. |
| `icons_local_repo` | `$HOME/dashboard-icons` | Local path to a cloned copy of the dashboard-icons git repository. When this directory exists, icons are served from it instead of being fetched over HTTP, avoiding per-icon network requests. |
| `icons_repo_url` | `https://github.com/homarr-labs/dashboard-icons.git` | Git URL used to clone the dashboard-icons repository into `icons_local_repo` if it does not already exist. Set to an empty string to disable automatic cloning. |
| `icons_repo_refresh` | `604800` (7 days) | How often, in seconds, the local dashboard-icons clone is pulled for updates. |

135
mirror-file-generator.sh
View file

@ -40,16 +40,32 @@ if (( EUID == 0 )); then
exit 1
fi
# Load the required configuration file or quit.
if [[ -f /etc/mirror-sync.conf ]]; then
# Load the required configuration file or quit. The default is where a system
# installation keeps it; MIRROR_SYNC_CONF names another one, so the script can
# be pointed at a configuration of its own for testing or for a mirror run out
# of a home directory.
CONFIG_FILE="${MIRROR_SYNC_CONF:-/etc/mirror-sync.conf}"
if [[ -f $CONFIG_FILE ]]; then
# shellcheck source=/dev/null
source /etc/mirror-sync.conf
source "$CONFIG_FILE"
else
echo "No configuration file defined, please setup a proper configuration file."
echo "No configuration file at '${CONFIG_FILE}', please setup a proper configuration file."
exit 1
fi
# Print the help for this command.
# Repos are collected into one list per section and only the sections named in
# SECTIONS are assembled into the index, so a default that is not one of them
# would drop every repo that does not name a section of its own.
# We are purposely adding quotes to match the space.
# shellcheck disable=SC2076
if ((index_generate)) && [[ ! " $SECTIONS " =~ " $section_default " ]]; then
echo "section_default '${section_default}' is not one of SECTIONS (${SECTIONS})."
exit 1
fi
# Print the help for this command. An exit status may be given so that printing
# help because of a mistake (an unknown mirror) exits non-zero, letting cron or
# a calling script tell a misconfiguration apart from a run that asked for help.
print_help() {
echo "Mirror File Generator (${VERSION})"
echo
@ -62,7 +78,7 @@ print_help() {
done
echo
echo "Note: Defaults to generating files for all mirrors."
exit
exit "${1:-0}"
}
# Output message in log format and to logger.
@ -111,6 +127,7 @@ image_copy() {
extension="${extension%\?*}"
local save_path="$path/$icons_dir_name/$file_name.$extension"
local http_code
local t_file
# Determine if the saved file needs to be updated.
local needs_update=0
@ -126,6 +143,14 @@ image_copy() {
if (( $(stat --format='%Y' "$file") != $(stat --format='%Y' "$save_path") )); then
needs_update=1
fi
else
# A name that resolves to a template file is copied the same way a local
# file is, so it goes stale the same way and is compared by mtime too.
t_file=$(template_file "$file")
if [[ -f $t_file ]] \
&& (( $(stat --format='%Y' "$t_file") != $(stat --format='%Y' "$save_path") )); then
needs_update=1
fi
fi
if (( needs_update )); then
@ -134,11 +159,15 @@ image_copy() {
# If failure, and is not the default image, attempt to grab the default file.
# --fail suppresses writing the response body on HTTP errors; rm -f cleans up
# any partial file so the fallback recursion isn't blocked by the needs_update check.
# The fallback must be skipped when we are already fetching the default
# image, otherwise a download failure recurses into itself forever.
if ! http_code=$(curl -sf --write-out "%{http_code}" -o "$save_path" "$file") \
|| ( ((http_code!=200)) && [[ "$file" != "$icons_default_img" ]] \
&& [[ "$file" != "$icons_default_source/$icons_default_img" ]] ); then
|| ((http_code!=200)); then
rm -f "$save_path"
image_copy "$icons_default_img" "$file_name"
if [[ "$file" != "$icons_default_img" ]] \
&& [[ "$file" != "$icons_default_source/$icons_default_img" ]]; then
image_copy "$icons_default_img" "$file_name"
fi
return
fi
# If the file exists, copy it preserving mtime.
@ -146,7 +175,6 @@ image_copy() {
cp -p "$file" "$save_path"
else
# Check to see if a template file exists with the file name.
local t_file
t_file=$(template_file "$file")
# If the file exists, copy it preserving mtime.
if [[ -f $t_file ]]; then
@ -155,6 +183,23 @@ image_copy() {
# If nothing else exists, try grabbing from the default source.
image_copy "$icons_default_source/$file" "$file_name"
return
else
# Nothing produced an image. Fall back to the default one, just
# as a failed download does, so a name that the icon source does
# not carry renders as the default rather than as a broken
# image. This matters most when the icon source is the local
# dashboard-icons clone, where a missing name is simply an
# absent file rather than a failed request. Skipped when this
# call is already placing the default, as there would then be
# nothing left to fall back to.
if [[ "$file" != "$icons_default_img" ]] \
&& [[ "$file" != "$icons_default_source/$icons_default_img" ]]; then
image_copy "$icons_default_img" "$file_name"
return
fi
# Warn on stderr, as stdout is this function's return value.
log "Warning: no image found for '$file_name', tried '$file'" >&2
return
fi
fi
fi
@ -162,6 +207,25 @@ image_copy() {
echo "$icons_dir_name/$file_name.$extension"
}
# Pull the size of a directory out of a du summary file. The path is matched
# exactly (with any trailing slashes removed) rather than with grep, as a
# substring match would also hit sibling repos sharing a prefix (for example
# "almalinux" matching "almalinux-stage") and return multiple sizes.
#
# The path is taken as everything after the size column rather than as the
# second field, since a repo path containing a space spans several fields and
# would otherwise be compared as its first word alone and never match.
dusum_size() {
awk -v dir="$1" '
{
p = $0
sub(/^[[:space:]]*[0-9]+[[:space:]]+/, "", p)
sub(/\/+$/, "", p)
}
p == dir { print $1; exit }
' "$2"
}
# Read the module's configuration.
read_config() {
eval timestamp="\${${MODULE}_timestamp:-}"
@ -190,7 +254,7 @@ while (( $# > 0 )); do
;;
# If help is requested, print it.
-h|h|help|--help)
print_help "$@"
print_help
;;
# Print version.
-v|--version)
@ -220,7 +284,7 @@ while (( $# > 0 )); do
if [[ -z $foundMirror ]]; then
echo "Unknown mirror '$mirror'"
echo
print_help "$@"
print_help 1
fi
# Add mirror to list.
@ -234,6 +298,7 @@ while (( $# > 0 )); do
done
# Redirect stdout to both stdout and log file.
mkdir -p "$(dirname "$LOGFILE")"
exec 1> >(tee -a "$LOGFILE")
# Redirect errors to stdout so they also are logged.
exec 2>&1
@ -370,6 +435,10 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
# Keep record of total kbytes of repo sizes.
totalKBytes=0
# Count the repos written into the index, so an index that ended up with no
# repos at all is not published over a good one.
repos_added=0
# For each directory under the mirror, generate repo data.
for dir in "$path"/*; do
# Some repos may be built with symbolic links, so get the real path.
@ -448,7 +517,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
# If a directory usage summary exists and we're not skipping, parse the size.
if [[ -n $dusum ]] && [[ -f $dusum ]] && ((${repo_skip:-0} == 0)); then
repo_size_kb=$(grep "$real_dir" "$dusum" | awk '{print $1}')
repo_size_kb=$(dusum_size "$real_dir" "$dusum")
if [[ -n $repo_size_kb ]]; then
totalKBytes=$((totalKBytes+repo_size_kb))
repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec)
@ -480,7 +549,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
repo_sync_time=$(date -d "@$(cat "$timestamp")" '+%c')
fi
if [[ -n ${dusum:-} ]] && [[ -f $dusum ]] && ((${repo_skip:-0} == 0)); then
repo_size_kb=$(grep "$real_dir" "$dusum" | awk '{print $1}')
repo_size_kb=$(dusum_size "$real_dir" "$dusum")
if [[ -n $repo_size_kb ]]; then
totalKBytes=$((totalKBytes+repo_size_kb))
repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec)
@ -541,10 +610,18 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
if [[ -z ${repo_size:-} ]] && ((${disable_size_calc:-0} == 0)); then
unknown_path="$dir_sizes_unknown_path/$mirror/$dir_name"
# If we should update the directory usage sizes, do so.
# shellcheck disable=SC2076
if ((update_unknown_dir_size)) \
&& [[ ! " ${repo_sizes_updated[*]} " =~ " ${real_dir} " ]]; then
# If we should update the directory usage sizes, do so. The
# already-updated check compares exact strings rather than a
# regex, as a path containing regex metacharacters (such as dots)
# could otherwise match a different repo and skip this update.
already_updated=0
for updated_repo in "${repo_sizes_updated[@]}"; do
if [[ $updated_repo == "$real_dir" ]]; then
already_updated=1
break
fi
done
if ((update_unknown_dir_size)) && ((already_updated == 0)); then
# Add to list of repos with updated sizes.
repo_sizes_updated+=("$real_dir")
@ -565,7 +642,7 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
# If the unknown repo size path exists, grab it.
if [[ -f $unknown_path ]]; then
repo_size_kb=$(grep "$real_dir" "$unknown_path" | awk '{print $1}')
repo_size_kb=$(dusum_size "$real_dir" "$unknown_path")
if [[ -n $repo_size_kb ]]; then
totalKBytes=$((totalKBytes+repo_size_kb))
repo_size=$(echo "$repo_size_kb*1024" | bc | numfmt --to=iec)
@ -577,10 +654,21 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
repo_size=$(html_encode "${repo_size:-Unknown}")
export repo_size
# If we're generating the index.html, do so.
# If we're generating the index.html, do so. A section that is not one
# of SECTIONS has no list to append to, so the repo would be written to
# a file that is never assembled into the index and is left behind in
# the mirror root. Report it and fall back to the default section, so a
# typo costs the repo its placement rather than its listing.
if ((index_generate)); then
section=${section:-$section_default}
# We are purposely adding quotes to match the space.
# shellcheck disable=SC2076
if [[ ! " $SECTIONS " =~ " $section " ]]; then
log "Unknown section '$section' for $dir_name, using '$section_default'"
section=$section_default
fi
envsubst < "$(template_file repo.html)" >> "$index_file_temp.$section"
repos_added=$((repos_added+1))
fi
# If we're generating the repo size file, add to it.
@ -610,8 +698,11 @@ for ((i=0; i<${#selected_mirrors[@]}; i++)); do
# Add footer subsituting environment variables.
envsubst < "$(template_file footer.html)" >> "$index_file_temp"
# Verify the index temp contains a repo before moving into place.
if grep -q "Last Sync:" "$index_file_temp"; then
# Verify the index temp contains a repo before moving into place. The
# count of repos written is used rather than grepping the built file for
# text from the stock template, so that a customized repo.html does not
# stop the index from ever being published.
if ((repos_added > 0)); then
[[ -f $index_file_path ]] && rm -f "$index_file_path"
mv "$index_file_temp" "$index_file_path"
else

720
mirror-sync.sh
View file

File diff suppressed because it is too large Load diff