mirror-sync/docs/mirror-file-generator.md
James Coleman 68cb51bb6f Add repo-sync method, sync change counts, and configurable trace saving
mirror-sync:
- Add a repo-sync sync method for rpm/deb/arch/apk repositories, with
  installer, prune setting, an optional type filter, and trace fields passed
  through to it.
- Count what each sync changed and publish SYNC_FILES_* / SYNC_CHANGED to
  the post successful sync hook, so a hook can stay idle on a no-op run.
- Make trace saving configurable with save_trace (global and per module)
  and publish traces from every method, not just rsync. Exclude the local
  trace from deletes, and stop hierarchy entries stacking up per run.
- Add parse_bool so boolean settings reject values that would otherwise be
  read as their opposite; keep an explicit bare=false from being
  auto-detected away, and require hide_remote on the clone path too.
- MIRROR_SYNC_CONF names an alternate configuration file.
- Fix docroot joining when repo has a trailing slash, unparsable
  last-modified headers skipping every sync, s3cmd --host taking a full
  URL, and the average rate line for methods with no byte count.
- Exit non-zero when help is printed because of a mistake.

mirror-file-generator:
- Match du summary paths exactly instead of by substring, so sibling repos
  sharing a prefix no longer return several sizes.
- Fall back to the default icon when no image is found, and stop the
  fallback recursing into itself.
- Validate sections against SECTIONS, count repos written to decide whether
  to publish the index, and create the log directory if missing.
- MIRROR_SYNC_CONF support, and exit non-zero on an unknown mirror.

Docs: rewrite README around quick start and sync methods, and move the
file generator reference to docs/mirror-file-generator.md.
2026-07-28 09:13:19 -05:00

8 KiB

mirror-file-generator

A tool to generate common mirror info files at the mirror document root.

It reads the same /etc/mirror-sync.conf file as mirror-sync, and follows the same two configuration rules: MODULES lists your modules, and every module setting is written as <module>_<setting>. See How the configuration file works if you have not read it yet. MIRROR_SYNC_CONF in the environment names a different file to read, the same as it does for mirror-sync.

Contents

Module settings

These settings are shared with mirror-sync and are read straight from its configuration:

Setting Description
repo Used to verify a module is the same repo under the mirror.
sync_method Used to determine if the module is a qfm mirror.
timestamp Used for sync time.
dusum Used for the disk usage summary.

These settings are added by this tool:

Setting Default Description
section section_default What section to associate the repo with.
repo_title directory name A title for the repo to show instead of the directory name.
repo_icon tux The repo icon. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from Dashboard Icons. The script automatically copies or downloads the icon into the image folder.
repo_icon_dark (none) The dark-mode variant of the repo icon, accepting the same sources as repo_icon. When set, browsers detecting a dark OS theme display this instead. If omitted, the light icon is used in all themes.
repo_description (none) A description to show at the bottom of the repo card.
repo_skip (unset) This repo should not be put in generated files.
disable_size_calc (unset) Set to 1 if you do not want a size to be calculated.
timestamp_file_stat (none) Path to a file or folder that is updated when changes are made, stat'd to determine the last sync time. Use this when you have no timestamp file containing a UNIX timestamp.

Custom modules

If you have a repo that is not synced via mirror-sync, but you still want to customize how it looks on the generated index.html, list it in CUSTOM_MODULES and then configure it with any of the settings below.

CUSTOM_MODULES is separate from MODULES — a repo goes in one list or the other, never both.

Available settings, all behaving exactly as they do for a regular module:

repo, timestamp, dusum, section, repo_title, repo_icon, repo_icon_dark, repo_description, repo_skip, disable_size_calc, timestamp_file_stat

Example

CUSTOM_MODULES="example example2"

example_repo="/home/mirror/http/"
example_section="official"
example_repo_title="Test repo"
example_repo_icon="terminal.png"
example_repo_description="Test, this is a test."

example2_repo="/home/mirror/windows/"
example2_repo_icon="windows.png"

Mirrors

You can define multiple mirrors for this tool to generate files for. Each mirror can have its own templates and repos, and they are configured the same way modules are — listed in a variable, then configured by prefix. Because mirror names share the same flat namespace, it is worth prefixing your mirror names with mirror_.

Setting Default Description
path (required) The path to the mirror under which repos are stored.
title the mirror name A title for the mirror.
logo tux The logo. May be an http(s) link, a file path, a file stored in the template directory, or a png image name from Dashboard Icons. The script automatically copies or downloads it into the image folder.
logo_dark (none) The dark-mode variant of the mirror logo, accepting the same sources as logo. When set, browsers detecting a dark OS theme display this instead. If omitted, the light logo is used in all themes.
description (none) A description to place below the logo. May be HTML formatted.
provider_site (none) A site for the global footer generation.
provider_name (none) A name for the global footer generation.

Example

MIRRORS="mirror_example"

mirror_example_path="/home/mirror/mirror_docroot"
mirror_example_title="My company"
mirror_example_logo="http://example.com/logo.png"
mirror_example_logo_dark="http://example.com/logo-dark.png"
mirror_example_description="A public mirror provided by this cool company."
mirror_example_provider_site="http://www.example.com/"
mirror_example_provider_name="Company"

Sections

Define the sections used in index.html with the SECTIONS variable, which defaults to official unofficial. Set a default section with section_default, which defaults to unofficial. A title is auto generated as {SECTION} Mirrors, which you can customize with a variable named section_{SECTION}_title.

Templates

Where templates are stored is configured by template_dir, which defaults to /usr/local/share/mirror-file-generator/templates. Default files are stored under the default sub directory, and customizations for individual mirrors go under a sub directory named after that mirror. You can add icons and logos into these template directories as well.

Default templates:

File Purpose
header.html The main index header.
section.html Template for a section.
repo.html The repo card template.
footer.html The footer of the index.
footer.txt Template for the global footer file.

The default header.html includes a @media (prefers-color-scheme: dark) CSS block that automatically switches the page to a dark theme when the OS reports a dark preference. The logo and repo icon templates use the HTML <picture> element, so a dark variant image is served to dark-mode browsers when logo_dark or repo_icon_dark is configured.

General defaults

Setting Default Description
index_generate 1 Whether to generate the index.html file. 1 enabled, 0 disabled.
index_file_name index.html Alternative name for the index file.
footer_generate 1 Whether to generate a footer file that can be configured as the mirror's global footer. 1 enabled, 0 disabled.
footer_file_name footer.txt Alternative file name for the footer file.
dir_sizes_generate 1 Whether to generate the directory sizes file. 1 enabled, 0 disabled.
dir_sizes_file_name DIRECTORY_SIZES.TXT Alternative file name for the directory sizes file.
dir_sizes_unknown_path $HOME/dusum/unknown_dirs Path to store directory size summaries for unknown repos.
dir_sizes_human_readable 1 1 for human readable, 0 for kbytes.
icons_dir_name img Where to store logos and icons, relative to the mirror path.
icons_default_source Dashboard Icons The default URL to pull icons from.
icons_default_img tux.png A default file to use if the icon or logo is either undefined or inaccessible.
icons_local_repo $HOME/dashboard-icons Local path to a cloned copy of the dashboard-icons git repository. When this directory exists, icons are served from it instead of being fetched over HTTP, avoiding per-icon network requests.
icons_repo_url https://github.com/homarr-labs/dashboard-icons.git Git URL used to clone the dashboard-icons repository into icons_local_repo if it does not already exist. Set to an empty string to disable automatic cloning.
icons_repo_refresh 604800 (7 days) How often, in seconds, the local dashboard-icons clone is pulled for updates.