August 9, 2019

Fedora -- Setting up a local mirror

For some time we have run a local "private" mirror, both to reduce our load on the mainstream mirrors and to allow our group of local machines to update and install more quickly. Our old mirror scheme used rsync against a /pub that was exported via rsync by whatever mirror site we chose.

Exporting /pub seems to have fallen out of vogue. These days you find sites exporting /fedora-enchilada or /fedora-buffet What I have typically done when investigating a prospective new site is to run the command: 

rsync rsync://download-ib01.fedoraproject.org
When I ran this command against the site "download-ib01.fedoraproject.org", I got the following. No /pub -- and an exhortation to not simply use "bare rsync". 
----------------------------
Fedora Master Mirror Servers
----------------------------

If you are using bare rsync to sync content, please stop and
take a look at https://pagure.io/quick-fedora-mirror.

See http://fedoraproject.org/wiki/Infrastructure/Mirroring for
further instructions.
----------------------------

You have connected to download-ib01.fedoraproject.org

fedora-alt     	non-Fedora Alternative Content
fedora-archive 	Fedora Release Archives
fedora-enchilada	Fedora - The whole enchilada
fedora-buffet  	Fedora - The whole buffet. All you can eat.
fedora-epel    	Extra Packages for Enterprise Linux
fedora-linux-releases	Fedora Linux Releases
fedora-linux-development	Fedora Linux Development
fedora-linux-updates	Fedora Linux Updates
fedora-secondary	Fedora Secondary Archs
fedora-stage   	Staging directory
fedora-live-respins	Fedora Live Respins
A quick look at the quick-fedora-mirror page gives the following warning:
A full rsync of fedora-buffet0 can take hours just to receive the file list, due to the fact that there are over 11 million files. This has to happen for every client, every time they want to update, which is murderous on the download servers and worse on the backend NFS server. It also slows the propagation of important updates because mirrors simply can't poll often.
So it sounds like a good citizen would want to use "quick-fedora-mirror"

Setting up Quick Fedora Mirror

Surprisingly, this is not available via DNF (i.e. as an RPM in the usual package collection). You would think something as important to the Fedora ecosystem would be! And none of the links on the page you are redirected yield the script or anything like it. Is it on Github? No! It turns out that "Pagure" is some new Git compatible replacement for Github for whatever reason. Think of it as a badly done Github and be thankful that it uses Git. The documentation omits the crucial first step, getting the software!
Getting it is as simple as: 
git clone https://pagure.io/quick-fedora-mirror.git
It is written is zsh, which is a little offbeat, but OK with me as long as it works and I don't have to monkey with it. Anything (with the exception of Tcl) would be better than bash. Since nobody ever uses or even installs zsh, you have to do this: 
su
dnf install zsh
Note that if this was an RPM as it should be, it would pull in zsh as a dependency. Why not ruby or python -- or even perl?

After this, you can begin fooling with it as per the instructions given.

I made some simple edits to the config file and started it running. It may be quick to do updates, but getting a mirror started the first time simply takes a long time.

Note also that it will pull all the releases from the source site. They estimate that each release will take up 250G of disk space and the entire mirror is 1.5 TB and growing! I really only want the latest two releases at most. The script does not provides a way to select only certain releases.

The script does have a pattern that can be used to exclude files matching certain patterns. An example pattern (that comes commented out) is: # FILTEREXP='(/i386|/armhfp|/source|/SRPMS|/debug/|\.iso|\.img|\.qcow2|\.raw\.xz|\.box|/releases/test)' Note that you can exclude armhfp and debug, which certainly make sense in my situation, as does i386 (we are a 64 bit house these days).

Once I find a way to exclude old releases I will start using this script.

Back to brute force

It turns out that the loss of /pub on many sites is not the end of the world. I need some path that gives me updates and releases.

Under the old /pub scheme this was typically /pub/fedora/linux.
Under the new /fedora-enchilada scheme this seems to be fedora-enchilada/linux.

Have any comments? Questions? Drop me a line!

Adventures in Computing / tom@mmto.org