From 4ccc706c7090f634b723752415143465b77d5e22 Mon Sep 17 00:00:00 2001 From: Aaron Taylor Date: Sat, 7 Nov 2020 03:56:37 -0800 Subject: [PATCH] Added partial notes for new mail/git/web server to website. --- data/notes/mail_web_git_server.md | 583 ++++++++++++++++++++++++ data/notes/mail_web_git_server.metadata | 6 + 2 files changed, 589 insertions(+) create mode 100644 data/notes/mail_web_git_server.md create mode 100644 data/notes/mail_web_git_server.metadata diff --git a/data/notes/mail_web_git_server.md b/data/notes/mail_web_git_server.md new file mode 100644 index 0000000..a9d0d89 --- /dev/null +++ b/data/notes/mail_web_git_server.md @@ -0,0 +1,583 @@ +# Overview # + +These notes cover the creation of a server, hosted with Linode, running Debian +Linux, offering the following services: + + - Mail server + + - SMTP via postfix + + - IMAP via dovecot + + - Realtime blacklists for SPAM rejection + + - MySQL for virtual domain and user management. + + - Web server + + - Apache as central HTTP server with multiple vhosts. + + - [CMless](git://git.subgeniuskitty.com/cmless) as CGI for content management + + - ACME for automated SSL certificate management + + - Git server + + - SSH-based, authenticated read-write access to all git repositories + + - Anonymous read-only access to a subset of git repositories via: + + - Customized [gitweb](git://git.subgeniuskitty.com/gitweb-sgk) for + GUI git browsing with syntax highlighting, diffs, etc + + - Git-daemon for cloning repositories via the `git://` protocol + +These notes are a high-level checklist for my reference rather than a +step-by-step installation guide for the public. That means they make no attempt +to explain all options at each step, rather that they mention only the options +I use on my servers. It also means they use my domains, my file system paths, +etc in the examples. + + +# TODO List # + + - Set hostname to `sgk-main-2020` and ensure Apache recognizes it as the + ServerName (see: `systemctl status apache2`). + + - Setup SSL with automatic certificate renewal. + + - Find a reliable way to alert me when renewal fails. + + - Websites should auto-redirect to the SSL version of the site for newer + browsers only. All sites should still be accessible on pre-SSL vintage + computers. + + - Take a snapshot on Linode's backup service once the basic services are + operational. + + - Delete old mail/web/git vservers after downloading a disk image. + + - Setup log rotation, or at least make sure everything ties into whatever is + pre-configured on Debian for log rotation. + + - Finish this documentation. + + - Improve CSS on gitweb, especially for displaying READMEs. + + +# Basic Configuration # + + +## General Information ## + +**Name:** SGK-Main-2020 + +**OS:** Debian 10 + +**Creation Date:** 2020-11-01 + +**Filesystem Points of Interest:** + + - `/srv/apache_vhosts`: Contains websites hosted by Apache2. See + `/etc/apache2/sites_available` for vhost configurations. + + - `/srv/git`: Master location for bare git repositories. All are private. + + - `/srv/gitweb_cache`: Contains checked out copies of git repositories from + `/srv/git`, publicly visible via `gitweb` and cloneable via `git-daemon`. + + +## Preparation ## + +Setup DNS entries for `subgeniuskitty.com` and `logicavalanche.com` through +Linode. Remember to do IPv4 and IPv6 entries for the bare domain, `www`, +`mail`, and `git`. + +Create a new Debian 10 on Linode and update the system with `apt-get update && +apt-get upgrade`. + +Add a user via `adduser ataylor` and following the prompts. Edit +`/etc/ssh/sshd_config` to set `PermitRootLogin: no` and restart SSH. For both +the `ataylor` user and `root`, add the line `set mouse=` to `~/.vimrc` in order +to disable mouse support in `vim`, allowing normal mark-and-paste in the +terminal. + +Install useful packages: + + apt-get install net-tools screen bzip2 zip + + +# Web Server # + + +## HTTP: Apache2 ## + +Install Apache2. + + apt-get install apache2 + +Since we use `/srv` instead of `/var/www`, edit `/etc/apache2/apache2.conf` to +comment out the `` entry for `/var/www` and replace it with +this: + + + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + + +Make and edit the file `/srv/apache_vhosts/default/index.html` with (rewritten) +contents: + +

Invalid VHost

+

Contact user @at@ domain .dot. com

+ +Ensure everything under `/srv/apache_vhosts`, including that directory itself, +is owned recursively by `www-data:www-data`. + +Edit `/etc/apache2/sites-available/000-default.conf` and +`/etc/apache2/sites-available/000-default.conf`, changing references for the +default sites from `/var/www/...` to `/srv/apache_vhosts/...` as necessary. + +Reload Apache2 with `systemctl reload apache2` and check status with `systemctl +status apache2`. + + +### SSL ### + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + + +## Basic Website ## + +Using as an example of a basic website, +create an Apache2 vhost configuration file at +`/etc/apache2/sites-available/archive.subgeniuskitty.com.conf`. + + + DocumentRoot "/srv/apache_vhosts/archive.subgeniuskitty.com" + ServerName archive.subgeniuskitty.com + ServerAdmin webmaster@subgeniuskitty.com + ErrorLog /var/log/apache2/error_log.archive.subgeniuskitty.com + CustomLog /var/log/apache2/access_log.archive.subgeniuskitty.com combined + + Options +FollowSymLinks + AllowOverride None + Require all granted + + + Options +FollowSymLinks +Indexes + AllowOverride None + Require all granted + + + +Make the directory `/srv/apache_vhosts/archive.subgeniuskitty.com`, move your +data into it, and ensure everything is owned by `www-data:www-data`. + +Enable the vhost with `a2ensite archive.subgeniuskitty.com` and `systemctl +reload apache2`. + + +## CMless Website ## + +Enable `mod_rewrite` and either `mod_cgi` or `mod_cgid` as appropriate with +these commands. + + a2enmod rewrite + a2enmod cgid + +Install `discount` to convert Markdown to HTML. + + apt-get install discount + +Create `/etc/apache2/sites-available/subgeniuskitty.com.conf`. + + + DocumentRoot "/srv/apache_vhosts/subgeniuskitty.com" + ServerName subgeniuskitty.com + ServerAlias www.subgeniuskitty.com + ServerAdmin webmaster@subgeniuskitty.com + ErrorLog /var/log/apache2/error_log.subgeniuskitty.com + CustomLog /var/log/apache2/access_log.subgeniuskitty.com combined + AddHandler cgi-script .py + + Options -ExecCGI -Indexes + AllowOverride None + Require all granted + + + Options ExecCGI + AllowOverride None + Require all granted + + RewriteEngine On + RewriteRule (.*) /srv/apache_vhosts/subgeniuskitty.com/site/data/$1 + RewriteCond %{REQUEST_FILENAME} !-f + RewriteRule .* /srv/apache_vhosts/subgeniuskitty.com/bin/cmless.py + + +Enable the site with `a2ensite subgeniuskitty.com`. + +Clone a copy of [CMless](git://git.subgeniuskitty.com/cmless) into +`/srv/apache_vhosts/subgeniuskitty.com` and ensure everything is owned by +`www-data:www-data`. + +Clone a copy of [the website +data](git://git.subgeniuskitty.com/website_subgeniuskitty.com) into +`/srv/apache_vhosts/subgeniuskitty.com/site`. Verify it is owned by +`ataylor:ataylor` (but still readable by all) so we can update the site +remotely with a simple script like this: + + #!/usr/local/bin/bash + # + # Usage: No cmdline arguments + # Update content of www.subgeniuskitty.com to the latest version. + + ssh ataylor@git.subgeniuskitty.com "cd /srv/apache_vhosts/subgeniuskitty.com/site && git pull" + +Reload Apache's configuration with `systemctl reload apache2` and test access +to the website. + +Repeat this process for . + + +# Git Server # + +The git server provides read-write access to a private collection of bare +repositories located at `/srv/git` via SSH. It also provides read-only access +to a public collection of normal repositories located at `/srv/gitweb_cache` +via through gitweb and via +[git://git.subgeniuskitty.com/repo_name](git://git.subgeniuskitty.com/repo_name) +through `git-daemon`. + + +## Read/Write: SSH ## + +Install git with `apt-get install git`. + +On my workstation, generate an SSH key with `ssh-keygen -t rsa`. + +On the server, as user `ataylor`: + + mkdir ~/.ssh + chmod 700 ~/.ssh + touch ~/.ssh/authorized_keys + chmod 600 ~/.ssh/authorized_keys + +Then `cat` the public SSH key from the workstation to the server, appending it +onto `~/.ssh/authorized_keys`. + +Verify ability to login using new certificate. + +Create a directory `/srv/git` owned by `ataylor:ataylor`. This will hold bare +git repositories and act as the central private store for SGK git repos. + +From the workstation, we can create a new bare repository on the server. For +example, packed up in a simple script: + + #!/usr/local/bin/bash + # + # Usage: sgkgit-new-repo project_name + # Setup a repository on the SGK git server. + + if [ "$#" -ne 1 ]; then + echo "Must specify repo name as only parameter." + exit 2 + fi + + ssh ataylor@git.subgeniuskitty.com "git init --bare /srv/git/$@" + +We then set the remote of an existing repository to the new bare repository. +Again as a script: + + #!/usr/local/bin/bash + # + # Usage: sgkgit-set-origin project_name + # Sets remote to the correct path for 'project_name' on the SGK git server. + + if [ "$#" -ne 1 ]; then + echo "Must specify repo name as only parameter." + exit 2 + fi + + git remote remove origin + git remote add origin ataylor@git.subgeniuskitty.com:/srv/git/$@ + echo "Remember to make the first push with \"git push --set-upstream origin master\"." + +After the first push to the bare repository on the server, simply `git push` +and `git pull` as normal. + +You can also list the repositories currently on the server with simple SSH commands. + + #!/usr/local/bin/bash + # + # Usage: No cmdline arguments. + # List all remote repos available on SGK git server. + + ssh ataylor@git.subgeniuskitty.com "ls -lt /srv/git/" + +Clone one of these repositories, with remote correspondingly pre-set. + + #!/usr/local/bin/bash + # + # Usage: sgkgit-checkout project_name + # Clone a local copy of repo "project_name" from SGK git server. + + if [ "$#" -ne 1 ]; then + echo "Must specify repo name as only parameter." + exit 2 + fi + + git clone ataylor@git.subgeniuskitty.com:/srv/git/$@ + + +## Read-Only: Gitweb ## + +Enable `mod_rewrite` and either `mod_cgi` or `mod_cgid` as appropriate with +these commands. + + a2enmod rewrite + a2enmod cgid + +Install `discount` to convert Markdown to HTML, `highlight` for syntax +highlighting, and `gitweb` to pull in any dependencies. + + apt-get install discount highlight gitweb + +Create `/etc/apache2/sites-available/git.subgeniuskitty.com.conf`. + + + ServerName git.subgeniuskitty.com + ServerAdmin webmaster@subgeniuskitty.com + + DocumentRoot "/srv/apache_vhosts/git.subgeniuskitty.com" + + ErrorLog /var/log/apache2/error_log.git.subgeniuskitty.com + CustomLog /var/log/apache2/access_log.git.subgeniuskitty.com combined + + + Options +FollowSymLinks +ExecCGI + AllowOverride None + Require all granted + AddHandler cgi-script .cgi + DirectoryIndex gitweb.cgi + RewriteEngine On + RewriteCond %{REQUEST_FILENAME} !-f + RewriteCond %{REQUEST_FILENAME} !-d + RewriteRule ^.* /gitweb.cgi/$0 [L,PT] + + + +Enable the site with `a2ensite git.subgeniuskitty.com`. + +Clone a copy of [the SGK gitweb fork](git://git.subgeniuskitty.com/gitweb-sgk) +into `/srv/apache_vhosts/git.subgeniuskitty.com` and ensure everything is owned +by `ataylor:ataylor` (but world-readable!) so we can update via SSH with a script like this. + + #!/usr/local/bin/bash + # + # Usage: No cmdline arguments + # Update git.subgeniuskitty.com to the latest version of forked gitweb repo. + + ssh ataylor@git.subgeniuskitty.com "cd /srv/apache_vhosts/git.subgeniuskitty.com && git pull" + +This fork makes a few changes to gitweb, displaying READMEs by default, etc. +Read the gitweb project's `README.md` for more details. + +Create a gitweb config file at `/etc/gitweb.conf`. Note that we are adding a +`clone url` to the toolbar with the address of our `git-daemon` server. +Remember to set that up. + + $site_name = "git.subgeniuskitty.com"; + @git_base_url_list = ("git://git.subgeniuskitty.com"); + $projectroot = "/srv/gitweb_cache"; + $git_temp = "/tmp"; + + @stylesheets = ("static/gitweb.css"); + $javascript = "static/gitweb.js"; + $logo = "static/sgk-logo.png"; + $favicon = "static/git-favicon.png"; + + # git-diff-tree(1) options to use for generated patches + @diff_opts = (); + + # Enable PATH_INFO so the server can produce URLs of the + # form: http://git.hokietux.net/project.git/xxx/xxx + # This allows for pretty URLs *within* the Git repository, + # also needs the Apache rewrite rules for full effect. + $feature{'pathinfo'}{'default'} = [1]; + + # HTML text to include as home page header. + $home_text = "indextext.html"; + + # Add a toolbar option with the 'git clone url'. + $feature{'actions'}{'default'} = [('clone url', 'git://git.subgeniuskitty.com/%n', 'summary')]; + + # Category name is read from .git/category, in the same manner as .git/description. + $projects_list_group_categories = 1; + $project_list_default_category = "misc"; + + # Needed for displaying README files. + $prevent_xss = 0; + + # Enable syntax highlighting. + $feature{'highlight'}{'default'} = [1]; + + ################################################################################ + + # Enable blame, pickaxe search, snapshop, search, and grep + # support, but still allow individual projects to turn them off. + # These are features that users can use to interact with your Git trees. They + # consume some CPU whenever a user uses them, so you can turn them off if you + # need to. Note that the 'override' option means that you can override the + # setting on a per-repository basis. + $feature{'blame'}{'default'} = [1]; + $feature{'blame'}{'override'} = [1]; + + $feature{'pickaxe'}{'default'} = [1]; + $feature{'pickaxe'}{'override'} = [1]; + + $feature{'snapshot'}{'default'} = [1]; + $feature{'snapshot'}{'override'} = [1]; + + $feature{'search'}{'default'} = [1]; + + $feature{'grep'}{'default'} = [1]; + $feature{'grep'}{'override'} = [1]; + +Create the directory `/srv/gitweb_cache`. It should be readable by the web +server (`www-data`), but owned by `ataylor:ataylor` so that simple scripts like +the following can make repositories public/private. + + #!/usr/local/bin/bash + # + # Usage: sgkgit-make-public project_name + # Make a repo accessible through gitweb and git-daemon. + + if [ "$#" -ne 1 ]; then + echo "Must specify repo name as only parameter." + exit 2 + fi + + read -p "Enter a category name: " category + read -p "Enter a description: " description + + echo "You entered:" + printf "\tCategory: $category \n" + printf "\tDescription: $description \n" + read -p "Confirm (y/n)?: " confirm + + if [ "$confirm" == "y" ]; then + ssh ataylor@git.subgeniuskitty.com "cd /srv/gitweb_cache && rm -rf $@ && git clone /srv/git/$@" + ssh ataylor@git.subgeniuskitty.com "cd /srv/git && \ + printf '#!/usr/bin/bash\ncd /srv/gitweb_cache/$@\ngit --git-dir=.git pull\n' > \ + /srv/git/$@/hooks/post-update && chmod +x /srv/git/$@/hooks/post-update" + ssh ataylor@git.subgeniuskitty.com "echo \"$category\" > /srv/gitweb_cache/$@/.git/category" + ssh ataylor@git.subgeniuskitty.com "echo \"$description\" > /srv/gitweb_cache/$@/.git/description" + fi + +In addition to cloning the git repo into `/srv/gitweb_cache` from the private, +SSH-only collection of repositories in `/srv/git`, this also creates a +`post-update` hook in the bare repo in `/srv/git` to ensure that gitweb's cache +is updated every time a push is made via SSH to the private repo. + +This script also sets the `.git/description` and `.git/category` files needed +by gitweb for displays like the project summary page. These are not under +version control and may be directly edited to change what is displayed in +gitweb. + +We can remove the public cache, making the repo private-only as shown in this +script. + + #!/usr/local/bin/bash + # + # Usage: sgkgit-make-private project_name + # Make a repo inaccessible through gitweb and git-daemon. + + if [ "$#" -ne 1 ]; then + echo "Must specify repo name as only parameter." + exit 2 + fi + + ssh ataylor@git.subgeniuskitty.com "rm -rf /srv/gitweb_cache/$@ && rm /srv/git/$@/hooks/post-update" + +Any repos made private/public via this method are immediately reflected in +`gitweb` and `git-daemon`. + + +## Read-Only: Git Daemon ## + +Since `git` was already installed, simply create a new service profile in +`/etc/systemd/system/git-daemon.service`. + + [Unit] + Description=Start Git Daemon + + [Service] + ExecStart=/usr/bin/git daemon --export-all --reuseaddr --base-path=/srv/gitweb_cache/ /srv/gitweb_cache/ + + Restart=always + RestartSec=500ms + + StandardOutput=syslog + StandardError=syslog + SyslogIdentifier=git-daemon + + User=ataylor + Group=ataylor + + [Install] + WantedBy=multi-user.target + +Then start the service with `systemctl daemon-reload` and `systemctl start +git-daemon`. Verify functionality before enabling daemon-autostart with +`systemctl enable git-daemon`. + +The `--export-all` flag tells `git-daemon` to export all repositories located +at the specified path, regardless of the presence (or lack) of the file +`.git/git-daemon-export-ok` in the individual repository. We do this because +only public repos are checked out to this folder. All private repos are kept +entirely elsewhere. + +The directory `/srv/gitweb_cache` should have already been created when +installing gitweb. If not, go read those instructions. + +A repository located at `/srv/gitweb_cache/repo_name` may be cloned through +`git-daemon` at +[git://git.subgeniuskitty.com/repo_name](git://git.subgeniuskitty.com/repo_name) +with `git clone`. + + +# Mail Server # + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + +TODO + diff --git a/data/notes/mail_web_git_server.metadata b/data/notes/mail_web_git_server.metadata new file mode 100644 index 0000000..39930f1 --- /dev/null +++ b/data/notes/mail_web_git_server.metadata @@ -0,0 +1,6 @@ +[DEFAULT] +page_title = Setting up a combined mail/web/git server. +meta_keywords = +meta_description = +menu_text = Mail/Web/Git Server +menu_priority = 8000 -- 2.20.1