box.matto.nl

home/

Structured and automated system notes in a Vimwiki

Last edited
Automate your system documentation

System notes

When you maintain one or more systems it is a good habit to maintain system notes. System notes are notes on every important aspect of your work.

Added a new disk ? Put it in your system notes. Note the date and enter some information on the disk.

Came up with some fantastic solution? Put it in your system notes. As hard as it is to imagine now, some day you will have forgotten how you did this and why. Note the date and enter a description on the why and how.

As your habit of taking system notes develops your notes are stacking up. It is time to get some structure into them and to create a single place where all your notes are gathered. Being a system maintainer, your notes are ofcourse digital. So put them under revision control. This way you can allways look back to previous versions.

You do have a revision control system don't you? If not, choose a system, learn how to use it and set up your own repository now. Anything will do, from CVS to Subversion, Mercurial or Git. Just as long as you are comfortable with it, your repository is easy to move from one system to another and you are able to develop a good backup strategy for it.

A revision control system without a good backup strategy in place is almost as bad as having no revision control system at all.

Structure your system notes with Vimwiki

Your system notes probably will contain sensible information and therefor it is not a good idea to put these somewhere in the cloud.

Vimwiki provides a great environment for structured notes. It runs on your own system, so you are the boss over your data.

Notes in Vimwiki are plain text files which gives the opporunity to use any revision control system to keep control of your wiki.

In your Vimwiki you can create a page "Systems". In this page you create a wiki-link to a page for every system. So every individual system has its own inidividual wiki-page. In these pages you write your own notes to document your work on them.

Everything that makes it easier to maintain your notes will help you to actually keep the habit of maintaining system notes.

Put something like the following in your .vimrc:

map <F3> :r! date +"\%Y-\%m-\%d \%H:\%M:\%S"<cr>

This makes it easy to insert a timestamp in vim (just hit F3).

The following will insert a timestamp and turn it into a level-1 Vimwiki header:

map <F3> :r! date +"\%Y-\%m-\%d \%H:\%M:\%S"<ESC>0=j    

Keep an automated updated version of the important configuration files in a Vimwiki

Beside your hand written system notes your wiki can easely be extended with per system pages that contain the contents of the most important configuration files. We make it so that these pages are updated automaticly. This is not ment to be a real backup, but as a well kept documentation method.

Because Vimwiki-pages are plain text files, it is easy to create some scripts that do this for you.

Global plan

This is our global plan:

  • Initiate a vimwiki structure in bring that into a version control repository
  • Check out the latest version of the wiki from the repository
  • Create updated wiki-files on the several systems we maintain
  • Transfer the wiki-files to the checked out wiki with scp
  • Integrate the updated wiki-files into the wiki
  • Commit the changes to the repository

Build your own system notes system

Create the global structure.

Create a FreeBSD jail or a Linx container for your wiki. Let us call this jail/container systemnotes.mydomain.org.

In this, you create a user for your self. Setup a .vimrc file and a .vim directory in your $HOME. Download the vimwiki tarball or zip file and populate your ~/.vim with it.

Now create a Vimwiki. To do this, start vim and in command mode press

\ww

This will start your wiki. Create a page that looks something like this.

= System notes =

== Handwriten notes ==

Systems

== Configuration files ==

SystemConfigs

Move to the line with "Systems" and hit Enter. The word Systems is now a wiki-link. Hit Enter again and a new page is opened. On this new page you can put a header ( = Systems = ) and for each system create a line containing the hostname of that system. Close with :wq

Now if you want to enter a system note on any of these systemes, open your vimwiki (with \ww) and visit this page, go to the line with the hostname and hit Enter to go to the wikipage for that host and enter your notes there.

Now for the automated part. In startpage of your wiki go to the SystemConfigs line and hit enter on the word "SystemConfigs" to make it a wiki-link. That is enough for now.

Leave vim.

You now have a subdirectory in your home directory called vimwiki. Import that directory in your favorite revision control system. Delete the directory and check it out from your revision control system, in order to put it under control of your revision control system.

Create the user for automated updates

In your jail or container add a user that will do the automated work. Let us call this user "sysnotes" (you can ofcourse pick any name you like).

Become this user and check out the Vimwiki. Every time our automated update script runs it first start with an update and end with a commit. This way if something goes wrong, we can allways revert to a previous version.

Make sure that automated login over ssh is possible with the use of the authorized_keys file in ~/.ssh of this user.

Preparation of the systems

On every system create a user to collect the current version of your configuration files. Let us call this user "sysnotes". Become this user and generate a ssh key. Scp this key to the systemnotes.mydomain.org and add it to the authorized_keys file in ~/.ssh of sysnotes user. Check that you now have the option to automate the login.

Create a file with a list of configuration files

Create a file named "filelist" and put a list of configuration files you whish to document. Put every configuration file on a new line, so it looks something like this:

/boot/loader.conf
/etc/rc.conf    
/etc/iscsi.conf
/etc/lighttpd/lighttpd.conf

Script on the system

Here follows a script that could be used as an example. It uses the c-shell so it can run on FreeBSD-systems that have no bash installed.

The script creates a Vimwiki-page that starts with a heading based on the hostname of the system. Below this heading for every file in the "filelist" file a level-2 header is created followed by the contents of that file. After creation it runs scp to transfer the page to the central wiki.

#!/bin/csh

set name = `hostname`
set pagename = "ConfigFiles_$name.wiki"
echo "= ConfigFiles of $name =" > $pagename
echo "" >> $pagename

foreach i (`cat filelist`)
    echo "== $i ==" >> $pagename
    echo "{{{" >> $pagename
    cat $i >> $pagename
    echo "}}}" >> $pagename
    echo "" >> $pagename
end

echo 'Last updated: $Date$' >> $pagename

scp $pagename sysnotes@systemnotes.mydomain.org:vimwiki/

Here is a comparable script, using bash:

#!/bin/bash

name=`hostname`
pagename="ConfigFiles_$name.wiki"
echo "= ConfigFiles of $name =" > $pagename
echo "" >> $pagename

for i in `cat filelist`
do
    echo "== $i ==" >> $pagename
    echo "{{{" >> $pagename
    cat $i >> $pagename
    echo "}}}" >> $pagename
    echo "" >> $pagename
done

echo 'Last updated: $Date$' >> $pagename

scp $pagename sysnotes@systemnotes.mydomain.org:vimwiki/

Test the script

Run the script and check that everything works as expected. There shoud now be a new file in the ~/vimwiki directory of the sysnotes user on systemnotes.mydomain.org.

Determine when to run your script

Think about a scedule that fits your circumstances best. On most systems a weekly run is good enough and perhaps you have some systems where you want it more often. Revision control systems are smart enough that a commit will do nothing when there are no changes so there is no penalty to have your script run on regular bases.

Edit crontab to run the script

When your script runs flawless create a line into crontab so cron runs it regular.

Process the automated wiki pages

Now go back to systemnotes.mydomain.org and become the sysnotes user.

After the scripts on your systems have run, there are now a bunch of vimwiki-pages that have to be integrated into your wiki.

First create a crontab entry to do a update of your vimwiki from the revision control system repository. Let this update run a little bit earlier then your scripts on the systems.

Now we are going to create a script that builds the SystemConfigs wiki page.

#!/bin/sh

cd ~/vimwiki
echo "= System Configuration Files =" > SystemConfigs.wiki
ls ConfigFiles_*.wiki | sort | sed "s/.wiki$//" | \
 awk '{ print "[[" $0 "]]"}' >> SystemConfigs.wiki

echo "" >>  SystemConfigs.wiki
echo 'Last updated: $Date$' >> SystemConfigs.wiki

Run the script and check that it works OK. It should create a file called SystemConfigs.wiki that contains a header and a list of wiki-links to your ConfigFile-wiki pages.

Add all the files to the revision control system (like "cvs add *" or "git add *") followed by a commit.

Determine when to commit

Based on the cron settings of your systems you get regular updated wiki files that have to be merged into your vimwiki repository. Determine what is the best scedule. You have to add three entries in cron for the sysnotes user on systemnotes.mydomain.org:

  • An entry to update your vimwiki from the revision control system
  • An entry to run the script that builds the SystemConfigs wiki page
  • An entry to commit the changes to the revision control system

You could also do this commit at the bottom of the script that builds the SystemConfigs wiki page.

Go to your peronal user and do an update of the Vimwiki

Login in with the user you created for your self on systemnotes.mydomain.org. and do a update of the wikivim to get the latest changes from your revision control system.

Enter the Vimwiki (start vim and hit \ww). Go to the SystemConfigs line. If the word "SystemConfigs" is not allready a wiki link hit Enter to make it a wiki link. Hit enter again and you should be in your automaticly build SystemConfigs page full of wiki-links to per-system pages containing the latest version of their config files.

Export your wiki to html

Do you want your system notes be more accessible?

Vimwiki comes with a built in html export function. Use this to export your system notes to html.

I have not yet discoverd a method to execute vimscripts from cron. Until then the export to html has to be started by hand.

On the "Last updated" line

I am using CVS for more then a decade and my examples are based on that experience.

In the examples above there is a line that contains "Last updated: $Date$". In CVS this results in a line containing a timestamp of the last commit to the cvs repository. If this line is not altered in your wiki, check how to get a last updated timestamp from your revision control system.