Using Arch for source control management with OpenACS

Table of contents

Email me if you have corrections or additions to this document.

Why use Arch

I've used CVS for a couple of years, and even though you can't live without version control, and it's decent, it is not well-suited to web application development. You run into problems with branching, tracking local customizations and keeping track of a changing code stream.

Arch handles these much better than CVS:

Compared to the competition (Subversion for example): Subversion is also a very worthy tool -- I use Subversion every day. But I chose to go with Arch for an earlier project. Arch is pretty interesting, and if I didn't occasionally run into annoying error messages I would recommend it more. I also find that the "user interface" is too focused on people who understand the mechanics of how Arch works, rather than on abstracting those details from developers.

Learning about Arch

The best place to go is the Arch Wiki. It has instructions for installing and learning Arch. You should at least look over the tutorial there to make sure you understand how to get help.

How we'll set things up

First of all, we're not going to tackle the more complicated issue of setting up a CVS to Arch gateway. This is very possible, but not something we're going to get into at this point.

We are setting up a very rigorous setup, which will have a developmental server, a staging server, and the live server. This will allow you to have localized changes to your OpenACS installation, use the remote repository without doing a bunch of CVS chicanery, and in general allow you to test changes before going live with them.

We will set up an OpenACS Arch repository that will mirror the stable releases of OpenACS core (currently 5.1.3). The website we are building is safe4all.

User Purpose Archive Category Branch Version Ports
safe4all-dev Mirror OpenACS core releases jade@safe4all.org--2005 openacs stable 5.1.0 None
safe4all-dev Import packages, html pages jade@safe4all.org--2005 openacs dev 5.1.0 8000/8443
safe4all-staging Local customization jade@safe4all.org--2005 openacs staging 5.1.0 8001/8444
safe4all Live site jade@safe4all.org--2005 openacs staging 5.1.0 80/443

Initial setup

# groupadd web
# useradd -m -g web safe4all-dev
# useradd -m -g web safe4all-staging
# useradd -m -g web safe4all
# apt-get install tla tla-doc
See the OpenACS wiki guide to installing OpenACS on Debian. Set up the Arch IDs for all the users 
# su - safe4all-dev
$ tla my-id "Jade Rubick <safe4all-dev@safe4all.org>"
$ exit
# su - safe4all-staging
$ tla my-id "Jade Rubick <safe4all-staging@safe4all.org>"
$ exit
# su - safe4all
$ tla my-id "Jade Rubick <safe4all@safe4all.org>"
$ exit
You should also edit your .bash_profile or equivalent to make your umask 0002 instead of 0022. This helps you with permissions issues later. Do this for safe4all-dev and safe4all-staging.

Setup OpenACS core release mirror (stable branch)

What we'll do is track the latest stable release of OpenACS from CVS. Currently, that is 5.1.5. What we will do is start with 5.1.0 (start later if you want to save yourself some time), and upgrade each version to get to the current one. This tests our upgrade process and makes sure we're on the right path.

The basic way we're setting things up is to set up to track a project that doesn't use Arch

If you haven't already: 

# su - safe4all-dev
$ mkdir ~/{archives}
Now set up your openacs-stable archive. 
$ tla make-archive jade@safe4all.org--2005 ~/{archives}/2005
$ tla my-default-archive jade@safe4all.org--2005
$ tla archive-setup openacs--stable--5.1.0
* creating category jade@safe4all.org--2005/openacs
* creating branch jade@safe4all.org--2005/openacs--stable
* creating version jade@safe4all.org--2005/openacs--stable--5.1.0
$ mkdir ~/wd
$ cd ~/wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot co -r openacs-5-1-0-final acs-core  
(see http://openacs.org/4/checkout)
$ mv openacs-4 openacs-5-1-0
$ cd openacs-5-1-0
$ tla init-tree openacs--stable--5.1.0
We have to get all the source into Arch. You should look up the inventory and tree-lint commands if you don't understand what they do. They essentially help us make sure that we're importing all the files into Arch the way we want to. 
$tla inventory --names --both --source |xargs tla add
$tla tree-lint
We have to get to a point where tla tree-lint doesn't report any issues. Here are some hints:

We now need to edit the =tagging-method file, to ensure that log files aren't stored in Arch, that the daemontools' supervise directory isn't stored in Arch, etc.. 

emacs {arch}/=tagging-method
Change this line:
precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn)$
to
precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn|supervise|content-repository-content-files)$
You also will need to add a .arch-inventory file to prevent Arch from storing database backups, server certificates, etc... See: http://wiki.gnuarch.org/moin.cgi/Tla_20Files 
echo "precious ^log|apm-workspace|rss|database-backup$" > .arch-inventory
echo "precious ^certs$" > etc/.arch-inventory
echo "precious ^down$" > etc/daemontools/.arch-inventory

Add in code that hasn't been added in yet (check without xargs first) 

$ tla tree-lint -t |xargs tla add
Do this until tree-lint doesn't complain.

Now we commit the new stable branch: 

$ tla import
* imported jade@safe4all.org--2005/openacs--stable--5.1.0
$ tla commit --summary "Imported OpenACS 5.1.0"
* update pristine tree (jade@safe4all.org--2005/openacs--stable--5.1.0--base-0 => openacs--stable--5.1.0--patch-1)
* committed jade@safe4all.org--2005/openacs--stable--5.1.0--patch-1

Set up a dev branch

At this point, we need to set up a dev branch, where we'll use to keep track of new packages we install, and HTML code we're given. At my company, the graphics department works on the HTML code, but there often are modifications I have to make after they have given it to me. For example, I may change a link after installing a package. The dev branch is where I upload their code, and the code for new packages I install via the remote repository.

We want to create a website called safe4all. You'll base this dev instance on the openacs stable version, by creating a branch:

This follows the recipe here: Maintain your own version of another project. See also Elementary branches. 

$ cd ~wd
$ tla archive-setup openacs--dev--5.1.0
* creating branch jade@safe4all.org--2005/openacs--dev
* creating version jade@safe4all.org--2005/openacs--dev--5.1.0
$ tla tag jade@safe4all.org--2005/openacs--stable--5.1.0 openacs--dev--5.1.0
Now we have a dev branch of the openacs-stable archive.

Now you can set up a dev server. This step is not strictly necessary. When you edit your config.tcl file, I recommend setting up the email redirecting, so email isn't sent from your staging server. See "Configure an AOLserver Service for OpenACS" on the OpenACS documentation. 

$ cd
$ tla get openacs--dev--5.1.0 safe4all-dev
* from import revision: jade@safe4all.org--2005/openacs--stable--5.1.0--base-0
* patching for revision: jade@safe4all.org--2005/openacs--stable--5.1.0--patch-1
* patching for revision: jade@safe4all.org--2005/openacs--dev--5.1.0--base-0
* making pristine copy
* tree version set jade@safe4all.org--2005/openacs--dev--5.1.0
$ ln -s /home/safe4all-dev/safe4all-dev /var/lib/aolserver/safe4all-dev
$ mkdir ~/safe4all-dev/log
$ emacs ~/safe4all-dev/etc/config.tcl
$ emacs ~/safe4all-dev/etc/daemontools/run
$ touch ~/safe4all-dev/etc/daemontools/down
Now set up the database. If you are starting from scratch:
Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.

If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:

# su - postgres
$ createuser -a -d safe4all-dev
$ exit
# su - safe4all-dev
$ createdb -E UNICODE safe4all-dev
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all-dev/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all-dev
$ locate vacuum |grep vacuumdb
$ export EDITOR=emacs;crontab -e
Put this in the crontab, changing the directory to pgsql depending on what which said for vacuum. Note the first line is commented out -- it is no longer necessary as of Postgres 7.4. 
# 0 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-dev > /dev/null 2>&1
0 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-dev > /dev/null 2>&1

Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand. 

$ su - 
# cd /home/safe4all-dev/safe4all-dev/etc/daemontools
# ./run
If you get this error message:
/usr/local/aolserver/bin/nsd: error while loading shared libraries: libnsd.so: cannot open shared object file: No such file or directory
See: error while loading shared libraries: libnsd.so

Check that the server actually starts up, look at the ~safe4all-dev/log/error.log files to troubleshoot. Post on the OpenACS Forums if you have problems. Once it works, then link it in.

$ su -
# ln -s /var/lib/aolserver/safe4all-dev/etc/daemontools /service/safe4all-dev
# svgroup web /service/safe4all-dev
# exit
$ svc -u /service/safe4all-dev

If the svgroup command doesn't work, you need to install it (see the OpenACS installation guide (here).

Once you get it right, commit the changes you've made. You can go through the web interface now and set up OpenACS if you didn't import anything.

Set up a staging branch

At this point, we need to set up a staging branch, where we'll use to keep track of our local customizations to OpenACS and the HTML code we have. For example, I may change the appearance of the OpenACS pages or packages, or change the wording of something in the OpenACS package.

We want to create a website called safe4all. We base this website on the safe4all-dev branch, but create another branch:

This follows the recipe here: Maintain your own version of another project. See also Elementary branches. 

# su - safe4all-staging
$ cd 
$ tla register-archive /home/safe4all-dev/\{archives\}/2005/
$ tla my-default-archive jade@safe4all.org--2005
$ tla archive-setup openacs--staging--5.1.0
* creating branch jade@safe4all.org--2005/openacs--staging
* creating version jade@safe4all.org--2005/openacs--staging--5.1.0
$ tla tag jade@safe4all.org--2005/openacs--dev--5.1.0 openacs--staging--5.1.0
Now we have a staging branch of the dev branch.

Now set up the staging server: 

$ cd
$ tla get openacs--staging--5.1.0 safe4all-staging
* from pristine cache: jade@safe4all.org--2005/openacs--dev--5.1.0--base-0
* patching for revision: jade@safe4all.org--2005/openacs--staging--5.1.0--base-0
* making pristine copy
* tree version set jade@safe4all.org--2005/openacs--staging--5.1.0
$ ln -s /home/safe4all-staging/safe4all-staging /var/lib/aolserver/safe4all-staging
$ emacs ~/safe4all-staging/etc/config.tcl
$ emacs ~/safe4all-staging/etc/daemontools/run
$ touch ~/safe4all-staging/etc/daemontools/down
When you edit your config.tcl file, I recommend setting up the email redirecting, so email isn't sent from your staging server.

Now set up the database. If you are starting from scratch:

Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.

If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:

# su - postgres
$ createuser -a -d safe4all-staging
$ exit
# su - safe4all-staging
$ createdb -E UNICODE safe4all-staging
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all-staging/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all-staging
$ export EDITOR=emacs;crontab -e
Add to the crontab entries for safe4all-staging: 
#15 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-staging > /dev/null 2>&1
15 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-staging > /dev/null 2>&1

Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand. 

$ su - 
# cd /home/safe4all-staging/safe4all-staging/etc/daemontools
# ./run
Check that the server actually starts up, look at the error.log files to troubleshoot. Once it works, then: 
$ su -
# ln -s /var/lib/aolserver/safe4all-staging/etc/daemontools /service/safe4all-staging
# svgroup web /service/safe4all-staging
# exit
$ svc -u /service/safe4all-staging

Set up the live server server

The live server then is based on the staging branch: openacs--staging--5.1.0

$ cd 
$ tla register-archive /home/safe4all-dev/\{archives\}/2005/
$ tla my-default-archive jade@safe4all.org--2005
Now set up the live server: 
$ cd
$ tla get openacs--staging--5.1.0 safe4all
$ ln -s /home/safe4all/safe4all /var/lib/aolserver/safe4all
$ emacs ~/safe4all/etc/config.tcl
$ emacs ~/safe4all/etc/daemontools/run
Now set up the database. If you are starting from scratch:
Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.

If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:

# su - postgres
$ createuser -a -d safe4all
$ exit
# su - safe4all
$ createdb -E UNICODE safe4all
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all
Add to the crontab entries for safe4all: 
$ export EDITOR=emacs;crontab -e

#30 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all > /dev/null 2>&1
30 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all > /dev/null 2>&1

Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand. 

$ su - 
# cd /home/safe4all/safe4all/etc/daemontools
# ./run
Check that the server actually starts up, look at the error.log files to troubleshoot. Once it works, then: 
$ su -
# ln -s /var/lib/aolserver/safe4all/etc/daemontools /service/safe4all
# svgroup web /service/safe4all
# exit
$ svc -u /service/safe4all

When a new version of OpenACS comes out

When updates are made to OpenACS core (such as 5.1.3 coming out), you update it in the oacs archive.

Tracking a project that doesn't use Arch, under Updating the vendor source.

Since we started with OpenACS 5.1.0, I'll now bring in 5.1.1 into the stable branch: 

# su - safe4all-dev
$ cd wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-1-final acs-core
$ mv openacs-4 openacs-5-1-1
The old version of OpenACS is at ~/wd/openacs-5-1-0, and the new version is at ~/wd/openacs-5-1-1. 
$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' openacs-5-1-1/* openacs-5-1-0
$ cd openacs-5-1-0
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
   perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
   perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_' | xargs rm -rf
$ tla tree-lint -m     (if you get anything do the next line)
$ tla tree-lint -m | xargs rm 
$ tla tree-lint -t | xargs tla add-id
$ tla tree-lint
Make sure tree-lint doesn't complain before continuing. 
$ tla commit --summary "Imported OpenACS 5.1.1"
We made the ~/wd/openacs-5-1-0 directory have all the changes from openacs-5-1-1, so just to avoid confusion, let's clean up the working directory: 
$ cd ~/wd
$ mv openacs-5-1-1 /tmp
$ mv openacs-5-1-0 openacs-5-1-1

Updating dev with stable changes

First make sure you don't have any local changes to commit: 
$ cd safe4all-dev
$ tla changes
If you do have changes to commit, then commit them first, using the directions below. 
# su - safe4all-dev
$ cd safe4all-dev
$ tla missing --summary openacs--stable--5.1.0
$ tla replay openacs--stable--5.1.0
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add
$ tla changes 
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB)    (edit the log entry)
$ tla commit

Installing and upgrading OpenACS packages

First make sure you don't have any local changes to commit: 
$ cd safe4all-dev
$ tla changes
If you do have changes to commit, then commit them first, using the directions below.

I usually check which packages are available using the acs-admin/install page. I don't actually install from there. Here is an example using calendar: 

# su - safe4all-dev
$ cd
$ cvs -d:pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-compat calendar 
$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' calendar/* dev-server/packages/calendar
$ find dev-server \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'
If you see any results from the find command, you can delete those directories as appropriate. See Tracking a project that doesn't use Arch. 
$ cd dev-server
$ tla tree-lint 
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add-id   (may have to do multiple times)
$ tla changes 
$ tla make-log
$ emacs +(TAB)    (edit the log entry)
$ tla commit
$ cd
$ mv calendar /tmp

Updating staging with dev changes

First make sure you don't have any local changes to commit: 
$ cd safe4all-staging
$ tla changes
If you do have changes to commit, then commit them first, using the directions below. 
# su - safe4all-staging
$ cd safe4all-staging
$ tla missing --summary openacs--dev--5.1.0
$ tla replay openacs--dev--5.1.0
If you get conflicts during the replay, see how to deal with .rej files, which are the files that you get when there are conflicts.

Now commit the changes to the staging branch

$ tla changes 
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB)    (edit the log entry)
$ tla commit

Updating the live server

This part is really easy. 
$ su - safe4all
$ cd safe4all
$ tla missing -s
$ tla replay
You can make local changes to files, and updates will not overwrite them. So you can edit the etc/config.tcl file for example. If there are conflicts, you will have to resolve them, however.

Making changes on the dev branch

As you recall, the dev branch is for adding new OpenACS packages. You may also want to make local changes such as the change to the etc/config.tcl and etc/daemontools/run file. When you have added code or made any changes, you simply: 
$ tla changes --diffs
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add      # (may need to do this multiple times)
$ tla commit --summary "Description of my changes"

Making customizations

When you want to make a customization, you do it on the safe4all-staging server. So first, we bring up the server (leave it off most times): 
$ svc -u /service/safe4all-staging
Make your changes.

Commit your changes 

$ tla changes 
$ tla tree-lint 
$ tla tree-lint -t
$ tla tree-lint -t |xargs tla add
$ tla commit --summary "Describe my customization here"
You'll have to resolve any conflicts if you find any. See how to deal with .rej files.

Now that we're done, we'll take down the staging server, then update the live server 

$ svc -d /service/safe4all-staging

Resetting the database

If you ever want to load more current data into your staging or dev branches. The Perl is to change the username in the backup dump. 
$ su - safe4all-staging
$ svc -d /service/safe4all-staging
$ dropdb safe4all-staging

$ createdb -E UNICODE safe4all-staging
$ cp ~safe4all/safe4all/database-backup/backup.dmp ~/safe4all-staging/database-backup/
$ perl -pi -e "s/^\\connect safe4all$/\\connect safe4all-staging/" backup.dmp
$ psql < backup.dmp safe4all-staging
$ svc -u /service/safe4all-staging

Making upstream contributions from your changes

This section is still being worked on.

If you have a changeset that you'd like to commit to OpenACS, here's how I currently I am doing it: 

$ su - safe4all-dev
$ ls -1
You'll see this: 
dev-server
oacs-5-1
oacs-HEAD
wd
{archives}
The directory: oacs-5-1 is a cvs checkout of OpenACS, from the oacs-5-1 branch.

Let's say the change I want to commit is patch-8. 

$ tla get openacs--dev--5.1.0--patch-7 patch-7
$ tla get openacs--dev--5.1.0--patch-8 patch-8
$ diff -ur patch-7 patch-8 -x '{arch}' > patch-file
Then you can use that patch-file to patch oacs-5-1, using the patch command 
$ man patch

Other resources

Changes to this document

If you want to keep up to date with the changes in this document, you shouldn't have to read every bit of the page and scan it for changes. Just look in this section, and I'll detail what you have to do to get up to the latest changes. For example, if you last used this document on January 12, 2005, then you'll need to make the January 13, 2005 changes. Then you'll be exactly on the same page as everything else in this document.