User:Dan Nessett/Technical/How to set up a CZ clone on Ubuntu

This page provides instructions describing how to set up a Citizendium clone. Creating a CZ clone is useful for a number of purposes, including:

• Storing a local copy of Citizendium on a lap-top or other device so articles are accessible when there is no available network connection.
• Creating a local copy of Citizendium for the purpose of fixing bugs in the modified version of Mediawiki that forms its software base or developing extensions to that code.

Installing the necessary software infrastructure

CZ is implemented by a modified version of the Mediawiki software. This software relies on several software components that underly its operation, specifically:

• PHP - an object-oriented interpreted language. Mediawiki is written in PHP and its software modules execute when accessed through a web server.
• Apache2 - an open source industrial grade web server.
• Postgres - an open source industrial grade database system.

Note that many Mediawiki implementations use the MySQL database system, rather than Postgres. While it might be possible to install Citizendium's wiki content using a MySQL database, doing so would represent experimental development. This how-to presumes the installer uses Postgres.

In order to install the Citizendium wiki software, the system running it must have PHP, Apache and Postgres preinstalled. So, the first step in creating a CZ clone is to install these software subsystems. There are a number of different platforms that support these and potentially any of them could be used to support a CZ clone, However, the most popular platform is Linux and the most popular of these is Ubuntu Linux. Consequently, these instructions assume the software is installed on Ubuntu Linux (they should also work on any other Debian Linux variant). However, it is possible to install CZ's software on other Linux platforms and even on other operating systems (e.g., Mac OS X, Windows). Those interested in using using a different OS platform will have to do the research necessary to accomplish that. These instructions focus on using Ubuntu.

Installing a LAPP stack

It is common to refer to the Apache, Postgres and PHP software infrastructure as a LAPP stack (i.e., a Linux Apache Postgres PHP stack). There are a number of ways to install this software. One simple approach is to install the Turnkey LAPP appliance available from the open source Turnkey Linux Appliance project. This appliance is available on a LiveCD distribution and installing it installs Ubuntu as well as the LAPP stack. There are many advantages to using this appliance, not the least of which is it is pre-configured to use the LAPP stack. However, this section assumes the installer either already has a Ubuntu installation to use or has decided to install Ubuntu for purposes in addition to running the CZ software. Those installers who use the Turnkey LAPP distribution may skip ahead to the section Installing the CZ wiki software.

Updating the apt-get database

To ensure the LAPP stack installation uses the most up-to-date software, the installer should first update the apt-get database using the following command at a terminal command-line prompt:

sudo apt-get update

Install Postgres

After updating the apt-get database, install postgres.

sudo apt-get install postgresql-8.3 php5-pgsal

Then install the cz postgres user:

sudo -u postgres createuser -D -A -P cz

This prompts for a password:

Enter password for new role:

Type in a password and when prompted to enter it again, do so.

After entering the password twice, postgres asks the following question:

 Shall the new role be allowed to create more new roles? (y/n)

Type "y" (without the quotes).

Next, ensure the postgres is secure by creating

Notes

• Had to modify createAndPromote.php to check password for validity before creating user. Otherwise, if the password is invalid a "ghost" user is created in mwuser table.
• Had to create a dummy user with user_id of 0, so XML dump import would work.
• Need to set Xdebug variables for both apache2/php and php-cli versions of php.ini
• Had to set xdebug.max_nesting_level=200 in /etc/php5/cli/php.ini so dump import wouldn't croak.
• Some useful information on MW XML dumps: http://www.mail-archive.com/wikitech-l@lists.wikimedia.org/msg01712.html, http://www.gossamer-threads.com/lists/wiki/wikitech/180598, http://meta.wikimedia.org/wiki/Data_dumps, http://meta.wikimedia.org/wiki/Xml2sql
• Used cloc to count lines of PHP code in CZ:

Directory	Files	Blank Lines	Comments	PHP code statements
CZ phase 3	1005	56590	69544	460125
CZ includes	321	14769	33313	97375
CZ extensions	142	3769	6742	27350
CZ includes+extensions	463	18583	40055	124725

• Using importDump.php in /maintenance I populated a version of CZ as a local development environment. The Statistics special page showed in excess of 129,000 pages. The import reported populating 116,400 pages (looking at the pages table, the exact number is 116,486). This checks out, since the daily dump of CZ does not include histories. There are approximately 12,700 live articles, each of which would have a history page. Noting, 116,500 + 12,700 = 129,200, it appears all content pages were loaded. However, it took in excess of 3 1/2 days (about 80 hours) to import the content. This suggests looking at more efficient import strategies (e.g., using mwdumper or converting to SQL with xml2sql and importing directly into the database).
• Had trouble getting skins to work. I needed to set $wgScriptPath to /mediawiki/CZ_1_13_2/phase3. Originally had it set to $IP. But, that expands to /usr/local/src/mediawiki/CZ_1_13_2/phase3, which is not accessible through the apache2 server. The correct value uses the /mediawiki apache2 alias.

I now need to run maintenance/runJobs.php. The statistics page shows 272,975 queued jobs, so running all queued jobs is going to take a while. Dan Nessett 22:39, 23 November 2009 (UTC)

• Had trouble getting texvc to work:

• The message "failed to parse cannot write to or create math temp directory" signals problems with permissions on the images directory in phase3.
• Need to ensure images directory has both a /math and /tmp subdirectory with read/write access and the images directory is accessible to the apache2 server (I simply chmod 777 both of them).
• Originally had $wgUploadPath to "$IP/images". This is incorrect. This variable must be set to a URL prefix that is accessible to the apache2 server. Set it to "$wgScriptPath/images" and TeX math worked.
• Ran into a strange problem where no matter how I changed the permissions on images/math and images/tmp, the message "failed to parse cannot write to or create math temp directory" appeared. Somehow this message stopped showing up. I don't know exactly why, but perhaps you need to clear the browser cache.
• I tried putting the directories/files in images into the www-data subgroup and owned by www-data and then changing permissions on everything below images to 775. However, subversion needs to get to locks in this directory tree (even when images has the svn:ignore property). So, while math rendering worked, when I committed changes to the repository, subversion failed on attempting to create a lock in images/.svn. So, I finally gave up and executed sudo chmod -R 777 images. This seems to fix all math rendering and subversion problems, but it is very insecure.

• Had trouble getting email to work. Since the installation is intended for local development, I chose to set up only local email. Therefore, every user must have an email address of <username>@localhost. When Ubuntu is installed, the exim4 MTA/MDA is installed by default. It is only necessary to set up an email client to receive emails. I used GNOME evoution (which is also installed by default). In order to set up evolution to receive local email, I had to set up the configuration as follows:

• Account name: Local Email Account
• Full Name: Dan Nessett
• Email Address dnessett@localhost
• Server Type: Local delivery
• Configuration (path): /var/mail/dnessett
• Server Type: Sendmail

• When we have a CZ repository set up, need to exclude some directories in phase3 from version control.

• In order to exclude all images in phase3/images from version control (other than those preloaded in icons), set property svn:ignore * on that directory.
• Svn copy LocalSettings.php into config (after potentially locally deleting any existing version of that file there). Then svn delete LocalSettings.php in phase3. Set properties on phase3 to include "svn:ignore LocalSettings.php". Commit these changes. Then locally (not using svn) copy LocalSettings.php from config to phase3. This effectively removes LocalSettings.php from version control. So, local developers can make modifications to it and commit other changes without saving LocalSettings.php to the repository. If it ever becomes necessary to change the repository version of LocalSettings.php, the developer should merge changes in phase3/LocalSettings.php into config/LocalSettings.php and then commit the changes.

• When ftp transferring a file created by svnadmin dump, make sure the transfer type is set to binary. Otherwise, when you attempt to import it, you will get an error like, "svnadmin: Dump stream contains a malformed header (with no ':') at:" Also when loading the dump, use svadmin load --ignore-uuid /path/to/repository < dumpfile. This will ensure the UUID specified in the dump file does not clobber the repository's existing UUID (this will happen if the repository being loaded has no revisions in it).