DocumentRoot and SCRIPT_NAME

When I was troubleshooting webwork + shibboleth integration, everything works fine except the wrong target URL for Shibboleth lazy session initialization generated by webwork shibboleth module. Webwork Shibboleth module uses perl CGI to get the URL of for redirecting target. The URL was always https://server/webwork2 instead of https://server/webwork2/COURSENAME, when I click on a course to login. I did some digging and found out that the SCRIPT_NAME was set to /webwork2 instead of /webwork2/COURSENAME.

The root of this problem is related to DocumentRoot was set incorrectly in apache config file. The DocumentRoot was set to /www_data/webwork2 where it should be /www_data. Another way to solve the problem is add Alias /webwork2 /www_data/webwork2 to map the path to correct local directory.

Failed to submit ‘replace facts’ to PuppetDB: certificate verify failed

The error message:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit ‘replace facts’ command for ipeerdbprod1.ctlt.ubc.ca to PuppetDB at puppet.ctlt.ubc.ca:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [unable to get local issuer certificate for /CN=puppet.ctlt.ubc.ca]

Cause:
Accidentally executed – puppet cert –clean –all

Fix:
mv /etc/puppetdb/ssl /etc/puppetdb/ssl.bak
/usr/sbin/puppetdb-ssl-setup -f
/sbin/service puppetdb restart

Check /var/log/puppetdb/puppetdb.log for any error.

Loading multibyte CSV files into Oracle

Here is the problem:

  1. A bunch of CSV files are generated under windows
  2. some of the content for those files were copied/pasted from  M$ Word
  3. Word uses “smart quotes” and some other special characters that are not in ISO 8859-1 standard
  4. The Oracle schema has max character limits defined on the content columns in above (item 2) and the content may reach to the maximum.

So when loading the CSV files into database using sqlldr, got the following error:
Record 108: Rejected - Error on table QUESTIONDIM, column QUESTIONABBREV.
ORA-12899: value too large for column QUESTIONABBREV (actual: 49, maximum: 50)

49 is larger than 50???

When you open the file with a plain text editor such as VIM and place the cursor on those special characters, you will see the column count at the bottom shows multiple values (49-51) instead of single value as on normal character. It shows this character are multibyte character.

By default, the character columns in Oracle database tables was created as byte-characters, which means varchar2(20) can only store 20 byte data and it is equals two 20 ISO 8859-1 characters. Because each character only takes one byte. However, when the multibyte character is in the content, the storage required will be larger than 20 bytes. Because each multibyte character may takes 3 or 4 bytes and this is why VIM showing multiple values (49-51) instead of one value and Oracle thinks 49 is larger than 50!

To solve the problem:

Oracle database is set up to use UTF-8 character set. Check with the following SQL:
SELECT * FROM NLS_DATABASE_PARAMETERS

Creating tables using Character Semantics, e.g.:
SQL> CREATE TABLE tab2 (
2 id NUMBER(10),
3 description VARCHAR2(20 CHAR)
4 );

Or change the default Character Semantics using:
ALTER SYSTEM SET NLS_LENGTH_SEMANTICS=CHAR; or
ALTER SESSION SET NLS_LENGTH_SEMANTICS=CHAR;

Then all the fiels using varchar or character are created by char character semantics instead of byte.

The CSV files generated by windows are WINDOWS-1252 encoded. To allow sqlldr correctly translate the text into UTF8, which used by Oracle database, we need to set up correct character set (WE8MSWIN1252) for control file used by sqlldr.

OPTIONS (SKIP=1, DIRECT=true)
UNRECOVERABLE LOAD DATA
CHARACTERSET WE8MSWIN1252
INFILE 'Infile.CSV'
TRUNCATE INTO TABLE MY_TABLE
fields terminated by "," optionally enclosed by '"'
(column1, column2....)

The $NLS_LANG environment variable is set to AMERICAN_AMERICA.UTF8. I didn’t test other values though. The above settings works well for me.

Final notes:

  • Using character semantics and UTF8 character set will use more storage than latin1 + byte.
  • Avoid using M$ Word “smart” features or don’t copy/paste from word, it causes more trouble…..

 

Replacing AnyConnect with OpenConnect on OSX

I was having so much trouble with AnyConnect 2.5.x.

  • Sometime after it connect, nothing received out (you can still see package sends out). Basically you lost connection.
  • The split traffic doesn’t work. I would like to be able to access my LAN while I’m on VPN. I couldn’t figure out how it works.
  • It replace my /etc/hosts file silently. You can work around it by editing /etc/hosts.ac. Sometime if I forget, my hosts file will be lost.
  • It doesn’t save the username and password. Have to retype them every time.
  • New version 3.x doesn’t solve the problem….

So I decided to switch to OpenConnect, which is an open source client for Cisco AnyConnect SSL VPN. (http://www.infradead.org/openconnect/). Under OSX, it can be easily installed through MacPorts (http://www.macports.org/).

  1. Install MacPort (http://www.macports.org/install.php)
  2. Install OpenConnect
    sudo port install openconnect
  3. Connect with OpenConnect
    sudo openconnect -u USERNAME https://YOUR_VPN_SERVER

Notes:

  • At the time of writing this blog, the version of openconnect on MacPorts is 3.18 and the latest version is 4.07. To use the latest version, I just replaced the Portfile in my ports repo. I also submitted the new version to MacPorts. Hopefully by the time you install it, it will be the latest version. If not, you can always download my Portfile from here (https://raw.github.com/xcompass/macports/master/net/openconnect/Portfile). Then copy the file to
    /opt/local/var/macports/sources/rsync.macports.org/release/ports/net/openconnect

    Or create a local repo as described here: http://guide.macports.org/#development.local-repositories

  • To split the traffic, use this script: http://lists.infradead.org/pipermail/openconnect-devel/2012-June/000606.html. Replace the ROUTES at the top and path to vpnc-script at the bottom (should be /opt/local/etc/vpnc/vpnc-script for vpnc installed through MacPorts)

DB ACL Data Source in CakePHP 1.3

When doing the integration testing, we always want to use the test database. However, it is a little bit difficult with CakePHP when DB ACL is enabled in your application.

CakePHP try to be smart. It tries to create separate tables (prefixed with test_suite_) when we have our fixtures set up. However, DB ACL uses separate settings in Configure class:
Configure::read('Acl.database');

So when we do testing, we need to have an extra line in our setup before we run any test.
Configure::write('Acl.database', 'test_database');

Fixture setting may be needed as well.

Using Model in Component or other class in CakePHP 1.3

Some posts mentioned using

App::import('User');
$user = new User();

The issue with this approche is when you access the database, it uses default configuration. There is no problem until you are testing the class with a test database configuration or asking CakePHP to generate the test_suite configuration.

The correct way of doing this is:
$modelName = ClassRegistry::init('ModelName');

Installing Trac 0.12.x Python 2.6 on RHEL 5.8

A quick note to install those stuff.

Install EPEL repo (http://fedoraproject.org/wiki/EPEL).
yum install python26 python-devel subversion subversion-devel
Install setuptools
easy_install trac
wget http://subversion.tigris.org/downloads/subversion-1.6.17.tar.bz2
tar xjvf subversion-1.6.17.tar.bz2
cd subversion-1.6.17
wget http://www.sqlite.org/sqlite-amalgamation-3.6.13.tar.gz
tar xzvf sqlite-amalgamation-3.6.13.tar.gz
mkdir sqlite-amalgamation
cp sqlite-3.6.13/sqlite3.c /home/sysadmin/subversion-1.6.17/sqlite-amalgamation/sqlite3.c
cd /usr/bin
rm python
ln -s python2.6 python
cd –
./configure
make
make swig-py
make install-swig-py
cp -R /usr/local/lib/svn-python/libsvn /usr/lib/python2.6/site-packages/
cp -R /usr/local/lib/svn-python/svn /usr/lib/python2.6/site-packages/

/sbin/service httpd restart

Actionscript/Flash mask experiences

Some notes:

  1. both masker and maskee need to set cacheAsBitmap to true
  2. have to use transparent png or transparent swf file for masker, no jpg
  3. call setMask after loadMovie finished
  4. maskee can be multiple layers. Only need to setMask on top layer.

Sample code:

/*this.createEmptyMovieClip("mask_mc", 2);
mask_mc.createEmptyMovieClip("mask_submc", mask_mc.getNextHighestDepth());
mask_mc.cacheAsBitmap = true;

mask_mc.mask_submc.beginFill(0xfff000);
mask_mc.mask_submc.moveTo(0, 100);
mask_mc.mask_submc.lineTo(200, 100);
mask_mc.mask_submc.lineTo(200, 300);
mask_mc.mask_submc.lineTo(0, 300);
mask_mc.mask_submc.lineTo(0, 100);
mask_mc.mask_submc.endFill();*/


attachMovie("masker","mask_mc", 2);
mask_mc.cacheAsBitmap = true;

this.createEmptyMovieClip("mcCharMask", 1) 
mcCharMask.loadMovie("a.png");

this.onEnterFrame = function(){
		if(mcCharMask.getBytesLoaded() >= mcCharMask.getBytesTotal() && mcCharMask.getBytesLoaded() > 0){
		delete this.onEnterFrame;
		// set cacheAsBitmap before setMask. Sometime cacheAsBitmap value changes after loadMovie
		mcCharMask.cacheAsBitmap = true;
		mask_mc.setMask(mcCharMask);
		
		/*mask_mc.createEmptyMovieClip("mask_submc1", mask_mc.getNextHighestDepth());

		mask_mc.mask_submc1.beginFill(0xff0000);
mask_mc.mask_submc1.moveTo(0, 200);
mask_mc.mask_submc1.lineTo(200, 200);
mask_mc.mask_submc1.lineTo(200, 300);
mask_mc.mask_submc1.lineTo(0, 300);
mask_mc.mask_submc1.lineTo(0, 200);
mask_mc.mask_submc1.endFill();*/

		}
}

Using saveAll to save model that has multiple levels of hasMany (hasAndBelongsToMany) relationship with CakePHP 1.3

It turned out CakePHP failed to link GroupSet and Groups. No go…..

For example, GroupSet hasMany Groups and Group hasAndBelongsToMany users. They can be saved using one saveAll call. No need to deal with the IDs.

One level hasMany saveAll looks like:

Array
(
    [GroupSet] => Array
        (
            [name] => "Title"
        )
    [Group] => Array
        (
            [0] => Array
                (
                    [name] => On Gwoo the Kungwoo
                )
            [1] => Array
                (
                    [name] => More on Gwoo
                )
        )
)

The multiple levels hasMany (or HABTM) should looks like this:

Array
(
    [GroupSet] => Array
        (
            [name] => "Title"
        )
    [Group] => Array
        (
            [0] => Array
                (
                    [Group] => Array
                             (
                              [name] => Group1
                             )
                    [GroupMember] => Array
                       (
                          [0] => Array
                              (
                                [user_id] => 1
                              )
                          [1] => Array
                              (
                                [user_id] => 2
                              )
         )
         [1] => Array
                (
                    [Group] => Array
                             (
                              [name] => Group2
                             )
                    [GroupMember] => Array
                       (
                          [0] => Array
                              (
                                [user_id] => 3
                              )
                          [1] => Array
                              (
                                [user_id] => 4
                              )
                        )
                )
 )

How do I completely remove a file/folder from a Subversion repository?

To remove a file or folder and all history from an SVN repository, create a dump of the existing repository and filter out the unwanted file or folder. Then create a new repository to replace the existing repository.

The process to completely remove a file or folder from a repository involves several steps. Use the following steps to completely remove an existing file or folder and all history from your current live repository:
Create a dump of the existing repository:

svnadmin dump [path_to_repo] > dumpfile1

Filter out the unwanted file/folder:

svndumpfilter exclude [path_to_file/folder] < dumpfile1 > filtered_dumpfile

Create a new repository, to swap for the live repository:

svnadmin create repo_temp

Load the filtered dumpfile to new repository:

svnadmin load repo_temp < filtered_dumpfile

Verify the new repository and replace the live repo with the new repository:

rm [path_to_live_repo] -rfmv repo_temp live_repo

From: http://help.collab.net/index.jsp?topic=/faq/removefile.html
Reference: http://svnbook.red-bean.com/en/1.0/ch05s03.html