Misty thoughts » programming

Setting up Solr for Drupal under tomcat

Kfir — Tue, 13 Jul 2010 06:37:56 +0000

Disclaimer: This method will probably not work for all servers running tomcat, it is more a personal guide then a general one.

Solr is a "popular, blazing fast open source enterprise search platform from the Apache Lucene project". Drupal search has traditionally been less then adequate. Using Solr to index and search you Drupal site is therefore quite helpful.

To start, download the Solr package (link) and the Solr drupal module (link). the current standard distribution is Solr 1.4 and Drupal 6.
Unarchive the tar files in a location that is reachable from your server.

We will assume that tomcat is installed in /opt/tomcat

In the Solr package there is an example application. That is the Solr application we are going to use. If you can write your own, you probably don't need this tutorial.

Step 1:
Copy the example application, found in the solr package under example/webapps to the tomecat webapps folder which should be /opt/tomcat/webapps. when copying the file rename it to something other the solr. this will enable you to have multiplie solr installations running under one Tomcat container. We will assume the app is renamed new_solr_inst. If you are in the tomcat root folder the command should like like this:
cp /path/to/solr/package/apache-solr-1.4.0/apache-solr-1.4.0/example/webapps/solr.war webapps/new_solr_inst.war

Tomcat will pick up the war file and open it automatically after a restart.

Step 2:
In the tomcat root folder create a folder to hold your Solr configuration and data.
mkdir new_solr_inst
Copy the example app configuration into that folder
cp -R /path/to/solr/package/apache-solr-1.4.0/apache-solr-1.4.0/example/solr/* new_solr_inst

Step 3:
You need to replace the original schema.xml and solrconfig.xml with the ones provided by the drupal module. Copy these two files to your Solr configuration folder. Assuming you are currently at the tomcat root folder
cp /path/to/solr/module/schema.xml new_solr_inst/conf
cp /path/to/solr/module/solrconfig.xml new_solr_inst/conf
The Solr app is now ready for use.

Step 4:
The last step is needed to inform Tomcat about the app and its settings. Go to conf/Catalina/localhost and create the file new_solr_inst.xml. In that file fill the following content:

Step 5:
You need to restart tomcat
/opt/tomcat/bin/shutdown.sh
/opt/tomcat/bin/startup.sh
And your Solr should be running. There are three tests we can use to check everything went well.

Tomcat has opened the war file. There is now a folder new_solr_inst in the webapps folder of Tomcat.
Solr has made a data directory in the Solr home folder (that is /opt/tomcat/new_solr_inst)
In your browser go to the Solr app http://domain:tomcat_port/new_solr_inst and you should see a welcome screen

If all is well you are ready to combine this Solr installation with your Drupal

Step 6:
Activate the Apache-Solr module and go to the Solr module settings page admin/settings/apachesolr
Assuming tomcat and drupal are running on the same server fill in localhost as the host name, fill in the port tomcat is running on and solr path should be your app name in the tomcat webapps folder. In our case is it new_solr_inst.

Save the settings. Drupal should inform you that a connection is made with Solr

Notes

It is not nessecary to put the app configuration in the Tomcat root folder. you can probably (though I have not tested this) place it anywhere you want. you do need to make sure that the environment tag in the Tomact configuration file (the one in the conf/Catalina/localhost folder) is pointing to the right place.
If you make changes to the schema you may need to dump all the indices Solr has made. you can do this via the Drupal admin interface for the Solr module.

jQuery animation problem in ie7

Kfir — Fri, 28 May 2010 11:24:30 +0000

For one of our projects, I was creating a photo bar. When the mouse was over one of the photos it was ment to increase in size and decrease back to original size when the mouse left. Unfortunately the client for which this was developed uses ie7 exclusively. This has brought two problems:
1. ie7 does not support the mouseleave event. I needed to replace it with mouseout event.
2. ie7 does not accept the position property in the jQuery animate function. Giving the position as a parameter will cause an error.

Getting session ID in Drupal 6

Kfir — Thu, 25 Feb 2010 20:49:22 +0000

Drupal saves its sessions in the sessions table. One can always query they table to get the session ID. There is however a faster way.
If the user is logged it, the session id can be found with the following code:

global $user;
$user->sid;

In any case the session id can also be read from the cookie:

$_COOKIE[session_name()];

How to truly disable drupal cache

Kfir — Fri, 29 May 2009 10:26:05 +0000

Drupal, like most CMSes uses cache intensively to speed things up. In a production site that is exactly what you what. However when developing, cache can cause you much pain.

If your an expert in drupal you probably already realized when you need to clear the cache in order to see changes and when not too. the developer module even makes it quite easy to do. However for me, and I am quite sure to quite a lot of other people it is not that clear.

"Wait", you might say. "Drupal has a disable cache setting". You would be right to say that. Only it doesn't completely disable the cache, only part of the cache, namely the page caching. To truly disable cache you need to do a bit of a hacking to the cache code. Yes I know they say to never hack core (cache is part of the core drupal release). for me this was worth it. I figure as long as you remember its there and don't use it in production you should be fine. The following piece of code needs to be added in include/cache.inc. There are two functions there that you need to edit:

cache_get
cache_set

In cache_get place the code right after the global $user. In cache_set place it at the start of the function. Here is the code that you need to add.
if(variable_get('cache', CACHE_DISABLED) == CACHE_DISABLED) { return 0; }
Kudos for this hack goes to Rolf van der Krol. Cheers mate.

Django and static files

Kfir — Fri, 26 Sep 2008 18:18:18 +0000

After a long break I have resumed my side project in django. Last night I came upon a problem. When testing a page outside django, the page was rendered correctly. When it was rendered through django the javascript files were not located.

At this point I have to admit I was a bit fullish and forgot to look at the web server's log to see if the files were correctly served. that was an hour and a half of trying to figure out why the javascript functions were not found. So after smartening up I found out that the javascript files were not found. It seemed that django kept looking for them in the wrong location. Using the example here I eventually got it all to working.

It comes down to this. in the settings.py file there is a reference to the media URL and to the file system path to the media directory. Adding the following code to the urls.py
if settings.DEBUG: urlpatterns += patterns('', (r'^tripcalc/media/(?P .*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}), )
Will make sure that while django is in debug mode, serving of the static files will be done via django. for production sites it is better to let a dedicated web server (such as Apache) serve these static files.
And that took care of it. I can now continue with the development. And it also enabled me to call the CSS file, so style is also shown. hurray.

Secret software

Kfir — Thu, 07 Aug 2008 11:17:40 +0000

An interesting talk about secret software and why it should not be allowed in the public sector

Installing Django on Dreamhost

Kfir — Mon, 19 May 2008 10:29:44 +0000

Dreamhost appears on the django site as one of the django friendly hosting services. Unfortunately, dreamhost does not officially support django. It does not have mod_python installed. Django is instead deployed using FastCGI. Hopefully sometime in the future mod_python will be added. There are a few good guides I have found, that explain how to setup django on a dreamhost account:

Jeff Croft has a good guide on his blog

Gordon Tillman also has a good informative page

The dreamhost wiki also has a guide

Between the three of them you can probably find all the information needed for installing django for use on a dreamhost account. I will not repeat what they explain but instead add some from my own experience.

Python

Dreamhost, at the moment of writing, is running python 2.4. Luckily it is possible for you to locally install python. I highly recommend it as it will enable you to setup the python environment exactly the way you want it, and it will make it easier to upgrade to future versions of python.

cd ~/soft wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz tar xvfz Python-2.5.2.tgz cd Python-2.5.2 ./configure --prefix ~/install/dir --enable-shared make make install

Where ~/install/dir is the directory you want python installed in. I followed dreamhost's Unix account setup guide and installed it under run. I recommend you do to, as it is easier to have full control over you /usr/local.
Also adding setuptools makes future installs easier

cd ~/soft wget http://peak.telecommunity.com/dist/ez_setup.py ~/path/to/yourpython/python ez_setup.py

This will add the easy_install script which will simplify adding packages to your own python install. The final step is adding the new MySQLdb package

cd ~/soft svn co https://mysql-python.svn.sourceforge.net/svnroot/mysql-python/trunk/MySQLdb MySQLdb easy_install MySQLdb

This is assuming easy_install is in your PATH. If it is not it needs to be added.

Your now ready to install django using your very own Python installation. That is detailed enough so I will move to two problems I met after the installation.

Post Installation Problems

I met with two problems that had frustrated me for a few hours. To save you the future installer some pain here they are in case you experience something similar

syncdb after admin activation: This should have been very obvious to me but for some reason it escaped me. After enabling the admin page you must run django-admin.py syncdb in your project home page. What happened was that I ran it before I enabled the admin application. This lead to the creation of the needed tables in the MySQL database, but no tables for the admin application. After enabling the admin application, more tables need to be created to accommodate the new application data. The errors I got gave me the impression that there was an error in the MySQLdb egg, so I reinstalled it, then tried to find some workaround. Eventually I realized that I'm just missing the admin tables.

.htacess: This was the real mind bender. I kept getting an internal server error saying that it reached maximum internal allowed redirects. It was obviously a configuration error so I compared my .htaccess with that of the guides and it looked the same. So I looked else, but I kept coming back to the conclusion it has to be in the .htaccess. Yet no matter how often I looked at it I couldn't find what was wrong. I need to point out that I am no expret when it comes to apache and that maybe if I knew more about it this would have been simple, but I didn't so I got some gray hairs before I realized I'm missing a space between the - and the [L]. A bloody white space! I was feeling furious and incredibly stupid at the same time.

Django is now up and running, and I like it so far. I sure is a lot nicer to work with then with JSP and servlets. Enjoy your python

Cascade delete in MySQL

Kfir — Fri, 25 Apr 2008 18:03:00 +0000

Now that I am actually getting payed to program instead of paying to program, I find myself in greater need of practical information. Putting some of it on my blog will help me keep all the knowledge in one place. To start off, I'll mention Cascading delete in SQL.
In the database schema there are 3 tables used to store different objects. As there is a many to many relationship between the the objects, there are relation tables.

Using cascading delete I can define that if an object is removed from its table, all relation tables entries with that object id are also removed. Quite handy. the way this achieves is MySQL is through the use of foreign keys. If I define in the table members the unique id of a user as a foreign key I can also define tell the RDMS to delete the entries in members where that foreign key exist. Here is an example of the definitions:

CREATE TABLE members ( usr_id VARCHAR(30) NOT NULL, grp_id VARCHAR(30) NOT NULL, FOREIGN KEY usr_id REFERENCE user (usr_id) ON DELETE CASCADE, FOREIGN KEY grp_id REFERENCE group (grp_id) ON DELETE CASCADE);

As you can see the table is created normally by defining the fields, then the foreign keys are named. The REFERENCE shows to which column in what table they refer to.

A Long day

Kfir — Fri, 17 Aug 2007 22:39:09 +0000

This has been one long, and interesting day

I finally got the code part of my project done. Though I still need to finish testing it, it looks like its all good. I had spent all yesterday developing the equations in Maple. I wont bore you with the math but there sure was a lot of it. I ended up with 3 full pages of math Luckily with some export and text replacement I could incorporate it into my python code. Then came the real hard part

I end up with an equation with my variable to the power of 8. Apparently there is no closed form solution for equations with a power greater then 4. Which leaves me with numerical technique. If your thinking what the? well your where I was this morning. Luckly google saved the day. What on earth did I do before google. There are ways to solve this given a range of numbers to look at, or a starting estimate. Both of which I could give. Scipy has some implementation to do that. So that was finally done. Now that I can find in focal length Y value, I finally could complete the whole calibration on the fly.

Amazing how two days of head scratching and math can be summed up by a few lines.

Then Paula got home, we had diner and I got IMed to come play some wow, which I just started doing again. It was fun, as it usually is.

Somewhere in there I also managed to learn how to fix the boot record for windows. I was removing linux from Paula's computer and while removing grub things went wrong and the windows partition just wouldn't boot. In come Are, which is one of the best IT guys I know. I really think there is very little he doesn't know about anything tech.

I wanted to make a longer post, but its late, and I'm a bit, eh, tipsy, so this will have to do.

Good night

Good Spirit

Kfir — Wed, 01 Aug 2007 15:28:55 +0000

The last few days I have been making great progress on my thesis. Things are finally starting to fall into their place and the pace is great. I managed to solve some weird problems and I am now at the point where the difference between the calculated and the measured matrix is practically zero. that is, when projecting using the both matrices, the relation between both projections is 1.00008, which is great. Also while projecting i get a max of 2.3 pixel error and an average of 0.8 which is totally acceptable for such an early stage. All this advancement sure makes me feel good. Not that I intend to keep going with the University. It's been enough for now, I really am looking forward to doing some more projects.

Which brings me to Joost. Still no word from them. I guess its normal, I should expect a week or two, but I really would like to know if i get a shot at working for them. the more i think of it, the more I like the idea. I guess Ill just have to wait and see.

And now its time for me to get back into my Kruppa's equation explicit form development. which is the last thing I still need to program.