Saturday, June 14, 2008

Installing OpenGrok On Ubuntu Linux

I am really impressed with OpenGrok, a web-based source code search engine that I've found while I was trying to look up OpenJDK's source code. It is pretty cool as OpenGrok allows you to point your browser into an exact line of source code in your respository, allowing citations directly for discussion using hyperlinks, rather than cutting and pasting chunks of code. I find useful for annotating code, like when I'm using a wiki in conjunction with it to document design considerations for the source.

I'm only supplementing OpenGrok documentation because there were some parts of it which were less clear, taking longer than I expected to get it running. Hopefully, these instructions will help you cut down your setup time.

The assumption is that you're installing from a bare-bones Ubuntu system, and all commands here assume that you are root, which if you're like me, coming from a Gentoo background and sick of typing sudo all the time, you stay rooted all the time by using:


sudo -i


The next thing to do is to get all the relevant software via aptitude. I'll be using Apache Tomcat as my application server:


aptitude install sun-java6-jdk tomcat5.5 exuberant-ctags


Before we set up OpenGrok, we need to create the directory structure to store the files. For the sake of brevity, I'll use the same directory structure from OpenGrok's EXAMPLE.txt:


/opengrok
|-- bin
|-- data
`-- source


Download the tar.gz archive from its website, unpack and extract it. Copy the OpenGrok binaries into the /opengrok/bin:


# cp -r run.sh opengrok.jar lib /opengrok/bin


Edit run.sh and setup up the following parameters:


SRC_ROOT=/opengrok/source
DATA_ROOT=/opengrok/data
EXUB_CTAGS=/usr/bin/ctags


Note that I have put in the default location for the installed ctags for Ubuntu, you may have different locations/application names depending on your Linux distro. You'll then have to configure the web application. Go to the directory where you've downloaded your files, and unzip source.war to make modifications:


# mkdir source
# cd source
# unzip /path/to/opengrok-release/source.war


And make changes into WEB-INF/web.xml. The completed changes look like this:


<context-param>
<param-name>DATA_ROOT</param-name>
<param-value>/opengrok/data</param-value>
<description>REQUIRED: Full path of the directory where data files generated by OpenGrok are stored</description>
</context-param>

<context-param>
<param-name>SRC_ROOT</param-name>
<param-value>/opengrok/src</param-value>
<description>REQUIRED: Full path to source tree</description>
</context-param>

<context-param>
<param-name>SCAN_REPOS</param-name>
<param-value>false</param-value>
<description>Set this variable to true if you would like the web application to scan for external repositories (Mercurial)</description>
</context-param>


The text coloured red are the parts where you need to make modifications. The tags in blue indicate where the XML has originally been commented out by <!-- and --> you'll have to take them away.

Once that's done, you'll have to rezip the .war file back in place, and put it into Tomcat's webapps directory:


# zip -r source.war ./
# mv source.war /usr/share/tomcat5.5/webapps


After which, we'll need to configure our source code for OpenGrok to use, and set it up:


# cd /opengrok/source
# cp -r /your/source/code/ .
# java -Xmx1524m -jar opengrok.jar -W /opengrok/configuration.xml -P -S -v -s /opengrok/source -d /opengrok/data


This will generate the list of indices that allows OpenGrok to cross reference your source code. With that done, the final task is to setup Tomcat so that it has correct privileges. Append the following lines to /etc/tomcat5.5/04webapps.policy:


grant codeBase "file:${catalina.home}/webapps/source/-" {
permission java.security.AllPermission;
};

grant codeBase "file:${catalina.home}/webapps/source/WEB-INF/lib/-" {
permission java.security.AllPermission;
};


I'm just being cavalier here by giving full security access to OpenGrok, which should be safe. But I only doing it given because my application is firewalled from the outside world, so do make your own security adjustments appropriately! Once that's done, restart Tomcat:


# /etc/init.d/tomcat5.5 restart


You should now have a functioning own OpenGrok respository to play with! However if you get an error with the stack trace showing Apache Lucene not being able to create a file, grant full permissions to the data directory:


# chmod -R 777 /opengrok/data/


 

15 comments:

Anonymous said...

thanks for this post (lasciate ogni speranza voi ch'entrate? do you love Dante?)

x said...

Haha!

I'm surprised that you've picked that out - it's quite a well known phrase from his famous work, the Divine Comedy, not that I could read it in Italian though :)

Anonymous said...

Yes, this is helpful. But, I'm trying to setup OpenGrok for sources of three different products, and their source roots are not under common root directory. Besides, I want to narrow down the searches of different products's source to just that source root.

So, I thought setting up different contexts and setting up different .war files to deploy on Tomcat would do the trick. But, unfortunately, for some source code elements, it just defaults to /source/s=? and it doesn't find the element. I know the element is defined in the source code...

Any extra docs you can put out for this kind set up would be great.

x said...

Hi anonymous,

I understand your problem here, and it isn't uncommon to have multiple projects running concurrently, but to only search sources of a particular project.

I've read OpenGrok installation's README again, but there wasn't anything specific that I found which changes the redirection to the /source/s?= url.

It is likely that the url redirection is hardwired, and it means deploying on different .war files will not solve your problem.

I think how OpenGrok separate projects is by identifying each source sub-directory as a separate project, and once files are setup this way, the dropdown box at the last item of the search field should allow you to select the project you want to limit your searches from.

Hope that helps.

Anonymous said...

also make sure that the path to configuration.xml file is correctly pointed to in web.xml

Anonymous said...

in your tut you mention

cp -r run.sh opengrok.jar lib /opengrok/bin

where are run.sh and opengrok.jar supposed to come from? in the tar.gz archive these files are not present o_O

x said...

Hi Anonymous (at May 16, 2009 1:58 AM)

You're kidding me?

The current version I've fetched: http://opensolaris.org/os/project/opengrok/files/opengrok-0.7.tar.gz


The output from it:

-rw-r--r-- 0/0 4838 2008-08-06 14:51 opengrok-0.7/CHANGES.txt
-rw-r--r-- 0/0 18943 2008-08-06 14:51 opengrok-0.7/LICENSE.txt
-rw-r--r-- 0/0 10830 2008-09-15 15:36 opengrok-0.7/README.txt
-rw-r--r-- 0/0 113 2008-08-06 14:51 opengrok-0.7/paths.tsv
-rwxr-xr-x 0/0 4496 2008-08-22 08:48 opengrok-0.7/OpenGrok
-rwxr-xr-x 0/0 686 2008-08-06 14:51 opengrok-0.7/run-quiet.sh
-rwxr-xr-x 0/0 948 2008-10-10 08:54 opengrok-0.7/run.bat
-rwxr-xr-x 0/0 1170 2008-10-10 08:54 opengrok-0.7/run.sh-rw-r--r-- 0/0 6749 2008-08-06 14:51 opengrok-0.7/EXAMPLE.txt
-rw-r--r-- 0/0 1289806 2008-10-17 12:00 opengrok-0.7/lib/ant.jar
-rw-r--r-- 0/0 515920 2008-10-17 12:00 opengrok-0.7/lib/bcel-5.1.jar
-rw-r--r-- 0/0 65261 2008-10-17 12:00 opengrok-0.7/lib/jakarta-oro-2.0.8.jar
-rw-r--r-- 0/0 524941 2008-10-17 12:00 opengrok-0.7/lib/jmxremote_optional.jar
-rw-r--r-- 0/0 538618 2008-10-17 12:00 opengrok-0.7/lib/lucene-core-2.2.0.jar
-rw-r--r-- 0/0 16281 2008-10-17 12:00 opengrok-0.7/lib/lucene-spellchecker-2.2.0.jar
-rw-r--r-- 0/0 25595 2008-10-17 12:00 opengrok-0.7/lib/org.apache.commons.jrcs.diff.jar
-rw-r--r-- 0/0 58159 2008-10-17 12:00 opengrok-0.7/lib/org.apache.commons.jrcs.rcs.jar
-rw-r--r-- 0/0 97693 2008-10-17 12:00 opengrok-0.7/lib/servlet-api.jar
-rw-r--r-- 0/0 133580 2008-10-17 12:00 opengrok-0.7/lib/swing-layout-0.9.jar
-rw-r--r-- 0/0 982475 2008-10-17 12:00 opengrok-0.7/opengrok.jar-rw-r--r-- 0/0 2915289 2008-10-17 12:00 opengrok-0.7/source.war

% tar -tzvf opengrok-0.7.tar.gz

Clarkster said...

This is an obvious newbie comment, but after I have done all this, how do I use opengrok? I did a netstat -a, and I don't see any ports open for open grok.

Jim P said...

Thanks for posting this. It was helpful. I'm still new to OpenGrok, but I think 3 updates worth mentioning are:
1) Opengrok 0.9 has a deploy option that does some setup for you.
2) I think run.sh needs to be put in a cron job following commands to update your source code from revision control.
3) I think the webapps policy changes here may be missing something. I had to set /etc/init.d/tomcat6 TOMCAT6_SECURITY=no

I hope this helps some people get this great tool working. Thanks again.

Barry Sprajc said...

Hi Vincent,

Thanks for you're blog on OpenGrok.
It is exactly what I needed to get started, and was very helpful to me.

Barry S

Anonymous said...

This looks promising and easier to setup: https://github.com/cbess/text-sherlock

Shikher said...

Great post!

You've specified SRC_ROOT=/opengrok/source
in the configuration shouldn't it be the same in the WEB-INF/web.xml file as well ?

/opengrok/source
instead of
/opengrok/src

Unknown said...

Hi Vincent,

I am facing the following exception java.lang.NoClassDefFoundError: org/apache/commons/jrcs/rcs/ParseException
while running
"java -Xmx1524m -jar opengrok.jar -W /opengrok/configuration.xml -P -S -v -s /opengrok/source -d /opengrok/data"

can you help me out to sort this problem.

Hemu said...

thanks !!!!

Anonymous said...

Is there a way to avoid downloading the entire source code? I mean to say can we use a shared server which has the code base?

Post a Comment