Ruby Code Snippets #3

Here's a simple Ruby code puzzle for you: if an argument is optional in Ruby, what do you think the result of the method would be if you passed 'nil' to arg2?

def mymethod(arg1, arg2='default')
 arg2
end

Well, (as expected?) if you pass a nil as an argument, then there is something being passed, and hence 'default' won't be passed:

mymethod('foo', nil) #=> nil

And if you passed it just a single argument, here's what it'll look like:

mymethod('foo') #=> 'default'

Helpful Methods for Ruby Debugging

Just something which may help you when trying to figure out what Ruby object you're dealing with:

object.methods   #-> returns all the available methods
object.inspect   #-> returns information about the object
object.ancestors #-> where the ruby object inherits from

Ruby Code Snippets #2

One thing I like about more 'recent' programming languages like Ruby is the ability to have multiple argument assignments, ie:

 var1, var2 = %w{ a b c }
  #=> var1 = "a", var2 = "b"

var1, *var2 = %w { a b c }
  #=> a = "a", b = [ "b", "c" ]

The first example, "c" vanished into the ether, while the second example, the remainder of [ "b", "c" ] is stored into var2. Multiple variable assignments isn't unique to Ruby by the way, I do know that Perl does it too, but I think that it is just that it always defaults to the second example behaviourally.

It's nice that the syntax allows you to choose between what you want and do not want.

Ruby Code Snippets #1

Here's yet another reason I find ruby so easy to code. Lets assume that you want to do filtering of a certain kind of input that is demarcated by 'START' and 'END' on a separate line by itself. Let's use an example like the data shown below:

This is an unwanted line 1.
This is an unwanted line 2.
START
This is a wanted line 1.
This is a wanted line 2.
END
This is an unwanted line 3.
This is an unwanted line 4.

Normally, the typical way of segregating the wanted lines with the unwanted ones is probably via writing code that logically looks something like this:

in_header = false         # variable to remember whether if in_header
while line = STDIN.gets
if line =~/^START$/      # found header, start trapping
  in_header = true
elsif line =~/^END$/     # end header, stop trapping 
  print line             # have to print the line boundary inclusive
  in_header = false
end
if in_header             # still in boundary, print line
  print line
end
end

But in ruby, the idea of applying ranges on regular expression matches simplifies the syntax down to a bare minimum:

while line = STDIN.gets
if line =~ /^START$/ .. line =~ /^END$/
  print line
end
end

This is one of the things that I have seen that's unlike any other language I've encountered before. Amazing and totally delightful!

Visualizing Differences Between Source Modifications in Vim

There's always something new to learn from #vim, and here's another gem that allows you to use vim's built-in diff capability to visually inspect the code changes on your source inline. Add the following into your .vimrc:

function() DiffPreview()
 diffthis | vsplit | enew | set buftype=nofile | read # | 1delete | diffthis
endfunction
map <F9> :call DiffPreview()<CR>

In this case, I've mapped it to the <F9> function key. How to use this: once you made all the changes you wanted, DON'T save the current buffer yet, and invoke the DiffPreview() function. It'll provide you with a highlighted 'diff' of all the changes you've made. Have fun!

Hacking MacOS X For Password Retrival

I had to resort to this trick as I have inherited an old Mac machine from a friend of mine, who in turn had inherited it from her children. The annoying thing is that the machine's password is lost, and I can't get access to updates or make configuration changes. Since I've gotten the 'green-light' to break into it, it's going to be a 'no-holds-barred'. Sorry Mac, but you're going down!

Mac Zealots are going to kill me for saying this, but breaking access to the Mac OS is incredibly trivial. In this case, I do not require and external medium for the exploit, but just the magic keystroke of '<Apple-s>', in order to get into it's 'single-user mode'. Do this by rebooting your machine and holding down your 'Apple' and 's' key simultaneously. You should get to a screen below:

standard timeslicing quantum is 10000 us
vm_page_bootstrap: 124982 free pages
mig_table_max_displ = 70
92 prelinked modules
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

using 1310 buffer headers and 1310 cluster IO buffer headers
ADB present:84
FireWire (OHCI) Apple ID 31 built-in now active, GUID 001451ff fe32b0e0; max speed s400.
CSRHIDTransitionDriver::probe: -s
CSRHIDTransitionDriver::probe booting in single user .. do not match
Security auditing service present
BSM auditing present
disabled
rooting via boot-uuid from /chosen: A827D24C-C773-3C25-8D8F-E027C540A4B8
Waiting on IOProviderClassIOResourcesIOResourceMatchboot-uuid-media
Got boot device = IOService:/MacRISC2PE/pci@f4000000/AppleMacRiscPCI/ata-6@D/AppleKauaiATA/ATADeviceNub@0/IOAT
ABlockStorageDriver/IOATABlockStorageDevice/IOBlockStorageDriver/TOSHIBA MK4025GAS Media/IOApplePartitionSchem
e/Untitled@3
BSD root: disk0s3, major 14, minor 2

If you want to make modifications to files:
 /sbin/fsck -fy
 /sbin/mount -uw /

If you wish to boot the system, but stay in single user mode
 sh /etc/rc

%

You might as well call this the 'super user mode', given that it gives out 'root privileges' on the Mac. While it may not be immediately apparent in Mac OS, its UNIX heritage is still apparent. (See the 'The Regents of University of California' copyright message above.)

The message shown above is not reproduced from fidelity given that there's no way to intercept stdout during startup, but rather roughly constituted from the output from 'dmesg', so don't panic if you don't see exactly the same messages.

And if you noticed the instructions given, the nice thing about the Mac's user friendliness is that it even tells you what to do on the command prompt on how to start all the Mac services required!

There's really no need to follow the instructions verbatim, given that we didn't boot into a single user because the harddisk has crashed, there no need to run 'fsck', which is just a fairly time-consuming process that checks your disk for errors. Just mount your root file system in write mode and start the Mac services:

% /sbin/mount -uw /
% sh /etc/rc
Checking disk
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
Mounting local filesystems
kern.sysv.shmmax: -1 -> 4194304
kern.sysv.shmmin: -1 -> 1
kern.sysv.shmmni: -1 -> 32
kern.sysv.shmseg: -1 -> 8
kern.sysv.shmall: -1 -> 1024
Resetting files and devices
Starting virtual memory
Removing /mach.sym /var/tmp/folders.501 /etc/nologin /private/_tmp_ /var/_run_ /
private/var/vm/swapfile0
Configuring kernel extensions

[... Other messages truncated ...]
%

There are more messages to that screen, but because I didn't trap the output from stderr, some of them are missing here. But they are not essential for our purpose.

And if you haven't realised it, at this moment you're in 'God Mode'. You have now full file, and system access to all accounts on the Mac OS system, just without the facade of the GUI.

If you have lost your 'Administrator' password, changing it is just a breeze, because you are 'root', you can just change the password of any account you want. However your 'Administrator' account is not actually called 'Administrator', and it does have it's own login name, so you'll need to find that out by checking your '/Users/' directory. For purposes of illustration, I'll assume that your Administrator's username is 'vincent'

% passwd vincent
Enter your new UNIX password: it's-a-secret!
Re-type your new UNIX password: it's-a-secret!

As expected, it doesn't even asks you what your old password is for the change! Well if really wanted to be nasty and insidious, you can create a 'backdoor' account, by changing the password just for the 'root' user, and you'll actually be able to access the computer even without the actual user knowing it!

Once you're done with the fiddling, type 'exit' to get out of the command line and back into the fancy GUI mode. Get into your 'System Preferences' and click on 'Accounts' icon. You'll probably see the screen like this:

As you see, the 'root' account is invisible to Mac OS, i.e. you won't see it as a valid user on your account panel. As typical Mac users, you won't be able to detect the intrusion, given that it doesn't show up as an additional user, normally a tell-tale sign that your machine is compromised.

Of course, you wouldn't have this problem if there is a way to disable the '<Apple-S>' keystroke. Well I don't for now, so it still poses a security risk for my home machine, which is why it remains for casual use as a really expensive music player.

Non-Greedy Regular Expression Matching

This applies to Ruby, and many other languages too (eg. Perl), just that I keep forgetting it. In order to have non-greedy matching enabled, append the question mark after the quantifiers, ie.

* (0 or more) greedy matching
+ (1 or more) greedy matching
*? (0 or more) non-greedy matching
+? (1 or more) non-greedy matching

For vim, the syntax is slightly more convoluted, so it's important to put a note here for my own reminder:

* (0 or more) greedy matching
\+ (1 or more) greedy matching
\{-} (0 or more) non-greedy matching
\{-n,} (at least n) non-greedy matching

There's no direct '(1 or more) non-greedy matching' in vim, but the 'at least n' operation should be the equivalent. For vim, these should normally be default and independent of whether if the 'very magic' or 'very nomagic' mode is set. See help in pattern.txt, for more pattern matching details.

Command Line Parsing using JFlex

What started out as a small set of commands for a tool I'm writing is slowing growing unwieldy to have to warrant enough repetitious code to parse the command line manually, and to wade through lines of if/else or switch statements (Don't you preach to me about the virtues of using the Command design pattern, for it is still unwieldy because it does not handle the parsing of arguments even the hash saves you from having long branching segments of code, which I don't mind. In my opinion, it's visually easier for me using folds, rather than to have file fragmentation of one command per file.)

Rather than having to deal with the unwieldy mess of buggy, manual coding using an ad-hoc mixture of Regular Expressions and StringTokenizers, I decided to start using a lexical analyzer instead. The one that I'm using is called JFlex, which is probably the most popular (or only?) one around.

Barring the initial learning curve, certainly having the lexical analyzer certainly makes life much easier, by automatically breaking down the command string into tokens each, without having to intervene to deal with handling white spaces and separators and such. A simple example for a lexical analyser that breaks up commands and arguments looks something like this:

/** The lexer for scanning command tokens. */
%%

%class CommandLexer

Parameter = [:jletterdigit:]+
WhiteSpace = [ \n\t\f]

%%

[:digit:]+ { return new Yytoken(Integer.parseInt(yytext())); }
{Parameter} { return new Yytoken(yytext()); }
{WhiteSpace} { /* Ignore Whitespace */ }
"-" { return new Yytoken('-'); }
"," { return new Yytoken(','); }

The example should hopefully be simple enough not to cause a 'cringe factor' or the need to refer to the Dragon Book.

There are 3 different sections in JFlex's definition file, separated by '%%' symbols. The first section is straightforward, it just allows you to include whatever that you wanted to include in the generated file.

The second section, is a list of definitions and directives that tells JFlex what to do. In this case, I've told JFlex to generate the the output to a file called 'CommandLexer[.java]'. Subsequently, the next two lines allows me to put in what I defined as 'WhiteSpace' or 'Parameter'.

The last section is where you define the grammar that helps the generated scanner code to discern what is a token, and in my case, what type of a token it is. In my example, rule 1 '[:digit:]+', matches 1 or more number and transforms that into a token, rule 2, matches what I call a parameter (which has either one or more digits or letters, and contains at least 1 letter in it). Rule 3, just tells the scanner to ignore all WhiteSpace characters, while Rule 4, 5 indicates what I define as separators, in my case the characters '-' and ','.

It must be noted that ordering is important. If I actually swapped the order of rule 2 with 1, because numbers will match the {Parameter} rule first, the [:digit:]+ rule will never match. JFlex will tell you that if that's the case (highlighted in red below):

Reading "commandlexer.jflex"
Constructing NFA : 16 states in NFA
Converting NFA to DFA :
.....

Warning in file "commandlexer.jflex" (line 13):
Rule can never be matched:
[:digit:]+ { return new Yytoken(Integer.parseInt(yytext())); }
7 states before minimization, 5 states in minimized DFA
Old file "CommandLexer.java" saved as "CommandLexer.java~"
Writing code to "CommandLexer.java"

The next thing to do is to actually create a actual token class, which is called Yytoken by default. An example of a typical Yytoken.java file looks somewhat like this:

/** A single scanner token. */
public class Yytoken {
public boolean is_separator = false;
public boolean is_int = false;
public boolean is_token = false;

public char separator;
public String token = null;
public int value = 0;

/** Default for range separator. */
public Yytoken(char c) {
   is_separator = true;
   separator = c;
}

public Yytoken(int value) {
   is_int = true;
   this.value = value;
}

public Yytoken(String token) {
   is_token = true;
   this.token = token;
}

public String toString() {
   if (is_separator) return "Range Token("+separator+")";
   else if (is_int) return "Int Token("+value+")";
   else return "Token ("+token+")";
}
}

To test it, you can write a simple harness to read from stdin:

/** Test class to try out the command lexer. */
public class UseCommandLexer {
public static void main(String args[]) throws Exception {
   CommandLexer command_lexer = new CommandLexer(System.in);
   Yytoken token = null;
   do {
       token = command_lexer.yylex();
       System.out.println("token = " + token);
   }
   while (token!=null);
}
}

That's probably a really basic tutorial in using JFlex, and to learn all of it probably requires having more of RTFM, but in the meantime, have fun in processing your command line!

Setting up Sequel for Postgresql

For those people who are using postgresql with sequel, just in case if you are having an error message like the one below:

Sequel::Error::AdapterNotFound: Could not load postgres adapter:
no such file to load -- postgres
    from /usr/local/lib/ruby/gems/1.8/gems/sequel_core-1.0.5/lib/sequel_core/database.rb:383:in `adapter_class'
    from /usr/local/lib/ruby/gems/1.8/gems/sequel_core-1.0.5/lib/sequel_core.rb:36:in `method_missing'

This probably means that you don't have the postgres adapter installed. One quick way to fix problem is to install the postgres gem via rubygems:

% rubygems install -r postgres

That should solve the dependency requirement problem.

Looking for Postgres Configuration's Path

Spent a rather long amount of time fumbling around trying to find where postgres's configuration files are resided. It didn't help at all that all postgres's configurations files are not found in the global '/etc' directory. Only after doing some source diving into postgres's SysV initialization script did I find where the locations of 'postgresql.conf' are:

Gentoo:

/var/lib/postgresql/data

Redhat:

/var/lib/pgsql/data

That's annoying because the configuration files, are stored in different locations for different distributions. Furthermore default configuration flags are different for distros as well, which I had to spent more time to look for the errors while trying to make sequel work with both distros, so it works whether if I'm using my standalone box or the server.

Also, it turns out that Gentoo has enabled localhost access by default, while RedHat didn't. So to 'fix' RedHat's 'pg_hba.conf' file, and uncomment localhost access to the database:

# IPv4-style local connections:
host    all         all         127.0.0.1         255.255.255.255   trust

Talk about 'distro fragmentation hell', which is no better than 'DLL hell' that Windows users suffer from!

Setting Up Postgresql

Was mucking around with postgresql again, given that I've decided to start everything at right footing, rather than to stick with mysql, which is what I'm more used to, for various reasons I've remembered from my database lectures in the past.

Installation's a little bit different from mysql, but with a bit of Googling, there are some pretty good instructions out there that makes things clearer. Subsequently after installation, there may be two error messages to take note of, which you may encounter after the install process:

psql: FATAL:  user "vincent" does not exist

If you're not running as user 'postgres', this error indicates that the user (in my case 'vincent') does not exist in the database. Upon installation, postgres only allow the user 'postgres' to access the database by default, so it's necessary have the right user profile to make configuration changes. This is done by invoking 'su' (or 'sudo' for some users) into 'postgres' and invoke the 'createuser' command:

% createuser
Enter name of user to add: vincent
Shall the new user be allowed to create databases? (y/n) y
Shall the new user be allowed to create more new users? (y/n) y

The rights allow users to create databases, and to add new users to the database, which on a standalone machine that I have, it's probably ok to have 'y' for both.

You're not done with the process yet if you haven't created your database for access, and if you executed 'psql' again, the following error props up:

psql: FATAL:  database "vincent" does not exist

This is relatively easy to fix. Since you've already have database creation access, just by issuing 'createdb' will initialize your database for use.

% createdb
CREATE DATABASE

Time to start creating and populating tables from here!

Understanding Rubygem's Load Errors

While rubygems is a great way of installing your packages without having yourself to dirty your hands, but if you installed your gems this way, it however does not mean that loading those newly installed gems is automatically.

It took me a while before I understood why my newly installed gems cannot be loaded through the 'require' statement, and giving me irksome messages like the 'no such file to load' error:

LoadError: no such file to load -- sequel
 from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
 from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
 from (irb):1

That unhelpful message actually implies that you're missing rubygems as a requirement. So make sure you load rubygems via 'require' before you load any other gems installed by it in your source file:

require 'rubygems'
# put all other requirements after this

The other way of permanently including rubygems is to export the -rubygems flag into your RUBYOPT environment variable. Different shells may vary, but here's how you'll do it in bash:

% export RUBYOPT=-rubygems

Do make sure you check if there are any existing flags in RUBYOPT first before overwriting it though!