Tricks on handling multiple line inputs and outputs

There are some of the trivial bag of tricks that I don't normally take notice, which does often comes back to bother me. I just have to trip on Finagle's Law whenever I forget these tricks after long disuse. Things like handling multi-line inputs and outputs seems to fit into that category, so here's all I can recall form my own reference:

Printing multiple line outputs with a single echo command

The special escape for newline in bash is $'\n'. Here's an example how it's being used:

% echo "Hello$'\n'World"
Hello
World

Other non-printable characters can be similarly escaped this way (e.g. The NUL terminator represented by $'\0').

Reading multi-line inputs from stdin

This is achieved by using the << operator:

% cat <<TEST
> This is a test
> Hello World
> TEST
This is a test
Hello World

Reading multi-lines of a file into a single variable

Because spaces are treated as field separators in bash, and it won't try to read past the first separator, you need to force the shell to treat nothing as a separator, by modifying the IFS (Input Field Separator):

OLDIFS=$IFS IFS= var=($(< filename.txt)) IFS=$OLDIFS

This above example assumes that you have a file called filename.txt, and at the end of the operation, the entire contents of the file is stored in the var variable.

Matching multiple lines using grep

When searching for two or more consecutive words within a sentence on a text file (or HTML), sometimes it is possible that the words are span across multiple lines, hence making typical word search, barring from using a script, difficult. One of the neat tricks to accomplish this without using any script, is through GNU grep's -P flag.

Assuming that you have a file called test.txt that spans two or more lines:

I like to say a nice warm "Hello
World" to all of you out there!

To search for the consecutive words "Hello World":

grep -P "Hello\WWorld" test.txt

Curiously, it does not seem to work on Gentoo/Ubuntu linux's version of grep but works only on my old RedHat distro only. Maybe it's something that has been phased out.

Splitting inputs into multiple lines

The easiest way to split things on the command line is to use the sed
command:

% echo "Hello World" | sed -e 's/ /\n/'
Hello
World

If you want to do this within vim, it is a little bit less straightforward but still doable:

:s/ /<CTRL-V><ENTER>/

Replace the items in the angled-braces with the actual keystroke itself. The screen the output actually looks like:

:s/ /^M/

So don't be concerned by the visual difference.

Wireless problems on ipw2200 (Intel Centrino) on Linux

I am quite pleased that the gentoo-sources for the Linux kernel has been upgraded to use version 1.2.0 of ipw2200 driver. The ipw2200 driver is used for all(?) of Intel's built-in wireless laptop chipset. The last stable update, 1.1.4, was incorporated into the kernel that has been unreliable for me in connecting to certain access points, especially to the Apple Airport that I have in the office.

On Ethereal/Wireshark, I keep seeing 10-15 resend packets per packet sent to the Airport successively, and 10-15 duplicated packets returning from time to time, annoying the least, but almost unlikely to be a connectivity problem, given that the airport just sits 0.5m and is facing squarely my laptop.

Access to open wireless access points, however, seems to work without a hitch. Maybe is just problems with WEP encryption with version 1.1.4, or maybe it is just something else. Whatever it is, it seems like version 1.2.0 has taken it away.

The other good news about the kernel source version 2.6.22, is that it allows the 'promiscuous' mode of the wireless card to be available as a configurable option now. This means that WEP cracking tools like aircrack-ng will be able to listen, and use replay-injection attacks, something that was only available to older, established wireless chipsets like Atheros, Prism or Orinoco in the past.

Not that I find it fun cracking WEP networks. But having to know I can, makes it useful if I ever were ever in desperate need for Internet access in a non-open wireless network environment. I very much doubt so though.

With the Blackberry, and in the future, more iPhone-like Internet access becoming ubiquitous, it'll be even harder to find an (il-)legitimate reason to need to piggyback on other people's wireless network, because you won't need to!

Find out version of libraries installed in Linux

It is just one of those things about Gentoo that I never could remember. I was needing to find out what version of libgtk I was using, and for the life of me, I couldn't remember what command was required to find it out, until someone kindly told me that you can do that using 'emerge -pv'.

Funnily, it didn't come to me intinctively, even though I use the command to query before upgrading packages all the time. It is just that I've never thought to use it as a tool for library version checking. Guess that might be the same way on other distributions to find out about their the versions of their installed libaries as well. For .deb (Debian/Ubuntu) based distros, try the 'apt' command, while those on the .rpm (RedHat/SuSE) based distros, try 'rpm' or 'yum'.

What package names should you search under? Frankly, I don't have a clue. If there is one bad thing about having a fragmented market distribution base like Linux, it'll have to be the confusion of configuration layouts among the competing distributions, which makes finding a solution a rather time consuming process.

Blog Action Day - Stop Being 'evian'!

'evian' in reverse is spelt 'naive', which sometimes lead me to suspect if the bottled water manufacturer is really thinking about consumers that way.

Perhaps so, given that the bottled water market is an unnecessary one, one that is fueled by consumers' irrational demand for wanting to pay for a healthier, chic drink to soda.

Unfortunately the problems caused by the demands for bottled water exhibit itself as a strain on the world, manifesting itself as a net contributor to global warming. If you think about it, transporting water from some Swiss mountain range a continent away and shipping to you as drinking water that has no material differences in terms of quality, is just a blatant waste of resources.

The energy costs of involved in storing, bottling, and transporting the water though, is substantial, especially when compared in relation to the negligible benefits it brings you compared to typical tap water.

Generally, there fundamentally nothing wrong with the water from the tap, as often is the case in most countries around the globe. In fact, in comparison, tap water usually have to pass even more stringent water quality tests, in addition to that, usually comes fortified with elements like fluoride, which helps in preventing tooth decay.

The increase in demand for bottled water has increased the drink industry's ravenous hunger for plastic, like the polyethylene terephthalate (PET) bottles that we are commonly use for soda drink bottles.

While recycling these bottles is a viable alternative, statistics are showing that our recycling efforts have not been stepping up with our consumption demands. In the end, a sizable amount of these single use bottles often end up polluting the lakes, rivers and seas of the world.

Some people I've met had strenuous objections to drinking tap water, citing reasons that it tasted awful compared to the refreshing taste of bottled water. I shall not argue on whether if these perceptions are just psychological, but instead I'll like you to think about it in a rational, economically-inclined way: paying for a bottle of water that costs 4-8 times more expensive than a litre of petrol, really sounds like extreme lunacy.

It's thus no wonder why companies can call us 'naive' and still be able to get away with it.

Education: NOT a Requisite for Success

By success, let me use an arbitrary (and pardon me, superficial) definition, as a person's net worth that is a magnitude greater than his peers of the same age group.

The first person I knew who qualified that criteria was a classmate of mine many years ago. He was someone who frequently skipped school, and eventually got himself held back for a year because of his failing grades. The reason, was that he was out and about, starting his own mobile phone business, initially by buying up used phones and reselling them, and later 'modding' them for resale at substantially marked-up prices.

I had a pretty bad impression of him, not just because of his truant behaviour, but of his cut-throat attitude, even towards friends. But in all fairness, he had always been upfront about it, telling us not to seek him for deals, because business was all about making profits, not friends.

Back in those days, I had a staunch but naive belief that success only comes from the books. But he proved me wrong, big time.

When he first started out, it was just running about the campus as a middleman, buying and selling phones from people and making small profits on the side. By the time I graduated, he had earned enough to start his own store, and proudly declared that he was raking sufficient money to pay for his own car upfront.

Which was no mean feat even for an adult, considering that a modest car in Singapore those days would have costed up of $100,000. My net worth was no more than $1000 then, making him the first person I know who got successful independently.

Since those days, I've gotten to know many other successful people, and those who did attain success through the education route. Many of them have reached the pinnacle of their careers, one of them who is a Vice-President of a Fortune 500 company, while another is an executive who holds directorship in a number of publicly listed companies.

But what fascinates me is not about them, but the stories that they tell of their bosses: One did not complete his high school education and the other was a university dropout. Surprisingly, even from my limited sample of the most successful people I've personally met, this trend holds, where the majority of them came from a background where education is not the decisive factor for their success.

I have met many people who have this misguided belief that education is a 'be all and end all'. In fact, I was one myself too. But that isn't really so. There are some who blindly pursuit the paper chase without considering the implications of their actions, only to end up without a purpose after graduation and laden with huge debt of student loan to repay.

Truth be told, education does not even guarantee you a job, let alone success in life.

I have known people that had graduated in my field who ended up becoming tele-marketers, insurance agents and even one who now works in a fast food outlet. They may be the few cases that I personally know of, but I believe they can't be the only ones who buck the maxim.

But don't mistake me for trying to lambast the value of education. I'm not.

Education is important, if you have the passion to learn. But we must be aware that a piece of academic qualification will only help to open that first door of opportunity for you, but it is certainly no guarantee of an automatic success in life. That is ultimately decided by factors like personal motivation, aptitude, and opportunities when they present themselves to you.

While we do not have a choice in deciding our smarts, or when lady luck will smile on us, but one thing that we're able to influence, that is how determined we are to achieve success. My philosophy on success is easily summed up by Margret Thatcher's quote:

"I do not know anyone who has got to the top without hard work. That is the recipe. It will not always get you to the top, but should get you pretty near."

Those who are motivated, will be the ones who make the most out of their education, or excel in whatever they do. But the reverse is almost never true.

Links Between Singapore and Burma

I've long known about the connection between Burma and Singapore before the recent increased media coverage on the plight of the Burmese under the military junta's rule that has brought the issue back into focus. The knowledge of the drug links between Singapore and Burma was first raised by Dr. Chee Soon Juan, highlighting the government's association with Steven Law, the son of the drug kingpin Lo Hsing Han. He was just a lone voice among the wilderness then, and there were hardly anybody who noticed or even bothered.

Not at least until the recent spate of atrocities committed by the military against the protesters for democracy. There has since been a number of reports that has resurfaced, linking Singapore's nefarious associations with the junta, through online sources from The Online Citizen, and foreign news agencies, like Sydney Morning Herald and The Age[1].

Singapore is known for being particularly litigatious when it comes to questioning the integrity of the government, often resulting in punitive lawsuits against individuals or even news agencies, whenever there are any remarks that even whiff anywhere near slanderous towards their political leaders.

But we are greeted with silence, when presented with these allegations that our country has been providing with infrastructure for spying on their citizens, providing military hardware for the junta to commit these atrocities, yet it was hardly given any investigative coverage upon, let alone to be open enough to be disputed or challenged in the political arena. To me, this amounts to a tacit admission, that the sources of these allegations are likely to be true.

This is also stark proof of the poor journalistic freedom in Singapore. If there is truly freedom for investigative journalism, such articles will have been on the front pages of our papers, and not though the Internet by the efforts of active citizen journalism[2]. It is the function of the free press 'in keeping them honest'[3], without which, is probably the reason why these clandestine transactions with Burma are left unchecked in the first place.

Our country is not as innocent from blame as we would like to have. Our police force behaves as if it is the natural extension of the Burmese junta, in deploying the same intimidation tactics against anybody who supports the cause of peaceful democratic protests. Ironically in a way, we are seeing a reflection of Burma through the actions of our own country. To quote lalaland9:

"I am totally ashamed when I saw how the police were stopping people from entering St Martin's Drive (where the Burmese embassy is situated), demanding their NRIC number and intimidating them to leave."

From the limitations of a free assembly by the people[4], like the Odex protest incident, where 8 youths were met with an excessive force of 4 anti-riot trucks, to the restriction of personal freedoms in expressing our moral views by preventing us from wearing shirts that supports the Burma cause, they are all gross violations of our rights as citizens of Singapore. When we cast ourselves in that light, how much better are we Singaporeans when compared to the Burmese?

Internationally, besides famous for being straight-laced and inflexible people, we are now slapped with an additional tag of being 'without a heart', as mercenaries, people who are swayed only by financial gains. If that is the impression the Burmese and the rest of the world have about us, it would only mean that there is something really wrong with our society. If so, then it is probably high time we examine and reflect upon our own actions.

If the government is complicit in aiding the military junta, the facts and details has to be made known to the public. On a personal capacity, there is little we can do, legally. As I have mentioned in the past, I believe that actions can and should be directed at the parliamentary process, that we should press the elected minsters to question and address these issues in parliament. It is only appropriate that we, as voting members of this country, let our constituency MPs know of our concerns about these purported links with the military junta.

While I am neither optimistic, nor counting on the responses (if even any at all) of the PAP constituents in regards to this issue, I do hope that other MPs, be it the opposition or NMPs, take up the cudgels, in at least motioning for a parliamentary inquiry to take place, or to table in these questions, questions that needs to be asked.

It is time we all know the facts, and learn how deep the rabbit hole truly goes.

---

[1] The Australian coverage from the two papers are apt, given that they circulate both in Sydney and Melbourne. They form the largest collective group of Singaporeans outside Singapore. Source: Straits Times, Sep 30, 2007, Home away from home For Singaporeans overseas - "[T]he OSU, which is under the Prime Minister's Office, began with Sydney and Melbourne as Australia has the largest concentration of overseas Singaporeans - some 40,000 reside there."

[2] I must commend on 'The Online Citizen's coverage of this issue in a factual, objective manner, as well as the other news that the mainstream press have been refusing to cover. The site is honesty starting to act like a contender to as an alternative journalistic outlet. Keep up the good work guys!

[3] The phrase is pinched from CNN's segment 'keeping them honest', where their investigative journalism helps to keep the government and politicians in check.

[4] IANAL, but as far as I know, the freedom of free assembly is enshrined in the Constitution. It is totally absurd, unconstitutional that a statue of limitation an assembly of more than 5 people can be legal, given that it runs against the spirit accorded by our rights that's defined by the Constitution.

This article is written in support of The Free Burma movement.

The Preference for Hard Languages

A language is a tool. When skilfully used, it allows you to craft great works. This is the same for computer languages. Engineering a good piece of software is, more art than science. The skill in designing and writing pieces of code that function seamlessly is like that of a sculptor, who is constantly crafting his works to achieve perfection. Those little touches of detail on usability and functionality, or in just providing a sensible error prompt, are equivalent to the amount of detail a dedicated sculptor puts in, chipping away the pieces of imperfection to reach the ultimate state of beauty.

Unsurprisingly, some tools are better than others. A chisel is and a hammer is better used for carving rocks than sandpaper is. A stick of dynamite makes it faster to shape Mount Rushmore than with just a pair of chisel and hammer. In the end, it is a matter of choosing the right tool for the right job.

It is not uncommon to come across people who are adamant on their choice of a 'hard' language when it comes to writing software. But the myth about computer languages is, that 'hard' is really subjective.

The choices of newly popular languages like Python or Ruby is generally no harder than old school languages like C or Java. 'Hard' languages can be easily spun around and to be interpreted as 'easy'. Easy because it will allow you to write solutions with considerable less lines of code.

Prolog and Lisp are more suited for solving logic problems, like the 'Missionaries and Cannibals' problem. It will probably just take a skilled programmer less than 10 lines of code to solve, whereas the same thing will take a good C or Java coder hundreds of lines to accomplish.

Conversely, higher level languages take away the nitty-gritty details of the underlying hardware for which may sometimes be undesirable. Someone can write a Ray Tracer in Python or Ruby, but nobody will expect the language will be a real fit, or to be executing at a speed that is of any realistic practicalities. Abstraction has its cost, and while this cost may be acceptable for some classes of applications, will not be so for others.

In the end, the choices of languages are all about its expressiveness and suitability. For people who love the self-celebration of their choice of 'hard' languages, it either smacks a sense of insecurity seeking justification, or a case of 'the person with a hammer' - everything else looks like a nail!

Such people if they can't stop ranting about how hard or cool 'language X' is, my remedy is to recommend them to learn Malbolge, the language that is hard. Good luck trying to code in that :)

Profiling in Java

In any Computer Science lectures that has anything to do with the topic of profiling, the story of how 'awk' got a performance boost by 25% just using gprof never fails to not be mentioned. The story ends with finding that offending loop code that accounted for most of the wasted computational time.

What makes it such a good story, is probably because that a quarter of the performance drag hinges on just a single line of code. But before anybody starts embarking on their own witch hunts in looking for such performance bottlenecks, I'll like to remind you about the other story that CS students are always told, the one related to by the 'Grandpa of Computer Science', Donald Knuth, on premature optimization:

'There is no doubt that the 'grail' of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.'

This is important, because the first thing that a programmer should look out for, is correctness. You can always speculate about where performance bottlenecks are, but unless you're using something as obvious as a 'Bubblesort', you can't really be sure where the major performance overhead is going to come from.

This is why profilers are handy. It tells you where exactly the performance bottlenecks are, rather than trying to make wild guesses about them before implementation, and writing obscured optimizations that will most likely add unnecessary complexities and potentially introduce more bugs.

Profiling in Java

The equivalent of 'gprof' in the GNU/C world is named, unimaginatively, as 'hprof' for Java, a natural succession from the letter 'h' to 'g', just as the 'C' language is the successor to 'B'. Gosh, aren't Computer Scientists lacking in imagination!

An example syntax for invoking a profiler looks like:

java -agentlib:hprof[=options] ToBeProfiledClass

Hprof allows for the measurement of CPU cycles, Memory Usage, and Thread traces, so it is absolutely helpful in determining the state of the application, and sometimes gives you additional insights in bug tracking either from traces, or locate the sources of memory leaks. I won't provide more usage details, but a more comprehensive run through of hprof can be found here.

Understanding Hprof's output

SITES BEGIN (ordered by live bytes) Sat Sep 29 13:46:46 2007
        percent          live          alloc'ed  stack class
rank   self  accum     bytes objs     bytes  objs trace name
  1 18.07% 18.07%    387192 16133    390912 16288 302005 java.lang.String
  2 12.04% 30.11%    258144 16134    258144 16134 302007 com.sun.tools.javac.util.List
  3  6.14% 36.25%    131512    2    131512     2 301079 com.sun.tools.javac.util.Name[]
  4  6.12% 42.36%    131088    1    131088     1 301080 byte[]
  5  6.12% 48.48%    131072    1    131072     1 301679 byte[]
  6  3.06% 51.54%     65536    1     65536     1 301187 byte[]
  7  3.06% 54.59%     65536    1     65536     1 301677 byte[]
  8  1.60% 56.19%     34240  428     34800   435 302250 byte[]
  9  1.11% 57.30%     23856  426     23856   426 302248 com.sun.tools.javac.code.Symbol$MethodSymbol

The various columns indicates the following:

• rank - self explanatory
  
• percent self - percentage of memory taken up
  
• percent accum - cumulative percentage of memory taken up from rank 1 to rank N
  
• live vs. alloc'ed - the actual objects that are active vs. objects still held in memory i.e, not garbage collected
  
• stack trace - The corresponding id given to a thread trace that is holding onto a reference to this memory
  
• class name - The type of object the memory is allocated
  

A profile dump is only meant to be an indicative measurement of the application at the given time, so when you profile is important as well. You'll most likely want execute a profile dump when the CPU load is high rather than when it is idle. The magic keystrokes that allows you to do a dump, is 'ctrl-\' in Linux and 'ctrl-break' in Windows.

Using Hat

Hat or (JHat as is now called in Java 6 SE), is a profile analysis tool which allows you to have make relational queries (like SQL) of memory usage, which is something that's pretty interesting.

( Screenshot of Hat's Web Application Interface )

While I've known Hat for quite a while already, but I have given up a long time ago for not managing to get it to work. But recently, there is a new bug report filed for the error, which explains the reason for my problem is because Hat only being able to read the binary format output from hprof.

So make sure you have the following parameter for binary format activated:

java -agentlib:hprof=agentlib:hprof=heap=sites,format=b ToBeProfiledClass

If you read the bug report, you'll realise I wasn't the only person being annoyed that such a trivial bug isn't fixed for such a long time, not to mention the immense confusion that it's caused. Perhaps someone should start renaming JHat as 'asshat' instead!