ahlawat.net

Python difflib - minimum fuss and impressive results

Pranay Ahlawat — Mon, 09 Aug 2010 15:57:15 +0000

So I wanted to write a little library that does a little website monitoring for me and send me an email every time the page changes.

I really did not want to keep wasting time looking at the page - so what would be ideal for me is to have the email send me a diff instead.

Long story short I discovered python difflib - which is ridiculously easy to use:

First I have my two files:

Pranay-Ahlawats-MacBook-Pro:test pranay$ more filea.txt
one
two
three
four
Pranay-Ahlawats-MacBook-Pro:test pranay$ more fileb.txt
one
two
four
five
Pranay-Ahlawats-MacBook-Pro:test pranay$

And here is the python script that generates the diff output:

import difflib

lines_a = open("filea.txt", "r").readlines()
lines_b = open("fileb.txt", "r").readlines()

for line in difflib.ndiff(lines_a, lines_b):
    print line,

Here is the output:

  one
  two
- three
  four
+ five

What I really liked about difflib is a little class called HtmlDiff. With HtmlDiff you can output an html file which gives a color coded diff:

So changing the script to:

import difflib

lines_a = open("filea.txt", "r").readlines()
lines_b = open("fileb.txt", "r").readlines()

html_diff = difflib.HtmlDiff()
print html_diff.make_file(lines_a, lines_b)

Generates:






    
    
    




    
          
          

        
            f 1 one f 1 one
            2 two 2 two
            n 3 three n 
            4 four 3 four
            t t 4 five
        
    
    
          Legends  
          
                       Colors  
                       Added 
                      Changed  
                      Deleted  
                  
              
                       Links  
                      (f)irst change  
                      (n)ext change  
                      (t)op

Which looks quite pretty:

Great attachment for an email.

Setting up amazon EC2 tools on a mac

Pranay Ahlawat — Wed, 04 Aug 2010 18:45:14 +0000

I wanted to setup an amazon VPC and had to use amazon ec2 tools for it.

It was not completely clear what environment variables to set and although the tools themselves are fairly easy to use setting them up so that you can talk to the amazon webservices securely is not completely straight forward. So I thought I will put down the steps.

Step 1. Download the tools
You can download the amazon ec2 tools from the amazon developer website here.

Step 2. Unzip the tools to a directory

Step 3. Generate the x.509 certificate and download the certificate and private key.

This was not totally clear. But you can go to https://aws.amazon.com/account and click on security credentials. Hit the x.509 tab and create a new certificate for the account. You will be asked to download the private key - download it and store it in a safe location on your computer. You can also click on the link there and download the actual crert file - store it on a safe location on your computer.

Step 4. Set up the environment variables.

You will have to set up 4 environment variables for this to work.

Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$ export EC2_HOME=`pwd`
Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$ export EC2_PRIVATE_KEY=/path/to/pk....pem
Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$ export EC2_CERT=/path/to/cert-...pem
Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$ export JAVA_HOME=`/usr/libexec/java_home`

After that you should be able to execute commands just fine.

Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$ ./bin/ec2-describe-instances
RESERVATION	r-7720491c	922634713555	desktoneSecurityGroup
INSTANCE	i-3e4cafff54	ami-c3e40ddfaa	ec2-18adsf4-ada73-116-67.compute-1.amazonaws.com	domU-12-31-39-02-D5-57.compute-1.internal	running	desktone	0		m1.small	2010-08-04T14:44:46+0000	us-east-1b			windows	monitoring-disabled	18.73.116.67	10.248.218.165			ebs					hvm
BLOCKDEVICE	/dev/sda1	vol-22fb894b	2010-08-01T00:04:56.000Z
RESERVATION	r-1777037c	922634713555
INSTANCE	i-36a1195dfc	ami-5b77fdf9c32			running	desktone	0		m1.small	2010-08-04T14:44:46+0000	us-east-1b			windows	monitoring-disabled		17.19.10.41 	vpc-0ae3ceb67	subnet-023cefb6b	ebs					hvm
BLOCKDEVICE	/dev/sda1	vol-6478060d	2010-08-03T21:05:41.000Z
Pranay-Ahlawats-MacBook-Pro:ec2-api-tools-1.3-53907 pranay$

Regenerating open ssh host keys for ubuntu/debian

Pranay Ahlawat — Thu, 08 Jul 2010 20:34:30 +0000

Often after you clone a linux VM with open ssh server installed - the ssh keys generated at the time open ssh server was installed are copied over and you might have problems sshing to that machine.

To regenerate the host keys -

# rm /etc/ssh/ssh_host_*
# dpkg-reconfigure openssh-server
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...

That is it.

Writing a jboss - webservices client using python suds

Pranay Ahlawat — Mon, 28 Jun 2010 03:31:03 +0000

I was experimenting with writing a secured web services client in python and I was surprised how easy it was using python suds.

First write your webservice in java:

@Remote
public interface TestService {
public String echo(String s);

public int add(int a, int b);
}

and the implementation:

@WebService
@Stateless
public class TestServiceImpl implements TestService {
public String echo(String s) {
return s;
}

public int add(int a, int b) {
return a+b;
}
}

Expose it using a webapp and configure security using web.xml:


project-name-war


WS
net.ahlawat.TestServiceImpl


WS
/*



web service
/*


user



BASIC


Known users of the Foobar service
user

and configure it to a jboss login module using jboss-web.xml:



java:/jaas/tws

Basically that is it - you can then code up your python clinet like this:

from suds.client import Client

url = "https://127.0.0.1:8080/webServiceTest-war?wsdl"

client = Client(url, username="user", password="userPassword2010!")

#print client

result = client.service.add(1,2)

print result

result = client.service.echo("Hello world")

print result

And not surprisingly the result:

C:\cygwin\home\deepti\python-suds-0.4>python client.py
3
Hello world

C:\cygwin\home\deepti\python-suds-0.4>

Its amazing how easy this was. I think this is where you think programming in java sucks - putting out a simple client using basic auth would have taken 5 pages of code. Suds is an excellent library to write a quick web service client. Great tool to have in your programmers arsenal.

Mounting NTFS partitions as read-write on Ubuntu Linux

Pranay Ahlawat — Wed, 26 May 2010 17:36:33 +0000

By default when you install ubuntu server and try to mount an ntfs file system you will only be able to mount it as a read only file system.

For example - I have attached a hard disk with a standard windows installation on device 1 (sdb) on my computer.

root@server:/dev$ ls
block            disk      input  loop3   mem                 pktcdvd  ram1   ram2  ram9    sda2  shm       tty    tty14  tty20  tty27  tty33  tty4   tty46  tty52  tty59  tty8     usbdev1.1_ep00  vcs1  vcsa1     zero
bus              ecryptfs  kmem   loop4   net                 port     ram10  ram3  random  sda5  snapshot  tty0   tty15  tty21  tty28  tty34  tty40  tty47  tty53  tty6   tty9     usbdev1.1_ep81  vcs2  vcsa2
cdrom            fd        kmsg   loop5   network_latency     ppp      ram11  ram4  rtc     sdb   sndstat   tty1   tty16  tty22  tty29  tty35  tty41  tty48  tty54  tty60  ttyS0    usbdev1.2_ep00  vcs3  vcsa3
char             full      log    loop6   network_throughput  psaux    ram12  ram5  rtc0    sdb1  sr0       tty10  tty17  tty23  tty3   tty36  tty42  tty49  tty55  tty61  ttyS1    usbdev1.2_ep81  vcs4  vcsa4
console          fuse      loop0  loop7   null                ptmx     ram13  ram6  scd0    sg0   stderr    tty11  tty18  tty24  tty30  tty37  tty43  tty5   tty56  tty62  ttyS2    usbmon0         vcs5  vcsa5
core             hpet      loop1  lp0     oldmem              pts      ram14  ram7  sda     sg1   stdin     tty12  tty19  tty25  tty31  tty38  tty44  tty50  tty57  tty63  ttyS3    usbmon1         vcs6  vcsa6
cpu_dma_latency  initctl   loop2  mapper  parport0            ram0     ram15  ram8  sda1    sg2   stdout    tty13  tty2   tty26  tty32  tty39  tty45  tty51  tty58  tty7   urandom  vcs             vcsa  xconsole
root@server:/dev$ mount
/dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
securityfs on /sys/kernel/security type securityfs (rw)

root@server:~# mkdir /mnt/ntfs
root@server:~# mount /dev/sdb1 /mnt/ntfs/
root@server:~# cd /mnt/ntfs
root@server:/mnt/ntfs# ls
AUTOEXEC.BAT  boot.ini  CONFIG.SYS  Documents and Settings  hiberfil.sys  IO.SYS  MSDOS.SYS  NTDETECT.COM  ntldr  pagefile.sys  Program Files  RECYCLER  sysprep_dat  System Volume Information  WINDOWS
root@server:/mnt/ntfs# touch hello.txt
touch: cannot touch `hello.txt': Read-only file system

To fix this problem all you have to do is install the ntfs-3g drivers on your machine - on ubuntu you can do that by using apt-get install - after that your file system mounts as read-write.

root@server:/mnt# sudo apt-get install ntfs-3g
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  fuse-utils libfuse2 libntfs-3g49
The following NEW packages will be installed:
...
root@server:/mnt# mount /dev/sdb1 /mnt/ntfs/
The disk contains an unclean file system (0, 0).
The file system wasn't safely closed on Windows. Fixing.
root@server:/mnt# cd /mnt/ntfs
root@server:/mnt/ntfs# ls
AUTOEXEC.BAT  boot.ini  CONFIG.SYS  Documents and Settings  hiberfil.sys  IO.SYS  MSDOS.SYS  NTDETECT.COM  ntldr  pagefile.sys  Program Files  RECYCLER  sysprep_dat  System Volume Information  WINDOWS
root@server:/mnt/ntfs# touch hello.txt
root@server:/mnt/ntfs# vim hello.txt
root@server:/mnt/ntfs# exit

Showing hidden files and unix folders in mac finder

Pranay Ahlawat — Fri, 30 Apr 2010 13:38:20 +0000

By default your mac finder will not show you hidden files like “~/.bashrc” and/or “~/.m2″ to show these files you have to:

1. Open terminal and type the following command:

defaults write com.apple.Finder AppleShowAllFiles YES

2. Relaunch finder - by clicking down on finder in the dock while keeping the “Alt/option” key pressed and then selecting relaunch.

Next this time you will see all your hidden files and folders and also core unix folders like /usr and /var.

JBoss JMX extensions to EJB3

Pranay Ahlawat — Sat, 27 Mar 2010 15:34:30 +0000

I don’t particularly like developing on JBoss - I think its bloated and slow. But with EJB3 the platform has become more viable. The turn around time to get a relatively complex application with web services etc. done is great.

One thing I like about JBoss 5.x are the JMX extensions to EJB3. Its really amazing how simple installing a JMX MBean is. Let’s say I have to write a CalculatorService represented by this interface:

package net.ahlawat.cricket;

import javax.ejb.Remote;

import org.jboss.ejb3.annotation.Management;

@Management
@Remote
public interface CalculatorService {
	public double add(double a, double b);
	public double subtract(double a, double b);
	public double performOperation(double a, double b);
	public String getDefaultOperation();
	public void setDefaultOperation(String operation);
}

The interesting annotations here are @Management which tells JBoss deployer that the interface is to be bound as a JMX MBean. The implementation of this bean can be simple pojo - listed here.

package net.ahlawat.cricket;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.math.BigDecimal;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jboss.ejb3.annotation.Service;

@Service(objectName = "ahlawat:service=CalculatorService")
public class CalculatorServiceImpl implements CalculatorService {
	static Log logger = LogFactory.getLog(CalculatorServiceImpl.class);

	Method defaultOperation;

	@Override
	public double add(double a, double b) {
		return a+b;
	}

	@Override
	public String getDefaultOperation() {
		return defaultOperation.getName();
	}

	@Override
	public double performOperation(double a, double b) {
		try {
			BigDecimal bd = (BigDecimal) defaultOperation.invoke(new BigDecimal(a), new BigDecimal(b));
			return bd.doubleValue();
		} catch (IllegalArgumentException e) {
			e.printStackTrace();
		} catch (IllegalAccessException e) {
			e.printStackTrace();
		} catch (InvocationTargetException e) {
			e.printStackTrace();
		}

		return Double.NaN;
	}

	@Override
	public void setDefaultOperation(String operation) {
		try {
			Method m = BigDecimal.class.getMethod(operation, BigDecimal.class);
			logger.info("Setting default operation to " + operation);
			defaultOperation = m;
		} catch (SecurityException e) {
			e.printStackTrace();
		} catch (NoSuchMethodException e) {
			e.printStackTrace();
		}

	}

	@Override
	public double subtract(double a, double b) {
		return a-b;
	}

	//lifecycle methods
	public void create() throws Exception {
		logger.info("Creating service");
	}

	public void start() throws Exception {
		logger.info("Starting service");
		setDefaultOperation("add");
	}
}

The only interesting thing here is @Service - which tells Jboss that this is a singleton bean. And the fact that MBean lifecycle methods like create() and start() are available to be implemented here.

The complete list of life cycle methods are:

   void create() throws Exception;
   void start() throws Exception;
   void stop();
   void destroy();

Once this is jar’d up and deployed to Jboss - you can go to jmx-console and see that the service is visible:

jmx-console

You can also look up the bean using the default JNDI name CalculatorServiceImpl/remote

public class Main {
	public static void main(String[] args) throws Exception {
		InitialContext ctx = new InitialContext();

		CalculatorService cs = (CalculatorService)ctx.lookup("CalculatorServiceImpl/remote");

		System.out.println(cs.add(2.1, 3.2));
	}
}

Here is the jndi.properties:

#jboss JNDI properties
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.provider.url=jnp://localhost:1099
java.naming.factory.url.pkgs=org.jnp.interfaces

And you can also inject the referece to another EJB using the @EJB annotation like this:

@Stateless
public class PlayerServiceImpl implements PlayerService {
	static Log logger = LogFactory.getLog(PlayerServiceImpl.class);

	@PersistenceContext(unitName = "default")
	EntityManager entityManager;

	@EJB
	CalculatorService calculatorService;

	@PostConstruct
	public void gotCalculator() {
		System.out.println(">>>>>>>>>>>>>>>>>> " + calculatorService.add(1.0, 1.0));
	}

...
}

Really small amount of code and guess what - no XML. I really like it.

A simple animated collapsible panel in JQuery

Pranay Ahlawat — Tue, 23 Mar 2010 21:30:45 +0000

How often does one have to write a collapsible panel? If you are a UI developer quite a lot - and how often have you wanted an animated one, if you are a UI developer who wants pizazz, like myself, you want it often.

But then comes the cross browser headaches and the pain of making it work consistently. I have been learning JQuery and it amazes me how painless JQuery can make this stuff.

Here is a simple example:

I start by writing a simple stylesheet (style.css):

body {
	font-family: Arial, "MS Trebuchet", sans-serif;
}

.bd {
	border: 1px solid #999;
}

Next here is my html page:


	
		jquery test
		
		
		

	
	
		Test page for jquery
		
		
			This is some content
		
		
			Click here to show this stuff
			
				This content will be revealed

Thats it - I express my collapsible panel with a marker style called “slider”. I next use a css selector “div.slider > div” to add a border to the content I wish to show and another selector element “div.slider > a” to add click handlers to my links. Since I really want to hide/show the content in the div next to the anchor element I use “this.nextElementSibling” to get to that div and simply call the slideToggle() method. The result is something like this:

Anytime I want a collapsible panel I just have to write a markup with a marker style like this:

Click here to show/hide

Your content here

And the rest is taken care of by jquery. Simple, effective, pretty … love JQuery.

Spring DM deployment problems with schema declaration

Pranay Ahlawat — Wed, 17 Mar 2010 22:43:20 +0000

OK - so I am trying to learn spring DM and finally got vanilla, non spring OSGI bundles deployed on Spring DM server. Things were finally looking good. Next I tried a basic Spring DM configuration like this:

project

Pretty straight forward project setup in STS.

I have a HelloService and a HelloImpl (which is an implementation of this service). My manifest.mf file exposes the net.ahlawat.hello package and looked like this (I am not using my bundle activator for anything):

Manifest-Version: 1.0
Bundle-Version: 1.0.0
Bundle-Name: HelloWorld
Bundle-ManifestVersion: 2
Bundle-SymbolicName: HelloWorld

The spring beans file exposes the osgi service using the osgi namespace and looks like this:

Nothing can possible go wrong. I downloaded spring DM server 2.x and setup the project for running - and got the following exception:

[2010-03-17 18:19:35.207] Thread-5                      Application context creation failure for bundle 'HelloWorld' version '1.0.0'. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#0': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'cacheTarget' of bean class [org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean]: Bean property 'cacheTarget' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1341)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1067)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:540)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:842)
	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:69)
	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:355)
	at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:320)
	at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:132)
	at com.springsource.kernel.agent.dm.ContextPropagatingTaskExecutor$2.run(ContextPropagatingTaskExecutor.java:106)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:637)
Caused by: org.springframework.beans.NotWritablePropertyException: Invalid property 'cacheTarget' of bean class [org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean]: Bean property 'cacheTarget' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
	at org.springframework.beans.BeanWrapperImpl.setPropertyValue(BeanWrapperImpl.java:1012)
	at org.springframework.beans.BeanWrapperImpl.setPropertyValue(BeanWrapperImpl.java:857)
	at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:76)
	at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:58)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1338)
	... 18 common frames omitted

It really was baffling - everything was done inside STS and till I introduced the spring DM config file everything worked well.

Some research pointed out what the problem was - the import declaration needed to be changed to 1.2.xsd instead of 2.0.m1.xsd in the xsi:schemaLocation attribute. Making the change leads to a clean deployment:

[2010-03-17 18:21:36.413] TCP Connection(13)-127.0.0.1  Installing bundle 'HelloWorld' version '1.0.0'.
[2010-03-17 18:21:36.461] TCP Connection(13)-127.0.0.1  Installed bundle 'HelloWorld' version '1.0.0'.
[2010-03-17 18:21:36.476] TCP Connection(13)-127.0.0.1  Starting bundle 'HelloWorld' version '1.0.0'.
[2010-03-17 18:21:36.596] start-signalling-3            Started bundle 'HelloWorld' version '1.0.0'.

I think spring needs better error messages.

A groovy one liner to recursively calculate the size of a directory

Pranay Ahlawat — Wed, 24 Feb 2010 02:22:35 +0000

A groovy one liner to recursively calculate the size of a directory.

C:\Users\deepti>groovy -e "s=0; new File('.').eachFileRecurse {s += it.length()}; println s/1024/1024"
41921.6436758041
C:\Users\deepti>

Pretty useful on windows - which unfortunately does not have a command to achieve this.