mandag den 31. august 2009

KHAN's ladder to programming excellence!

In Maxwell Gladwells book 'Outliers' he mentions that in order to excel in anything you have to put atleast 10000 hours in to it, which is the equivalent of 6 hours each day in 5 years. 356 * 5 * 6 = 10680, well almost.

Here I present the ladder of programming excellenceu as perceived by me :)
Legend: HoP = Hours of Programming

KHAN's ladder to programming excellence!

I reckoned I hit the programming experience mark of 5000 hours 5 years ago.

Programming in Basic and DOS before my education = 1000 HoP

beginning programmer - 1000 HoP

Programming in high school Assembler, electronics, VB, Java, ASP, HTML: 1000 HoP

greenhorne programmer - 2000 HoP

Higher education as datamatician(mostly Java) 2 1/4 years, with an average of 140 hours a month: 4000 HoP

- at this point I broke the 5000 HoP

jorneyman programmer - 4000 HoP

After the bulk of work with non-contractor jobs 3 years, with an average of 180 hours a month: 6500 HoP
This involved mostly Java with J2EE, but I began on more complex tasks like configuring WebSphere servers, designing parts of enterprise wide technical solutions. Here I picked up many new complex skills, Eclipse RCP, Version control, functional programming, unit testing, server administration etc.
- here i accelarated because I had to break into many new concepts and frameworks, working after hours and reading a lot while experimenting.

veteran programmer - 8000 HoP

At this point I started my own company and began working as a freelancer. I had no products, no particular approach or plan, only my skill as a programmer. Though I discovered I also needed economical and business skills.
In this work I now have 2 years of experience, my pace has slowed down and I would now estimate to having 170 hours per month of IT experience: 4000 HoP, making a total of 16500 HoP.

expert programmer - 16000 HoP

master for me is 5 years away

master programmer - 32000 HoP

I might achieve programming guru before retirement

programming guru - 64000 HoP

//TODO insert lots of effort

deity programmer -
128000 HoP

I think of this scale as a logarithmic one. Each step requires the double amount of effort and I find this to be quite true. At a time while being greenhorne I experienced discovering new things more often, towards becoming a guru I can only guess to what I learn.

mandag den 13. april 2009

Automatically testing advanced web applications (revisited)

More than one and a half years ago I first posted a blog article 'high automatic test coverage of web applications'. While working with this and later projects I realized that in order to have complete coverage and a more dynamic testing environment I had to take it a step further and therefore started inventing my own test framework, described in the article 'full testing with javascript'.

Soon thereafter I discovered the Selenium Core framework which was still rather immature. Since it functions similar to the framework I was working on, I gave it a go and never looked back.

Testing priorities in a web application
When testing you have to first plan what areas need the most testing. Test driven development requires a lot of focus on unit testing and functional testing. I will not discuss unit testing here, since it depends heavily on the platform of the web application.
Aside from unit testing, functional testing is the most important testing for web applications, because it can be used to directly measure progress and prohibits regression.
On a second place comes integration testing, because alot of problems tend to occur in communication with other systems.
In third comes performance testing. This tends to be the focus of automatic tests, but that is an ancient view imo. Performance testing is important, but it's rather easy once functional testing has been done, because you can leverage the functional tests and scale them to performance tests.
On a fourth place comes installation testing. Redeploying a web application is important, both for testing environments, but also for reestablishing an environment.

Functional testing
this is exactly what Selenium is for. It's fast and works for Python, Java and several other popular platforms. I would go for Python because the interactive compiler is killer for writing test cases. Java is not a bad alternative with Seleniums excellent integration with JUnit.
You should not need any other tool for this task.

Performance testing
For performance testing you could use HTTPUnit, which also integrates well with JUnit. It's a rather simple, but powerful framework for controlling all parameters of requests and monitoring the responses.
Functional test cases written for functional testing using Selenium could be rewritten easily and scaled to do performance testing.

Integration testing
This depends on what technology is used for communication, I'll only present the most common one here. Integration in web applications are typically over Web Services, therefore testing could be done with a framework for easily creating web clients, like Axis 2 and using JUnit to execute it, but HTTPUnit is also viable.

Installation testing
Installation testing involves two parts. Scripts to deploy the application and verification of the function.
Scripts would typically depend on the platform of the web application. For WebLogic or WebSphere, you would use ant targets available from the vendor. How you manage to deploy the application automatically is rather irrelevant, since you needn't verify anything during the process.
The second part of verifying function after installation is rather easy, since it's just a matter of running the functional tests.

Conclusion
Using a few open source tools is an easy, fast and effective way to test advanced Web Applications with lots of javascript and complexity.
Standardizing the executing can be achieved using a common executing framework like JUnit.

torsdag den 26. februar 2009

KHANDI the temporal database manager

Introduction
On contract for a customer I had to develop a temporal database manager.

The problem is very simple. You have an entity that exists from a data to another date and is dependent on another entity to exist in the same period.

Consider the example below, this is a valid dependency, because, the dependency exists in atleast the same period as the entity.

Entity
I-------------I

Dependency Entity
I-------------------I

Legend:
lines represent the existance of the entities

Considering you have to edit an entity, you have two basic changes you can make.
Edit attributes of an entity, creating a new record from that point, or
Edit the period in which the entity exists

Editing attributes
Entity
I-------I (attribute changed on this date)
I----I

Dependency Entity
I-------------------I

- dependency is still valid

Editing period
Entity
I-------I (end date of entity is sooner)

Dependency Entity
I-------------------I

- dependency is still valid

Now consider the situation where a user makes an invalid edit.

Invalid edit
Entity
I-------------------I (end date of entity is later)

Dependency Entity
I-------------------I

- dependency is NOT valid

The main point of this last example is that there's a lot of checks that need to be made before editing something.

The database I've made with the customer consists of multiple tables 20+, that are all interdependent.

The component I've written is an ORM with temporal management built-in and transparently handles these relation restrictions as defined in the business logic.

Although very clean, it poses some challenges in a UI. If you need to edit several types of entities in the same submit, you could end up having dependencies, that may put the user in a deadlock. Therefore, one should consider if temporal management of an entitty is really necessary to store temporally.

KHANDI Component short:
temporal ORM manager
compliable with java 1.4
Connects with SAS and MySQL out of the box, easily integratable with other dbs

Write me at kennethhn@gmail.com, if interested in knowing more.

mandag den 12. januar 2009

TOMCAT UTF-8 gotcha

If you are having problem with your encoding running on a platform with other than UTF-8 as a default, try going through your code and make sure request.getParameter() is not called before setting request.setCharEncoding().

example:

request.setCharacterEncoding("UTF-8");
String adUser = request.getParameter(TAM_AD_USER);


If you have no hook for it, consider making a filter: http://cagan327.blogspot.com/2006/05/utf-8-encoding-fix-tomcat-jsp-etc.html

mandag den 8. september 2008

Full JS applications

Recently I asked my network for novel ideas for small web applications or sites.

My accountant mentioned that he had a hard time calculating distance travelled for some of his customers. He needs to calculate the specific distance for any trip made in a private car for a refund from the danish IRS.

For each trip made by an employee in one of the companies he used to go to a GeoCoding site like http://www.maps.google.com. Plot in 1 trip with From and To addresses, take the distance and put it into a spreadsheet. And so his days went. He made sure to mention to me that it is a problem for many smaller companies.

Using Google's public GeoCoding services I managed to put together an application run entirely by javascript. It is available at http://www.kennethhn.dk/map.html, it is in danish, until I bother using JS I18N. To give it a go, I included a demo, click 'tryk her for hjælp og demo', in the appeared box click 'klikke her'. Then at the bottom of the page click 'beregn alle'. This will calculate all the inputs with the rate of DKK/KM.

Then to print the results click 'print'.

The tricky thing here is to transfer all the data. What I did was serialize all the data in JSON and put it as a parameter for the print page.

Aside from some escape issues, this transfer works perfectly. It might aswell have been a server request, but this way, you don't need all that performance reducing Serverside rendering..

torsdag den 6. marts 2008

cross site javascript

This is an ajax widget I wrote, while learning about cross site scripting (xss). Read more about the Webspace wormhole here.

The idea is that on any page on any browser I should be able to embed this script and it will share content. Much like Google maps and a ton of other useful stuff from Yahoo and Google.

On the page itself the script only needs a small amount of information, like the id of the div it will use for the graphics and what 'tag' it will subscribe to.

The transport it uses is not the typical Ajax.Request from the prototype lbirary, because this uses solely XMLHttpRequest, which for security reasons default in browsers is not cross site.

Instead I had to write a transport of my own. The basic work-around is to embed a script element in the dom created by a dynamic page. This way, when the script is loaded it can inject data into the javascript execution, just like a plain script file. I implemented a reusable class for it, thusly:

var Transport = Class.create({
initialize: function(url, jsonCallback) {
this.url = url;
this.jsonCallback = jsonCallback;
},
activate: function() {
this._cleanUpScript(this.url);
var script_object = new Element('script', {src: this.url, type: 'text/javascript'});
var head=document.getElementsByTagName('head')[0] ;
head.appendChild(script_object);
this._xssScriptLoad();
},
_xssScriptLoad: function() {
var _this = this;
if(typeof(wh_xss_result) != 'undefined') {
this.jsonCallback(wh_xss_result);
void(wh_xss_result);
} else {
setTimeout(function(){_this._xssScriptLoad();}, 100);
}
},
_cleanUpScript: function(url) {
$A(document.getElementsByTagName('script')).each(function(el){
if(el.src == url){
el.remove();
};
});
}
});

The only 'gotcha' here is that I have to know the name of the variable that the result is stored in. This could be set on the serverside if I pass it as a parmater if I want the serverside to be more reusable, but for the sake of my little wormhole, I hardcoded the variable on both sides.

Here is an example usage of the transport:

var url = 'http://www.kennethhn.dk/webhole/get.php?tag=science';
var t = Transport(url, function(json){alert("json object = " + json});
t.activate();

In this case, the url is a reference to a php script on my site that returns a url for content submitted for the tag.

The second argument is a callback function, that will be called with the resulting json object.

The result might look like this:

var wh_xss_result = {
message: 'fetched url for tag science',
result: 'http://tbn0.google.com/images?q=tbn:-U28WILXKSXv5M:http://www.familycourtchronicles.com/philosophy/wormhole/wormhole-diagram.jpg'};

The transport class will then embed the result as a script and set a timeout event to repeat until the browser has loaded the script. Loading of the script is determined by the variable wh_xss_result, which will be 'undefined' until executed.

Xss can be very useful, but also very dangerous, just be vary of what scripts you choose to embed.

Update 13/04/2009: removed the WebSpace wormhole from this page.

tirsdag den 26. februar 2008

Full testing with JavaScript

In my efforts to achieve a full and robust test of web applications I have run into several difficulties.
1. simulating different browsers
2. keeping test scripts simple
3. fragile tests

Simulating different browser


Initially I achieved this using the Canoo Webtest. Though powerful, this integration fo different third party frameworks has shown not to be able to handle JavaScript very well. This means that if you have no control over the javascript in an application, the test becomes very fragile.

Instead I concluded that the best way to simulate the different browsers is not to actually simulate them. Instead it is very efficient to inject JavaScript along with your tests into the page that you wish to test and thereby testing on the executed page.

Several issues comes with this. Firstly, the security in the browser won't allow you to execute JS on another page, if they are not from the same domain, so you have to execute the test from the domain itself. For inhouse testing this is not a problem, but involves some more deployment.

keeping test script simple


This is really up to the implementation of the web application. If its based on poor standard, test script become increasingly difficult. If for example id's are duplicated or tags are broken, the traversal of the DOM becomes difficult. The best solution to this is to stick to strict xhtml and keep your styles in CSS while maintaining ids on all divs.

A second worry is that the script becomes complicated with lots of hacks and too much code, but this is really up to the framework being used. The better the framework and web application coexist, the easier and cleaner the scripts become.

fragile tests


With a low sense of 'standards' in a project, the test become very fragile. The team must agree upon how to use divs and lists etc. Once referring to an id of a div or anchor at a specific position, that has moved or is renamed, the script might have to be rewritten. The framework should help avoid too specific referenced. Whenever referencing an element in the DOM, reference it as lenient as possible.

here is an example of how the framework might be implemented

MyTestFramework.clickLink = function(text) {
$$('a').each(function(el){if(el.innerHTML==text){window.location=el.href}});
}

MyTestFramework.clickLink('login');


I'll be working on this and post when I have a functioning framework. I'll proably pull in some JSUNit library and use that for a backbone for TestCase and TestRunner.