Archives
Visitors
  • 46708This month:
  • 263Today:
  • 15Currently online:



LeaseWeb CDN

Automatically provision your bare metal infrastructure

At LeaseWeb we are all about automating delivery processes. Be it for our virtual products or bare metal products. This post shows you one of the many things you can do with our API.

If you have a bare metal server at LeaseWeb I encourage you to login to our customer portal The LeaseWeb Self Service Center at https://secure.leaseweb.com and
In the API section you can manage your api keys for accessing the LeaseWeb API. To read more about what you can do with our API head over to the LeaseWeb Developer Portal

Recently we have published new api calls on our developer portal for customers to manage dhcp leases for their bare metal servers.

These api calls expose our internal dhcp infrastructure, that we use for automation, to our customers as a service.

    GET    /bareMetals/{bareMetalId}/leases                 # list all leases
    POST   /bareMetals/{bareMetalId}/leases                 # create a lease
    DELETE /bareMetals/{bareMetalId}/leases/{macAddress}    # delete a lease

Customers can use it to install operating systems which are not available in the LeaseWeb Self Service Center or if they would like to automatically provision their bare metal infrastructure.

When you use our api to create a dhcp lease you have the possibility to specify the dhcp option 67 Bootfile Name. We chainload the open source ipxe network boot firmware which has http support (read more about ipxe on their website http://ipxe.org/). This means that you can provide a valid http url for dhcp option 67 Bootfile Name that points to a pxe script containing instructions what the the boot loader should do next.

For example: let’s say you own the webserver at webserver.example.com where you have placed the following ipxe script at /boot.ipxe:

    $ curl http://webserver.example.com/boot.ipxe

    #!ipxe
    dhcp
    kernel http://webserver.example.com/archiso/boot/x86_64/vmlinuz archisobasedir=archiso archiso_http_srv=http://webserver.example.com/ ip=:::::eth0:dhcp
    initrd http://webserver.example.com/archiso/boot/x86_64/archiso.img
    boot

You can now create a dhcp lease for your bare metal server using our api:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases -d bootFileName="http://webserver.example.com/boot.i

Obviously replace {bareMetalId} with the id of your bare metal server. To view the dhcp lease that we just created you can use this call:

    $ curl -H 'X-Lsw-Auth: my-api-key' https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases
    
    {
        "_metadata": {
            "limit": 10, 
            "offset": 0, 
            "totalCount": 1
        }, 
        "leases": [
            {
                "ip": "203.0.113.1", 
                "mac": "AA:AA:AA:AA:AA:AA", 
                "options": [
                    // ...
                    {
                        "name": "Bootfile Name", 
                        "optionId": "67", 
                        "policyName": null, 
                        "type": "String", 
                        "userClass": "gPXE", 
                        "value": "http://webserver.example.com/boot.ipxe", 
                        "vendorClass": ""
                    }
                    // ...
                ], 
                "scope": "203.0.113.0"
            }
        ]
    }

Now you have to manually reboot your server or use our api to issue a power cycle:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/reboot

The server will reboot, ask for dhcp lease and eventually read the instructions provided by you in /boot.ipxe which in this example is downloading a kernel and the archlinux live cd which are both served from your web server at `webserver.example.com`.

You should be careful and not forget to remove a dhcp lease when you are done. Otherwise during the next reboot it will boot from the network again.

    $ curl -H 'X-Lsw-Auth: my-api-key' -X DELETE https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases/AA:AA:AA:AA:AA:AA

We automatically remove dhcp leases after 24 hours .

This service allows our customers to implement creative ideas that can automate their bare metal infrastructure.

Example: install arch linux over ssh without kvm

To continue the example I used this service to boot my modified version of the archlinux live cd which includes and starts openssh at boot and includes my public ssh keys. I use this trick to be able to manually install an operating system which is not available through the LeaseWeb Self Service Center.

I don’t need to contact technical support or have kvm on my server. Everything is done remotely over ssh. The modified live image is published on github here https://github.com/nrocco/archiso-sshd.

Clone the repository from github:

    $ git clone https://github.com/nrocco/archiso-sshd.git
    $ cd archiso-sshd

Add your ssh keys to authorized_keys of the root user:

    $ vim airootfs/root/.ssh/authorized_keys

Now build the image (you need to have the archiso package installed).

    $ make build

This might take a while. When done, copy the kernel, initrmfs and other generated files to the document root of your http server:

    $ cp -r work/iso/arch /var/www

Your document root might look like this now:

    $ find /var/www -type f
    /var/www/boot.ipxe
    /var/www/archiso/pkglist.x86_64.txt
    /var/www/archiso/x86_64/airootfs.md5
    /var/www/archiso/x86_64/airootfs.sfs
    /var/www/archiso/boot/x86_64/archiso.img
    /var/www/archiso/boot/x86_64/vmlinuz

That’s it. Now you boot from the network using our service.

Refer to airootfs/root/customize_airootfs.sh and airootfs/root/.ssh/authorized_keys for the specific customatizations.

What can you do with it?

This example is just the tip of the iceberg of possibilities. Let us know your ideas and use cases.

You might use it to boot into your own live image that does an automated installation of the operating system and kicks off the provisioning tool of your choice (chef, ansible, puppet) so your bare metal servers joins your infrastructure that helps supporting your business.

All fully automated.

How to mock MySQLi when unit testing with PHPUnit

PHPUnit is the most used unit testing framework for PHP. Today I wanted to unit test some PHP code that relies on MySQLi. The dilemma is that you either need an actual database and load a fixture or you need to mock the database. As Claudio Lasalla clearly puts:

Unit tests are not “unit” tests if they test things other than the System Under Test (SUT).

And further explains:

Unit tests check on the behavior of units. Think of a class as being a unit. Classes, more often than not, have external dependencies. Tests for such classes should not use their real dependencies because if the dependencies have defects, the tests fail, even though the code inside the class may be perfectly fine.

This theory made total sense to me. That’s why I decided to mock the MySQLi dependency. In this post I will show you just how far I came before I realized this was not going to work out (for me).

The code

The test class, that extends “PHPUnit Framework TestCase”, has an extra method “expectQueries()”. The class looks like this:

<?php

class MySQL_CRUD_API_Test extends PHPUnit_Framework_TestCase
{
	private function expectQueries($queries)
	{
		$mysqli = $this->getMockBuilder('mysqli')
			->setMethods(array('query','real_escape_string'))
			->getMock();
		$mysqli->expects($this->any())
			->method('real_escape_string')
			->will($this->returnCallback(function($str) { return addslashes($str); }));
		$mysqli->expects($this->any())
			->method('query')
			->will($this->returnCallback(function($query) use ($queries) {
				$this->assertTrue(isset($queries[$query]));
				$results = $queries[$query];
				$mysqli_result = $this->getMockBuilder('mysqli_result')
					->setMethods(array('fetch_row','close'))
					->disableOriginalConstructor()
					->getMock();
				$mysqli_result->expects($this->any())
					->method('fetch_row')
					->will($this->returnCallback(function() use ($results) {
						static $r = 0;
						return isset($results[$r])?$results[$r++]:false;
					}));
				return $mysqli_result;
			}));

		return $mysqli;
	}

	public function testSomeSubjectThatUsesMysqli()
	{
		$mysqli = $this->expectQueries(array(
			"SELECT * FROM `table`" =>array(array('1','value1'),array('2','value2'),array('3','value3')),
			"SELECT * FROM `table` LIMIT 2" =>array(array('1','value1'),array('2','value2')),
			// other queries that may be called
		));
		// do something that uses $mysqli
	}
}

The subject-under-test is actually doing something like this:

$result = $mysqli->query("SELECT * FROM `table`");
while ($row = $result->fetch_row()) {
	// do something with the data in $row
}
$result->close();

And in the test it will return the corresponding rows for the queries that you execute. Nice huh?

Not ready

This is a proof-of-concept of a mock of the MySQLi component for PHPUnit. The ‘real_escape_string’ function has a sloppy implementation. It does not (yet) support the much used ‘prepare’, ‘execute’ or ‘fetch_fields’ methods. To give an idea of the completeness, for MySQLi it now support 2/62 functions and properties, for MySQLi Statement 0/28 and for MySQLi Result 2/15. Apart from this incompleteness there is the problem that you may need to support meta information, such as field names and types, to have a fully working mock. If you feel like continuing my work, then feel free to take my code.

Conclusion

Although this was a nice exercise and it may even be the right thing to do in theory, it did not seem to make much sense (to me) in practice. So I gave up on this approach and my current implementation runs all tests against a real database. It loads a database from a SQL file (fixture) in the static ‘setUpBeforeClass()’ function. This may not be so ‘correct’ or ‘clean’ (from a unit testing point of view), but it is much faster to write and easier to maintain.

My question for you: Am I wrong or is the theory wrong? Please tell me using the comments.

MindaPHP now has Memcache support

The PHP framework I am building (MindaPHP) already contains support for MySQL and cURL. Today I have added Memcache support. Memcache can be used for two main purposes in PHP: session storage and application caching. In the framework we only support debugging Memcache for application caching. This is how the debugger looks when the Cache class is used:

mindaphp_cache

The cache is used to store the results from the Bing query. You can try this on: http://maurits.server.nlware.com/hello/bing (click the debugger link in the bottom bar after searching).

Memcache for application caching

Now you can speed up your application using the following commands:

$var = Cache::get($key)
$success = Cache::set($key,$var,$expire=0)
$success = Cache::delete($key)
$success = Cache::add($key,$var,$expire=0)
$success = Cache::replace($key,$var,$expire=0)
$var = Cache::increment($key,$value=1)
$var = Cache::decrement($key,$value=1)

The commands “get” and “set” do retrieval and storage of values in the cache based on the “key” parameter. The commands “add” and “replace” are comparable to “set”, but either fail when the key does (in case of add) or does not (in case of replace) exist. The “increment” and “decrement” commands can be used for counters, but beware that “increment” fails when the key does not exist. This is why you may want to call “add” before you increment.

Memcache for session storage

If you want to use Memcache for session storage in PHP (with any framework), you configured this in “php.ini” with the following statements:

session.save_handler = memcache
session.save_path = "tcp://localhost:11211"

Note that you need the Memcache daemon and the php Memcache extension installed. The following command installs the required software on a Debian based Linux (like Ubuntu):

sudo apt-get install php5-memcache memcached

Have fun accelerating you application!

MySQL-CRUD-API now supports SQL Server 2012

Although the project was initially aimed at only providing support for MySQL, now MS SQL Server 2012 is also supported. MySQL-CRUD-API is a single PHP file that will provide a full REST API for your data structure. With the now added SQLSRV-CRUD-API class you can also connect to a SQL Server database. This only works if the SQLSRV driver is installed in PHP and this is only available for Windows.

sql_server

The SQL Server code relies on the “OFFSET” command, which was added to SQL Server in version 2012. It also allows for UTF-8 character encoding, IMHO a character set any modern database should use. The offset command was added by popular demand (to SQL Server), because the pagination in SQL Server was quite cumbersome, especially when compared to MySQL. Now they are on par again. Also the choice of the SQLSRV driver over the more compatible, but inferior, FreeTDS driver was intentional.

SELECT * FROM [posts] ORDER BY [published] DESC OFFSET 0 ROWS FETCH NEXT 10 ROWS ONLY;

The response on this project has been mostly positive. There are people arguing that it is bad to expose the data structure of your database. My counter argument is that if you do proper database administration (data management) this is not true. And in that case you do not need all that boilerplate code that APIs generally consist of as it can all be automated. That is this project’s philosophy.

https://github.com/mevdschee/mysql-crud-api

Check out the code on Github and tell me what you think. Use the comments for feedback.

MindaPHP now has RESTful API support

When building applications today you need to follow cool new architectures like “Microservice design” and “API-first”. APIs play an increasingly important role in applications today. Together with database abstraction layers they bring the database technology further from the business logic than ever.

In line with the “Ease of learning” vision for MindaPHP, I decided to add a minimal RESTful API client in the form of a cURL wrapper. It has full integration with the debugger as you can see below:

mindaphp_api

The wrapper class hardly influences the performance of cURL when the debugger is disabled. When the debugger is enabled the performance and the memory usage may be affected, but this gives you a great deal of control as you can see in the image above. In the example below you see how you can use the cURL wrapper to call the Bing search engine and extract the first 10 links for a search query.

<?php
$query = isset($_POST['q'])?$_POST['q']:'';
$results = array();

if ($query) {
    if (Curl::call('GET','http://www.bing.com/search',array('q'=>$query),$result)==200) {
        
        $dom = new DOMDocument();
        @$dom->loadHTML($result);
        
        $xpath = new DOMXpath($dom);
        $elements = $xpath->query('//ol["b_results"]/li[@class="b_algo"]//h2/a');

        foreach ($elements as $element) {
            $text = $element->nodeValue;
            $link = $element->getAttribute("href");
            $results[] = compact('text','link');
        }
    }
}

For more information check out Github or the demo on one of the links below:

 

Code: https://github.com/mevdschee/MindaPHP
Demo: http://maurits.server.nlware.com/hello/bing