https://github.com/LeaseWeb/LswDoctrinePdoDblib

If you are trying to use Doctrine 2 (for Symfony 2) to connect to MSSQL server from a Linux machine, you might encounter the same problems we did. To be able to connect to MSSQL from a Ubuntu Linux server you will have to install “php5-sybase” (Sybase / MS SQL Server module for php5).

$ sudo apt-get install freetds-bin php5-sybase

Add the following lines to “/etc/freetds/freetds.conf”, and make sure you set it to talk the right MSSQL protocol. The TDS protocol version should be 8.0 for MSSQL server 2005 and up, which may be confusing. Also replace “{mssql-server-ip-address}” with your MSSQL servers IP address (without the brackets).

    [mssql_freetds]
        host = {mssql-server-ip-address}
        port = 1433
        tds version = 8.0
        client charset = UTF-8
        text size = 20971520

If you only connect to one server, you can also set the global TDS protocol version to 8.0 like this:

     [global]
         # TDS protocol version
         tds version = 8.0

Now test that your freetds is working by using the command line client. You have to replace {username} and {password} with your username and password (without the brackets):

$ tsql -S mssql_freetds -U {username}
Password: {password}
locale is "en_US.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> quit

If you specify in “app/config/parameters.ini” that you want to use the “pdo_dblib” driver like this:

database_driver="pdo_dblib"

You will see the error:

[Doctrine\DBAL\DBALException]
The given 'driver' pdo_dblib is unknown, Doctrine currently supports only the following drivers:
pdo_mysql, pdo_sqlite, pdo_pgsql, pdo_oci, oci8, ibm_db2, pdo_ibm, pdo_sqlsrv

This is why you should install our LswDoctrinePdoDblib bundle, and following the instructions, you should comment out the “driver” and add a “driver_class” like this:

    # Doctrine Configuration
    doctrine:
        dbal:
            #driver:   %database_driver%
            driver_class:   Lsw\DoctrinePdoDblib\Doctrine\DBAL\Driver\PDODblib\Driver

After that you should be able to run this command without any errors:

$ php app/console doctrine:database:create

Now do not expect everything to work flawlessly, since MSSQL is not officially supported, and Doctrine is very picky in what it does and does not accept as database structure. However, this should get you started. Post your experience and questions in the comments. Good luck!

Habit 1: Be Proactive

Take initiative in life by realizing that your decisions (and how they align with life’s principles) are the primary determining factor for effectiveness in your life. Take responsibility for your choices and the consequences that follow.

If you choose Symfony2, follow the best practices and believes rigorously, also read everything you can about it (and join meetups). Choose Symfony DI, choose Symfony security, choose Doctrine, choose APC, choose Symfony coding guidelines, (like PSR-0, PSR-1 and PSR-2 standards) and most important: choose other people’s bundles. Choosing Symfony’s best practices can be hard sometimes, because Symfony leaves many choices open and dictates less choices than other frameworks do (e.g. Rails).

Habit 2: Begin with the End in Mind

Self-discover and clarify your deeply important character values and life goals. Envision the ideal characteristics for each of your various roles and relationships in life. Create a mission statement.

What do you want to achieve, set goals, choose technologies and strategies without limiting you to what is available. This seems like a contradiction with the previous one, but I strongly believe you should find a balance: Choose the Symfony solution or someone else’s bundle (maybe even do a pull request to add any missing functionality) if you can, but do create your own bundle if you need to, do not compromise before you even started. If you have a clear and unique goal, you can craft your bundle to fulfill it.

Habit 3: Put First Things First

Prioritize, plan, and execute your week’s tasks based on importance rather than urgency. Evaluate whether your efforts exemplify your desired character values, propel you toward goals, and enrich the roles and relationships that were elaborated in Habit 2.

Have the guts to refactor when you need to and believe me: sometimes you need to. Also, make sure you do not accept bug reports unless you really have to. Often enough these are just feature requests that have urgency, but no importance. Following SCRUM, which has the same concept (work in order of importance), is not sufficient to achieve this habit, but may help you and give you a way of measuring progress.

Habit 4: Think Win-Win

Genuinely strive for mutually beneficial solutions or agreements in your relationships. Value and respect people by understanding a “win” for all is ultimately a better long-term resolution than if only one person in the situation had gotten his way.

Bundles do not only compete with each other, they also complete each other. Ownership of bundles can drive competition, which motivates, but open source will allow people to copy solutions from each other, which makes sure no effort has ever to be in vain. Even if a bundle is less popular it does not mean it is less important. Maybe it provided important functionality that was merged into the more popular bundle and this makes the author an major contributor.

Habit 5: Seek First to Understand, Then to be Understood

Use empathic listening to be genuinely influenced by a person, which compels them to reciprocate the listening and take an open mind to being influenced by you. This creates an atmosphere of caring, and positive problem solving.

Do not write your own bundle or criticize other people’s bundles before the understand what the available bundles offer and actually do. After you did that try to contribute in a positive manner, not by criticizing in comments, but by being positive and providing pull requests. But also when being criticized on your bundle, be positive and try to understand what people are actually saying and what their needs are. Negative comments may hold valuable information.

Habit 6: Synergize

Combine the strengths of people through positive teamwork, so as to achieve goals no one person could have done alone.

We can not create bundles alone, so let others help you and do help other people. Whether it is by providing support, contributing code or reporting bugs. All roles matter and all efforts should be appreciated with a positive attitude. I do not pretend to know why the Open Source paradigm works, but I do know it is all about “synergy” and that with Open source we can achieve goals that no single person could have achieved alone.

Habit 7: Sharpen the Saw

Balance and renew your resources, energy, and health to create a sustainable, long-term, effective lifestyle. It primarily emphasizes exercise for physical renewal, prayer (meditation, yoga, etc.) and good reading for mental renewal. It also mentions service to society for spiritual renewal.

The most important advice I heard about this point is: to sharpen the saw, be humble and start sawing. The reason for this is that a lack of experience and the Dunning–Kruger effect will make you grossly overestimated your performance and ability. It will make you become a victim of the seductive and destructive Not-Invented-Here syndrome. Sharpen the saw is about continuous improvement and especially in IT it is important to keep learning new things, because the technology moves fast. So do not hesitate and start creating Symfony2 bundles right now!

Check out our open sourced Symfony bundles on Github!

The LeaseWeb labs blog is getting more and more popular every day. This month we had more than 300 reads per day from our 7000 visitors in the month of April. Looking at the previous months we were seeing 6500 visitors in March and 5300 for February.

Please keep visiting our blog and tell us what you would like to write us about in the comments. Also please like us on Facebook, share our links through Twitter and other social media.

Thank you for visiting!

Given n men and n women, where each person has ranked all members of the opposite sex with a unique number between 1 and n in order of preference, marry the men and women together such that there are no two people of opposite sex who would both rather have each other than their current partners. If there are no such people, all the marriages are “stable”. – source: Wikipedia

Fortunately, it is already proven that all marriages can be stable, the proof and the algorithm for such stable marriage problems has been implemented in many languages on rosettacode.org.

Since this blog is about PHP, and this was the language missing from the implementations, I decided to port the Python implementation. So this one is for you:

    $guyprefers = array(
     'abe'  => array('abi', 'eve', 'cath', 'ivy', 'jan', 'dee', 'fay', 'bea', 'hope', 'gay'),
     'bob'  => array('cath', 'hope', 'abi', 'dee', 'eve', 'fay', 'bea', 'jan', 'ivy', 'gay'),
     'col'  => array('hope', 'eve', 'abi', 'dee', 'bea', 'fay', 'ivy', 'gay', 'cath', 'jan'),
     'dan'  => array('ivy', 'fay', 'dee', 'gay', 'hope', 'eve', 'jan', 'bea', 'cath', 'abi'),
     'ed'   => array('jan', 'dee', 'bea', 'cath', 'fay', 'eve', 'abi', 'ivy', 'hope', 'gay'),
     'fred' => array('bea', 'abi', 'dee', 'gay', 'eve', 'ivy', 'cath', 'jan', 'hope', 'fay'),
     'gav'  => array('gay', 'eve', 'ivy', 'bea', 'cath', 'abi', 'dee', 'hope', 'jan', 'fay'),
     'hal'  => array('abi', 'eve', 'hope', 'fay', 'ivy', 'cath', 'jan', 'bea', 'gay', 'dee'),
     'ian'  => array('hope', 'cath', 'dee', 'gay', 'bea', 'abi', 'fay', 'ivy', 'jan', 'eve'),
     'jon'  => array('abi', 'fay', 'jan', 'gay', 'eve', 'bea', 'dee', 'cath', 'ivy', 'hope'));
    $galprefers = array(
     'abi'  => array('bob', 'fred', 'jon', 'gav', 'ian', 'abe', 'dan', 'ed', 'col', 'hal'),
     'bea'  => array('bob', 'abe', 'col', 'fred', 'gav', 'dan', 'ian', 'ed', 'jon', 'hal'),
     'cath' => array('fred', 'bob', 'ed', 'gav', 'hal', 'col', 'ian', 'abe', 'dan', 'jon'),
     'dee'  => array('fred', 'jon', 'col', 'abe', 'ian', 'hal', 'gav', 'dan', 'bob', 'ed'),
     'eve'  => array('jon', 'hal', 'fred', 'dan', 'abe', 'gav', 'col', 'ed', 'ian', 'bob'),
     'fay'  => array('bob', 'abe', 'ed', 'ian', 'jon', 'dan', 'fred', 'gav', 'col', 'hal'),
     'gay'  => array('jon', 'gav', 'hal', 'fred', 'bob', 'abe', 'col', 'ed', 'dan', 'ian'),
     'hope' => array('gav', 'jon', 'bob', 'abe', 'ian', 'dan', 'hal', 'ed', 'col', 'fred'),
     'ivy'  => array('ian', 'col', 'hal', 'gav', 'fred', 'bob', 'abe', 'ed', 'jon', 'dan'),
     'jan'  => array('ed', 'hal', 'gav', 'abe', 'bob', 'jon', 'col', 'ian', 'fred', 'dan'));

    function check($engaged)
    {
        global $guyprefers, $galprefers;
        $inverseengaged = array_combine(array_values($engaged),array_keys($engaged));
        foreach ($engaged as $she => $he)
        {
            $shelikes = $galprefers[$she];
            $shelikesbetter = array_slice($shelikes,0,array_search($he,$shelikes));
            $helikes = $guyprefers[$he];
            $helikesbetter = array_slice($helikes,0,array_search($she,$helikes));

            foreach ($shelikesbetter as $guy)
            {
                $guysgirl = $inverseengaged[$guy];
                $guylikes = $guyprefers[$guy];
                if (array_search($guysgirl,$guylikes) > array_search($she,$guylikes))
                {
                    printf("%s and %s like each other better than ".
                           "their present partners: %s and %s, respectively\n",
                           $she, $guy, $he, $guysgirl);
                    return false;
                }
            }
            foreach ($helikesbetter as $gal)
            {
                $girlsguy = $engaged[$gal];
                $gallikes = $galprefers[$gal];
                if (array_search($girlsguy,$gallikes) > array_search($he,$gallikes))
                {
                    printf("%s and %s like each other better than ".
                           "their present partners: %s and %s, respectively\n",
                           $he, $gal, $she, $girlsguy);
                    return false;
                }
            }
        }
        return true;
    }

    function matchmaker($engaged)
    {
        global $guyprefers, $galprefers;

        $guys = array_keys($guyprefers);
        sort($guys);
        $gals = array_keys($galprefers);
        sort($gals);

        $guysfree = $guys;
        $engaged  = array();
        $guyprefers2 = json_decode(json_encode($guyprefers),true);
        $galprefers2 = json_decode(json_encode($galprefers),true);

        while (count($guysfree))
        {
            $guy = array_shift($guysfree);
            $guyslist = &$guyprefers2[$guy];
            $gal = array_shift($guyslist);
            $fiance = isset($engaged[$gal])?$engaged[$gal]:false;
            if (!$fiance)
            {
                // She's free
                $engaged[$gal] = $guy;
                printf("- %s and %s\n", $guy, $gal);
            }
            else
            {
                // The bounder proposes to an engaged lass!
                $galslist = $galprefers2[$gal];
                if (array_search($fiance,$galslist) > array_search($guy,$galslist))
                {
                    // She prefers new guy
                    $engaged[$gal] = $guy;
                    printf("- %s dumped %s for %s\n", $gal, $fiance, $guy);
                    if ($guyprefers2[$fiance])
                    {
                        // Ex has more girls to try
                        $guysfree[] = $fiance;
                    }
                }
                else
                {
                    // She is faithful to old fiance
                    if (count($guyslist))
                    {
                        // Look again
                        $guysfree[] = $guy;
                    }
                }
            }

        }

        return $engaged;
    }

    $engaged  = array();
    print("Engagements:\n");
    $engaged = matchmaker($engaged);

    print("Couples:\n");
    ksort($engaged);
    foreach ($engaged as $guy => $gal)
    {
        printf("- %s is engaged to %s\n", $guy, $gal);
    }
    $success = check($engaged);
    print("Engagement stability check ".($success?'PASSED':'FAILED')."\n");

    print("Swapping two fiances to introduce an error\n");
    $gals = array_keys($galprefers);
    ksort($gals);
    list($engaged[$gals[0]], $engaged[$gals[1]]) = array($engaged[$gals[1]], $engaged[$gals[0]]);
    print("Couples:\n");
    ksort($engaged);
    foreach ($engaged as $guy => $gal)
    {
        printf("- %s is engaged to %s\n", $guy, $gal);
    }
    print("Engagement stability check ".(check($engaged)?'PASSED':'FAILED')."\n");

Expected output:

Engagements:
- abe and abi
- bob and cath
- col and hope
- dan and ivy
- ed and jan
- fred and bea
- gav and gay
- hope dumped col for ian
- abi dumped abe for jon
- hal and eve
- col and dee
- ivy dumped dan for abe
- dan and fay
Couples:
- abi is engaged to jon
- bea is engaged to fred
- cath is engaged to bob
- dee is engaged to col
- eve is engaged to hal
- fay is engaged to dan
- gay is engaged to gav
- hope is engaged to ian
- ivy is engaged to abe
- jan is engaged to ed
Engagement stability check PASSED
Swapping two fiances to introduce an error
Couples:
- abi is engaged to fred
- bea is engaged to jon
- cath is engaged to bob
- dee is engaged to col
- eve is engaged to hal
- fay is engaged to dan
- gay is engaged to gav
- hope is engaged to ian
- ivy is engaged to abe
- jan is engaged to ed
fred and bea like each other better than their present partners: abi and jon, respectively
Engagement stability check FAILED

Unfortunately Symfony2 is not that strict everywhere. This lack of strictness allows a group of programmers to argue over and mess up a project. One of the discussions could be about naming routes. Symfony1 had a really nice option for “default routing”, allowing you to specify a single rule called “default” that had url: “/:module/:action/*”, but unfortunately that feature is not present in Symfony2. In Symfony2 we can either define routes individually or we can generate them using the CRUD generator (see my previous post). This is not a bad option as I will show below.

Naive routing

Routing to specific actions in controllers in Symfony2 can be put in a “routing.yml” file as described in the Routing section of the Symfony2 manual.

# app/config/routing.yml
product:
    pattern:   /product/
    defaults:  { _controller: AcmeDemoBundle:Product:index }

product_edit:
    pattern:  /product/edit/{id}
    defaults: { _controller: AcmeDemoBundle:Product:edit }

It is easy to understand that this way the file becomes large and unless people agree on a very strict naming policy it needs to be consulted every time you want to make a link from a view or controller to any action.

CRUD generated routing

After using the CRUD generator to generate scaffolding for an Entity called “Product”,

maurits@pc:~/project$ app/console generate:doctrine:entity --entity=AcmeDemoBundle:Product
...
Configuration format (yml, xml, php, or annotation) [annotation]:
...
maurits@pc:~/project$ app/console generate:doctrine:crud --entity=AcmeDemoBundle:Product
...
Do you want to generate the "write" actions [no]? yes
...
Configuration format (yml, xml, php, or annotation) [annotation]:
...
Routes prefix [/product]:
...

I edited my routing to include this rule:

# app/config/routing.yml
demo_bundle:
    resource: "@AcmeDemoBundle/Controller/"
    type: annotation
    prefix: /

This is the only routing entry you need per bundle when using (CRUD generated) routing annotation in your controllers. The “@Route” and “@Template” annotations can be seen below:

/**
 * Product Controller
 *
 * @Route("/product")
 */
class ProductController extends Controller
{
    /**
     * Lists all Product entities.
     *
     * @Route("/", name="product")
     * @Template()
     */
    public function indexAction()
    {
        $em = $this->getDoctrine()->getManager();
        $entities = $em->getRepository('AcmeDemoBundle:Product')->findAll();
        return array('entities' => $entities);
    }

    // ...
}

Your routes will look like:

maurits@pc:~/project$ app/console router:debug
[router] Current routes
Name                                                   Method Pattern
...                                                    ...    ...
product                                                ANY    /product/
product_show                                           ANY    /product/{id}/show
product_new                                            ANY    /product/new
product_create                                         POST   /product/create
product_edit                                           ANY    /product/{id}/edit
product_update                                         POST   /product/{id}/update
product_delete                                         POST   /product/{id}/delete

Default Routing in Symfony2

But would it not be prettier when the only routing entry you needed (per bundle) was this one?

# app/config/routing.yml
demo_bundle:
    resource: "@AcmeDemoBundle"
    type: default
    prefix: /

And your code would NOT contain “@Route” and “@Template” annotations and look like:

/**
 * Product Controller
 */
class ProductController extends Controller
{
    /**
     * Lists all Product entities.
     */
    public function indexAction()
    {
        $em = $this->getDoctrine()->getManager();
        $entities = $em->getRepository('AcmeDemoBundle:Product')->findAll();
        return array('entities' => $entities);
    }

    // ...
}

Your routes could look like this:

maurits@pc:~/project$ app/console router:debug
[router] Current routes
Name                                                   Method Pattern
...                                                    ...    ...
acme_demo.product.index                                ANY    /product/index.{_format}
acme_demo.product.show                                 ANY    /product/show/{id}.{_format}
acme_demo.product.new                                  ANY    /product/new.{_format}
acme_demo.product.edit                                 ANY    /product/edit/{id}.{_format}
acme_demo.product.delete                               ANY    /product/delete/{id}.{_format}

To achieve this we needed to combine the “new” & “create” and “edit” & “update” actions and also remove
the “$request” argument from the “delete” action and replace it by a simple “$request = $this->getRequest();” call. So this code:

    /**
     * Displays a form to edit an existing Product entity.
     *
     * @Route("/{id}/edit", name="product_edit")
     * @Template()
     */
    public function editAction($id)
    {
        $em = $this->getDoctrine()->getManager();

        $entity = $em->getRepository('AcmeDemoBundle:Product')->find($id);

        if (!$entity) {
            throw $this->createNotFoundException('Unable to find Product entity.');
        }

        $editForm = $this->createForm(new ProductType(), $entity);
        $deleteForm = $this->createDeleteForm($id);

        return array(
            'entity'      => $entity,
            'edit_form'   => $editForm->createView(),
            'delete_form' => $deleteForm->createView(),
        );
    }

    /**
     * Edits an existing Product entity.
     *
     * @Route("/{id}/update", name="product_update")
     * @Method("POST")
     * @Template("AcmeDemoBundle:Product:edit.html.twig")
     */
    public function updateAction(Request $request, $id)
    {
        $em = $this->getDoctrine()->getManager();

        $entity = $em->getRepository('AcmeDemoBundle:Product')->find($id);

        if (!$entity) {
            throw $this->createNotFoundException('Unable to find Product entity.');
        }

        $deleteForm = $this->createDeleteForm($id);
        $editForm = $this->createForm(new ProductType(), $entity);
        $editForm->bind($request);

        if ($editForm->isValid()) {
            $em->persist($entity);
            $em->flush();

            return $this->redirect($this->generateUrl('product_edit', array('id' => $id)));
        }

        return array(
            'entity'      => $entity,
            'edit_form'   => $editForm->createView(),
            'delete_form' => $deleteForm->createView(),
        );
    }

has to be changed reduced into this:

    /**
     * Edits an existing Product entity.
     */
    public function editAction($id)
    {
        $em = $this->getDoctrine()->getManager();

        $entity = $em->getRepository('AcmeDemoBundle:Product')->find($id);

        if (!$entity) {
            throw $this->createNotFoundException('Unable to find Product entity.');
        }

        $editForm = $this->createForm(new ProductType(), $entity);
        $deleteForm = $this->createDeleteForm($id);

        $request = $this->getRequest();
        if ($request->getMethod() == 'POST') {

            $editForm->bind($request);

            if ($editForm->isValid()) {
                $em->persist($entity);
                $em->flush();

                return $this->redirect($this->generateUrl('edit', array('id' => $id)));
            }

        }

        return array(
            'entity' => $entity,
            'edit_form' => $editForm->createView(),
            'delete_form' => $deleteForm->createView(),
        );
    }

Relative Routing

As a bonus this bundle allows you to refer relatively to another route, so instead of writing in the view:

<!-- Acme/DemoBundle/Resources/views/Product/index.html.twig -->
<a href="{{ path('acme_demo.product.show', { 'id': entity.id }) }}">show</a>

You can simply write this:

<!-- Acme/DemoBundle/Resources/views/Product/index.html.twig -->
<a href="{{ path('show', { 'id': entity.id }) }}">show</a>

Or instead of writing this in the controller:

$this->generateUrl('acme_demo.product.show', array('id' => $id))

You can simply write this:

$this->generateUrl('show', array('id' => $id))

Since routes are now expected to be “{bundle}.{controller}.{action}” format and the current route is “acme_demo.product.index” the undefined route “show” can be automatically matched to “acme_demo.product.show”. Apart from enforcing consistency this also allows for greater reuse of (parts of) code.

Get the bundle

Now quickly go grab a copy of the LswDefaultRoutingBundle and enjoy its benefits!

0. How to Generate an Entity in Symfony2 (11:41)
http://www.youtube.com/watch?v=0SE09AlJSEI

1. How to Generate a Repository in Symfony2 (1:20)
http://www.youtube.com/watch?v=asC9LL4cHrs

2. How to Generate CRUD for an Entity in Symfony2 (4:36)
http://www.youtube.com/watch?v=Jqp781knyPM

3. How to Install and Configure the Sonata Admin Bundle (11:19)
http://www.youtube.com/watch?v=_ZGRs4Rd94A

4. How to Create a ManyToMany Relationship with Doctrine2 & Symfony2 (27:35)
http://www.youtube.com/watch?v=akV4D7cq4UQ

5. Hack to Update a ManyToMany Relationship Through the Inverse Side in Doctrine2  (4:00)
http://www.youtube.com/watch?v=kPrgoe3Jrjw

It clearly explains the process of generating applications using the (Doctrine) CRUD generator in video’s 0,1 and 2. In video 3 the (Sonata) admin generator is explained.  Video 4 and 5 explain the more advanced entity relationships “ManyToMany” and “inversed ManyToMany”.

Automatic Update Bundle

So, what is new? Well, would’nt it be cool to have a WordPress-style automatic update button? That’s exactly what we deliver with LswAutomaticUpdateBundle! Whether or not this desirable can be argued about, but if you are after fast and consistent deployments, this might work for you. You get an extra icon in the Web Debug Toolbar indicating how long ago you last ran the Composer ‘update’ command. By clicking the icon you get to a panel where you can enter the password to actually execute the upgrade. Before the upgrade is executed the dry-run is executed. Only if all commands succeed, the upgrade button is shown.

Features

The bundle does some cool tricks, such as:

• Showing app/console and composer.phar output in color, thanks to ANSI to HTML support.
• Showing days not updated to remind you to keep your software up-to-date.
• Listing all requires and installed versions of the packages with their descriptions.
• Identifying unstable (dev) packages to encourage you to run stable versions

It also has some password based security, so that it can only be invoked by authorized people.

Pros and Cons

Maybe system administrators could see this bundle as a stability threat, because it enables quick updating of the code. Also, from a security point of view one could argue that this bundle is not desirable, since the use of ‘system()’ calls from PHP to execute shell commands is frowned upon.

On the other hand the bundle encourages frequent updates and use of versioned packages which should enhance security. Also, because this is a single click upgrade, upgrades should not fail because somebody forgot to type a command or makes a mistake in a command.

Installation

As always, you can find the bundle on Github and Packagist. It supports installation using Composer as described in the README.

<?php
// get memcache object
$memcache = $this->get('memcache.default');
// get value from cache
$value = $memcache->get('key')
// if value is not in cache
if ($value===false) {
 // calculate the value
 $value = $this->calculateValue();
 // store the value in the cache for 1 hour
 $value = $memcache->set('key', $value, 3600);
}
// do something with the value
print "value: $value\n";

Now let us examine a high traffic website case and see how that will work:

Your cache is stored for 90 minutes. It takes about 3 second to calculate the cache value and 1 ms second to read from cache the cache value. You have about 5000 requests per second and that the value is cached. You get 5000 requests per second taking about 5000 ms to read the values from cache. You might think that that is not possible since 5000 > 1000, but that depends on the number of worker processes on your web server  Let’s say it is about 100 workers (under high load) with 75 threads each. Your web requests take about 20 ms each. Whenever the cache invalidates (after 90 minutes), during 3 seconds, there will be 15000 requests getting a cache miss. All the threads getting a miss will start to calculate the cache value (because they don’t know the other threads are doing the same). This means that during (almost) 3 seconds the server wont answer a single request, but the requests keep coming in. Since each worker has 75 threads (holding 100 x 75 connections), the amount of workers has to go up to be able to process them.

The heavy forking will cause extra CPU usage and the each worker will use extra RAM. This unexpected increase in RAM and CPU is called a “cache stampede” or “dog-piling” and is very unwelcome during peek hours on a web service. This problem is also explained in this “Memcache dog pile” article. Another good explanation of the problem can be found on github in a SIFO issues called “Cache: Avoid the dogpile effect”.

There is a solution: we serve the old cache entries while calculating the new value and by using an atomic read and write operation we can make sure only one thread will receive a cache miss when the content is invalidated. The algorithm is similar to Nginx’s “proxy_cache_use_stale” with “updating” mechanism and is implemented in AntiDogPileMemcache class in LswMemcacheBundle.

Below you see a test showing how it is working. Since phpunit does not support forking I had to go PHP native to make it work. First you see how to flush the Memcache from command line and after that you see the output when we run the test.

maurits@pc:~/src/Lsw/MemcacheBundle/Tests/Cache$ telnet 0 11211
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
flush_all
OK
quit
Connection closed by foreign host.
maurits@pc:~/src/Lsw/MemcacheBundle/Tests/Cache$ php DogPileTest.php
THREAD | SECOND | STATUS
1 |      0 | STALE!
3 |      0 | STALE!
2 |      0 | STALE!
1 |      0 | SET 1363565115
3 |      0 | SET 1363565115
2 |      0 | SET 1363565115
1 |      1 | 1363565115
3 |      1 | 1363565115
2 |      1 | 1363565115
1 |      2 | 1363565115
3 |      2 | 1363565115
2 |      2 | 1363565115
1 |      3 | STALE!
3 |      3 | 1363565115
2 |      3 | 1363565115
1 |      3 | SET 1363565119
2 |      4 | 1363565119
3 |      4 | 1363565119
1 |      4 | 1363565119
2 |      5 | 1363565119
3 |      5 | 1363565119
2 |      6 | 1363565119
3 |      6 | 1363565119
1 |      5 | 1363565119
1 |      6 | 1363565119
3 |      7 | STALE!
2 |      7 | 1363565119
3 |      7 | SET 1363565123
1 |      7 | 1363565123
2 |      8 | 1363565123
3 |      8 | 1363565123
1 |      8 | 1363565123
2 |      9 | 1363565123
1 |      9 | 1363565123
3 |      9 | 1363565123
maurits@pc:~/src/Lsw/MemcacheBundle/Tests/Cache$

Before we get to the code of the test and you are going to apply this everywhere, I want to stress the following considerations:

• ADP might not be needed if you have low amount of hits or when calculating the new value goes relatively fast.
• ADP might not be needed if you can break up the big calculation into smaller, maybe even with different timeouts for each part.
• ADP might get you older data than the invalidation that is specified. Especially when a thread/worker gets “false” for “get” request, but fails to “set” the new calculated value afterwards.
• ADP “get” and ADP “set” are more expensive than the normal “get” and “set”, slowing down all cache hits.
• ADP does not guarantee that the dog pile will not occur. Restarting Memcache, flushing data or not enough RAM will also get keys evicted and you will run into the problem anyway.

Now this is the code used to run the test:

<?php
namespace Lsw\MemcacheBundle\Tests\Cache;

use Lsw\MemcacheBundle\Cache\AntiDogPileMemcache;

require_once "../../Cache/LoggingMemcacheInterface.php";
require_once "../../Cache/LoggingMemcache.php";
require_once "../../Cache/AntiDogPileMemcache.php";

class DogPileTest //extends \PHPUnit_Framework_TestCase
{
  public function testDogPile()
  {
    for ($t=1; $t<3; $t++) {
      $pid = pcntl_fork();
      if ($pid == -1) {
        die('could not fork');
      }
      if ($pid==0) {
        break;
      }
    }
    $c = 10;
    $m = new AntiDogPileMemcache(false);
    $m->addServer('localhost', 11211);
    if ($t==1) {
      echo "THREAD | SECOND | STATUS\n";
    }
    for ($i=0; $i<$c; $i++) {
      sleep(1);
      if (false === ($v = $m->getAdp('key'))) {
        echo sprintf("%6s | %6s | %s\n", $t, $i, "STALE!");
        sleep(1);
        $v = time();
        $m->setAdp('key', $v, 2);
        echo sprintf("%6s | %6s | %s\n", $t, $i, "SET $v");
      } else {
        echo sprintf("%6s | %6s | %s\n", $t, $i, $v);
      }
    }
    sleep(3);
  }
}

$t = new DogPileTest();
$t->testDogPile();

For further reading on applying memcache I highly recommend this article from IBM.

• http://www.php.net/manual/en/book.memcache.php (sudo apt-get install php5-memcache)
• http://www.php.net/manual/en/book.memcached.php (sudo apt-get install php5-memcached)

Should you use memcache or memcached? There are some people writing about this:

• http://code.google.com/p/memcached/wiki/PHPClientComparison
• http://stackoverflow.com/questions/1442411/using-memcache-vs-memcached-with-php
• http://serverfault.com/questions/63383/memcache-vs-memcached

Memcached 16% faster than Memcache?

Nobody seemed to benchmark the two clients to see which one is faster and that is why I did just that. I tested both clients on the same machine with default settings in Ubuntu 12.04 for both the clients and the server. This is the benchmark script I used:

<?php
// Initialize values: 10000 keys of 20 bytes with 40 bytes of data
$c = 10000;
$values = array();
for ($i=0;$i<$c;$i++) $values[sprintf('%020s',$i)]=sha1($i);
echo "memcache vs memcached: $c keys\n";
// Memcached
$m = new Memcached();
$m->addServer('localhost', 11211);
$start = microtime(true);
foreach ($values as $k => $v) $m->set($k, $v, 3600);
$time = microtime(true)-$start;
echo "memcached set: $time\n";
$start = microtime(true);
foreach ($values as $k => $v) $m->get($k);
$time = microtime(true)-$start;
echo "memcached get: $time\n";
// Memcache
$m = new Memcache();
$m->addServer('localhost', 11211);
$start = microtime(true);
foreach ($values as $k => $v) $m->set($k, $v, 0, 3600);
$time = microtime(true)-$start;
echo "memcache set: $time\n";
$start = microtime(true);
foreach ($values as $k => $v) $m->get($k);
$time = microtime(true)-$start;
echo "memcache get: $time\n";

And this is the output:

maurits@maurits-Aspire-X3960:~$ php memcache.php
memcache vs memcached: 10000 keys
memcached set: 0.91661500930786
memcached get: 0.86234307289124
memcache set: 1.0546097755432
memcache get: 1.0519700050354

We clearly see that memcached is faster than memcache, but to find out how much faster we have to use xdebug profiling (install with “sudo apt-get install php5-xdebug”). We enabled xdebug profiling by setting “xdebug.profiler_enable = 1″ in “/etc/php5/conf.d/xdebug.ini”. After running the php script a “cachegrind.out” file is created in the “/tmp” directory. KCachegrind (install with “sudo apt-get install kcachegrind”) can analyze that file and make pretty graphs:

picture 1: KCachegrind call graph

picture 2: KCachegrind callee map

picture 3: KCachegrind flat profile

Conclusion

The difference is really small, so there are probably better reasons to choose the memcached client over the memcache client

The  SncRedisBundle integrates Predis and phpredis into your Symfony2 application.

“Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.” – redis.io

At LeaseWeb we developed the LswMemcacheBundle. This is not because we do not like Redis. Maybe Redis serves a different purpose, since it has support for data structures, but is single threaded on the other hand. Having data structures could lead to better (more efficient) cache usage, while being single threaded does make a difference on the maximum amount of concurrent get and set operations (see benchmark). So without choosing for either Redis or Memcache we want to tell you about SncRedisBundle for Symfony.

• This is an awesome set of slides about Redis
• Redis has very good documentation
• It even has a manifesto
• The Symfony2 bundle for redis is really good
• If you do high traffic sites be careful not to run into open file limit issues

The biggest differences between Redis and Memcache we found:

• Redis has support for data structures
• Redis has support for persistent storage
• Redis has more features (in general)
• php-redis and memcached are both C based clients, but php-redis is not yet in the Ubuntu repository and predis might be slower.
• Memcache is multi-threaded and gets higher IOPS on a multi core server

If you run a high traffic PHP site, consider running Redis and/or Memcache and share your thoughts with us.