• 50117This month:
  • 147Today:
  • 13Currently online:

LeaseWeb CDN

Posts Tagged ‘php’

Wikipedia client for the command line


Don’t you want the power of Wikipedia at your fingertips when you are logged into your Linux server? With this tool you can! It is build with PHP/Curl and is based on the Wikipedia API.

Wikipedia (open)search API

Wikipedia allows us to program software that takes advantage of their huge database of articles and definitions. They do this by exposing an API. I implemented their “opensearch” API call to search for articles. We are not doing anything unintended. Wikipedia made the API to allow people to build useful (or less useful, you decide) tools like this one. They do ask people to behave nicely in their Wikipedia API Etiquette, so read them and behave!

HTML to text

Instead of making my own wikitext parser, I used the Wikipedia “parse” API call. This call transforms articles from the internal wikitext format to HTML. I use the command line application “elinks” (alternative to the “lynx” text-mode web browser) to convert the HTML to text so it can be displayed on the console. This browser software is configured not to use the network or write files to disk, so you do not have to worry about your privacy.

How does this work?

It is a PHP script that uses Curl. The whole wped script is currently only 70 lines of code. The opensearch call is really easy to implement as you can see below:

$url = '';
$user_agent = 'Wped/0.1 ( PHP-Curl';

$curl = curl_init();
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_USERAGENT, $user_agent);

$arguments = array(

curl_setopt($curl, CURLOPT_URL, $url.'?'.http_build_query($arguments));
$results = simplexml_load_string(curl_exec($curl));

With this example it should be easy for you to start building your own Wikipedia tool, let me know if you did!


To install, just copy-paste the following lines in your console:

sudo apt-get install php5-cli php5-curl elinks
wget -O wped
chmod 755 wped
sudo mv wped /usr/bin/wped

To uninstall use the following:

sudo rm /usr/bin/wped

Congratulations, now you can search for elephants using the “wped” command! And if you need the full article you can use the “-f” flag.

Yes! I need this, fast!

How can you have lived without it? Why would you ever use “wget”, when you can use “wped”? It adds so much fun to all your Wikipedia searches. The source and full instructions are available on my Github account:


PHP 5.3 is now officially end-of-life (EOL)

PHP 5.3 last regular release (5.3.27) was done in July 2013, back then we read the following statement on the release notes:

Please Note: This will be the last regular release of the PHP 5.3 series. All users of PHP are encouraged to upgrade to PHP 5.4 or PHP 5.5. The PHP 5.3 series will receive only security fixes for the next year. –

So, back then it was not a big deal, since security fixes would be released for one more year (and a year seems very long). But last week PHP 5.3.29 was released and since that year has passed PHP 5.3 is now officially end-of-life (EOL). This means there are no further updates, not even security fixes, as you can read in the release notes:

This release marks the end of life of the PHP 5.3 series. Future releases of this series are not planned. All PHP 5.3 users are encouraged to upgrade to the current stable version of PHP 5.5 or previous stable version of PHP 5.4, which are supported till at least 2016 and 2015 respectively. –

Ubuntu Linux users that run the still supported (and popular) 12.04 LTS release on their web server should not be worried too much: Ubuntu maintainers will backport security fixes until 2017. But running PHP 5.3 might be cumbersome, especially if you want to develop using the latest PHP frameworks or libraries. These often contain “array short syntax” and thus require PHP version 5.4 or higher . The simplest option is to upgrade your Ubuntu 12.04 LTS to 14.04 LTS, since that comes with PHP 5.5. If you decide to stay at 12.04 for a while, you will be stuck with 5.3.10 from the repo, unless you…

Upgrade PHP from 5.3 to 5.4 in Ubuntu 12.04 LTS

This is more or less the only option you have. Since it is not officially supported you have to install a PPA. I normally do not recommend this, since you could mess up your system badly and/or severely endanger the security of your machine. But I must admit that Ondřej Surý’s PPA is a very famous and widely used one, which would make it a bit more trusted. So, I will include the instructions, but you have been warned:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ondrej/php5-oldstable
sudo apt-get update
sudo apt-get dist-upgrade

Why you should not upgrade PHP to 5.5 in Ubuntu 12.04 LTS

PHP 5.5 and it’s dependencies are provided by the “ppa:ondrej/php5″ repo. And even though PHP 5.5 is longer supported and more powerful than PHP 5.4, you should probably stick to PHP 5.4. The reason for this is that PHP 5.5 requires Apache 2.4, where Ubuntu 12.04 comes bundled with Apache 2.2 by default. This means that when you upgrade PHP 5.3 to PHP 5.5 you also have to upgrade Apache 2.2 to Apache 2.4 (as a dependency). This could break many things, but it will (most certainly) break your virtual host configuration. So this is something I can’t recommend unless you are really sure what you are doing. Do not upgrade PHP to version 5.5 without having a tested upgrade plan. I’m serious… be very very careful!

Session locking: Non-blocking read-only sessions in PHP

I ran into an excellent article titled PHP Session Locks – How to Prevent Blocking Requests and it inspired me. Judging by the comments, it seems that not everybody fully understands what session locking is, how it works, and why it is necessary. This post tries to clear these things up and also gives you a dirty way of speeding up your AJAX calls significantly.

What session locking is

To understand this, we first need to know that a web server does not run your PHP code in a single process. Multiple worker processes are running concurrently and they are all handling requests. Normally, visitor requests of your web page are serialized. This is also where HTTP persistent connections (a.k.a. keep-alives) come into play. By keeping the connection open for the requesting of all the assets of the page, the connection overhead is avoided. Browsers are quite smart and will always try to serialize requests for HTML pages. For the assets (images, scripts, etc.) on the page there is another strategy. The browser will download multiple assets in parallel from each unique hostname it sees referred in the HTML. It can do this by opening multiple TCP connections or by pipelining. When a browser thinks it is downloading assets it may download these for a single visitor in parallel. Session locking avoids this parallelism (by blocking) to provide reliable access to the session data in this situation.

How session locking works

This is quite easy: When you call “session_start()” PHP will block (wait) in this call until the previous script has called “session_write_close()”. On Linux it does this by relying on the “flock()” call. This is an advisory locking mechanism that blocks until the lock is released. NB: This locking time is not counted as part of the “max_execution_time” (see: set_time_limit()).

Why session locking is necessary

Session locking prevents race conditions on the shared memory that is used to store session data. Every PHP process reads the entire session storage when starting and writes it back when closing. This means that to reliably store the logging-in of a user (which is typically done in the session data) you must make sure no other process has read the session data and will overwrite your data after you have written it (since the last write wins). This is needed even more when using AJAX or IFrames since the browser considers those loads to be assets and not HTML pages (so they will be parallelized).

Read-only sessions to the rescue

Many websites use AJAX calls to load data. While retrieving this data we would like to know whether the user logged in to deny access if  needed. Moreover, we would not like the loading of this AJAX data to be serialized by the session locking, which slows down the website. This is where the following (arguably dirty) code comes into place. It will allow you to gain read-only access to the session data (call it instead of “session_start()”). This way you can check permissions in your AJAX call, but without locking, thus not blocking and serializing the requests. It may speed up your PHP powered AJAX website significantly!

            function session_readonly()
                    $session_name = preg_replace('/[^\da-z]/i', '', $_COOKIE[session_name()]);
                    $session_data = file_get_contents(session_save_path().'/sess_'.$session_name);

                    $return_data = array();
                    $offset = 0;
                    while ($offset < strlen($session_data)) {
                        if (!strstr(substr($session_data, $offset), "|")) break;
                        $pos = strpos($session_data, "|", $offset);
                        $num = $pos - $offset;
                        $varname = substr($session_data, $offset, $num);
                        $offset += $num + 1;
                        $data = unserialize(substr($session_data, $offset));
                        $return_data[$varname] = $data;
                        $offset += strlen(serialize($data));
                    $_SESSION = $return_data;

I think this call should be added to the next PHP version. What do you think? Let me know in the comments.

Buffered Nginx log reading using PHP and logrotate

The problem: We want to continuously read the Nginx access logs on our high-traffic web nodes for real-time analytic purposes.

Logrotate every 10 seconds

Not possible! The minimum that logrotate allows is hourly. Apart from that I do not think this is very practical. I would still prefer to keep the logfiles hourly, for easy disk management. I do not like an hour of delay on my statistics. I would prefer something like one minute until it reaches the MySQL database.

Using a named pipe with mkfifo

The idea is that you can make a pipe with “mkfifo” and configure Nginx to log to that file. Sounds good until you realize that the buffer size of the pipe is quite limited (a few kb) and it will just silently drop log lines on buffer overflow. Also Igor Sysoev advised that you should not do it and since he knows what he is talking about, I suggest you stay away from it. :-)

Watch the log file with inotify

This idea is to install a watcher on the log file that triggers an event when the file is written to. This is a good idea, but the amount of events on a busy server may be just a little too high for this to be really efficient. We would rather not be notified every time the filesystem sees a change on the Nginx access log file. EDIT: This is actually not a bad idea as long as you enable buffered Nginx access logging, see next post.

Open the file and read until EOF every x seconds

Open the file and keep it open. Sleep for x seconds and read the file until the end. Send that data to your statistics cluster and return to the sleep. This works great until the file gets rotated. Once the file gets rotated, you first need to finish reading the old file until you start reading the new one. Fortunately, logrotate moves the file by default and a open moved file can still be read using the original file descriptor. When we find that that the file has been moved y seconds ago and not been written to for z seconds, then we can decide to close the rotated file and open the new current file. This way we can ensure that we do not lose any log lines. Certainly x, y, and z need some values found using trial and error. I think five, three, and one seconds should be a good starting point.

Configure logrotate to rotate hourly for 48 hours

This can be done using the following logrotate script:

/var/log/nginx/*.log {
        rotate 48
        create 0640 www-data adm
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
                [ -s /run/ ] && kill -USR1 `cat /run/`

Make sure you also move the logrotate cron job from daily to hourly using:

sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

Read the log files

The following code is an example in PHP that will demonstrate how this log reader will work. Note that Python and Golang may be more suitable languages for implementing these kinds of long running programs. This script just prints the data it reads. Streaming the log lines to your big data analytics cluster is left as an exercise to the reader. ;-)

$logfile = "/var/log/nginx/access.log";
$var_x = 5;
$var_y = 3;
$var_z = 1;
$f = fopen($logfile, "r");
while (true) {
  $data = stream_get_contents($f);
  echo $data; // or send the data somewhere
  $s1 = fstat($f);
  $s2 = stat($logfile);
  $renamed  = $s1["dev"] != $s2["dev"] || $s1["ino"] != $s2["ino"];
  if ($renamed && time()-$s1["ctime"]>$var_y && time()-$s1["mtime"]>$var_z) {
    echo "renamed\n";
    while (!file_exists($logfile)) sleep(1);
    $f = fopen($logfile,"r");
  } else sleep($var_x);

While running the script above you can make sure the access log is written to another terminal using:

siege -c 100 http://localhost/

Now you can force the rotation to be executed using:

sudo logrotate -f /etc/logrotate.d/nginx

And see how the script gracefully handles it.

Command line access from your browser using shell.php


Sometimes you want shell access from the browser. It can be achieved using PHP if the security settings allow it. I implemented this functionality in shell.php (available on Github). In the above screenshot you see how shell access from a browser works. The script allows you to upload, download, view edit and remove a file, zip and unzip a directory and traverse the directories on the server using the mouse, but you can also type in custom commands using the keyboard.

Security warning and disclaimer

Please run this script only on machines you own (or during an authorized pentest). Also make sure the machine is properly firewalled (port 80 should not be reachable from the Internet). Do not use it for malicious purposes! Read more on abuse of shell scripts here.

Known issues

If the script does not work it may be because the PHP “passthru” function on which it relies is disabled. To list disabled PHP functions execute the following PHP code:


On a out-of-the-box Ubuntu 14.04 that will output:

    [0] => pcntl_alarm
    [1] => pcntl_fork
    [2] => pcntl_waitpid
    [3] => pcntl_wait
    [4] => pcntl_wifexited
    [5] => pcntl_wifstopped
    [6] => pcntl_wifsignaled
    [7] => pcntl_wexitstatus
    [8] => pcntl_wtermsig
    [9] => pcntl_wstopsig
    [10] => pcntl_signal
    [11] => pcntl_signal_dispatch
    [12] => pcntl_get_last_error
    [13] => pcntl_strerror
    [14] => pcntl_sigprocmask
    [15] => pcntl_sigwaitinfo
    [16] => pcntl_sigtimedwait
    [17] => pcntl_exec
    [18] => pcntl_getpriority
    [19] => pcntl_setpriority
    [20] =>
    [0] =>

PHP shell execution commands

If the script does not run using passthru(), it will try a few other commands. The following commands are similar:

  • exec() Returns last line of commands output
  • passthru() Passes commands output directly to the browser
  • system() Passes commands output directly to the browser and returns last line
  • shell_exec() Returns commands output
  • popen() Opens read or write pipe to process of a command
  • proc_open() Similar to popen() but greater degree of control
  • pcntl_exec() Executes a program

Hardening your server with open_basedir

If the above script seems scary to you, then you may want to prevent it from executing on your server. You can do this by enabling safe mode (deprecated), using the “disable_functions” php.ini variable and/or the Suhosin function execution blacklist.

I have found a well written post on securing your PHP installation, check it out! Apart from limiting the executable functions they also recommend the “open_basedir” php.ini config variable. It limits the files that can be accessed by PHP to the specified directory-tree. I believe this is a powerful tool.

Also it could be a good idea to secure your “/tmp” directory with “nodev”, “nosuid” and “noexec” flag as described here.

Cannot find Suhosin?

Note that the “php5-suhosin” package (a PHP security extension) is no longer installed nor available on Debian based systems. Some of the security improvements have been incorporated into the latest PHP versions (5.4 and 5.5). If you want to install Suhosin (from Github) on Ubuntu 14.04 (PHP 5.5.9) you can follow this tutorial.

You can read more about the controversy around removing Suhosin on

Privilege separation

If there are multiple users on the system “privilege separation” is a MUST. This means running the PHP code in the user context  (e.g. as user “maurits”) and not as user “www-data”. I have found a great article explaining how this can be achieved. The easiest solution is to run:

sudo apt-get install libapache2-mpm-itk

And then adding the “AssignUserID” directive to every “VirtualHost” configuration. Note that this may not be the safest solution, but it performs well and is easy to install.


You should always update and patch PHP to the latest version to prevent exploitation of known security holes. Tools like “disable_functions”, “open_basedir”, Suhosin and filesystem flags reduce the attack surface and prevent exploitation of unknown security holes. You can use them to create implement a layered security strategy. Also do not forget about privilege separation.