PyCon 2014, full conference on YouTube

pycon2014-logo_sub

PyCon US 2014 is being held in Montreal and the conference was last weekend (April 11-13).

PyCon is the largest annual gathering for the community using and developing the open-source Python programming language.

For all of you that did no go, the full conference is available as 138 YouTube videos:

  1. Python Epiphanies
  2. Opening Statements – PyCon 2014 (2014/04/11)
  3. mrjob: Snakes on a Hadoop
  4. Mining Social Web APIs with IPython Notebook
  5. Lightning talks – Saturday morning
  6. How to formulate a (science) problem and analyze it using Python code
  7. Hands-on intro to Python for beginning programmers
  8. Getting Started with Django, a crash course
  9. Exploring Machine Learning with Scikit-learn
  10. Diving deeper into Machine Learning with Scikit-learn
  11. Descriptors and Metaclasses – Understanding and Using Python’s More Advanced Features
  12. 0 to 00111100 with web2py
  13. What Is Async, How Does It Work, And When Should I Use It?
  14. Twisted Mixing
  15. Turn Your Computer Into a Server
  16. The Young Coder: Let’s Learn Python
  17. The State of Crypto in Python
  18. The Python Pipeline: Why you should reach out to local teachers
  19. Teaching Python: To Infinity and Beyond
  20. TDD for web applications, from scratch
  21. Straightening Out AngularJS with Python
  22. So you want to be a full-stack developer? How to build a full-stack python
  23. Shiny, Let’s Be Bad Guys: Exploiting and Mitigating the Top 10
  24. Search 101: An Introduction to Information Retrieval
  25. Realtime predictive analytics using scikit-learn & RabbitMQ
  26. Python Scraping Showdown: A performance and accuracy review
  27. Python + Geographic Data = BFFs
  28. Python for Social Scientists
  29. Python 3/2 Web Development with Pyramid
  30. Puppet Modules: Apps for Ops
  31. PostgreSQL Proficiency for Python People
  32. Pickles are for Delis, not Software
  33. Lightning talks – Sunday morning
  34. Let’s Learn Twisted Python
  35. Kneel And Disconnect: Getting The Fastest Connection Out Of A Hostname
  36. Keynote – John Perry Barlow
  37. IPython in depth: high productivity interactive and parallel python
  38. Introduction to Web (and data!) Scraping with Python
  39. Introduction to SQLAlchemy
  40. Introduction to Regular Expressions
  41. Introduction to game programming
  42. Improving automated testing with py.test
  43. Import-ant Decisions
  44. How to Get Started with Machine Learning
  45. Hello Physical World: A Crash Course on the Internet of Things
  46. Hands-on with Pydata: how to build a minimal
  47. Getting Started with SaltStack
  48. Getting Hy on Python: How to implement a Lisp front-end to Python
  49. Generators: The Final Frontier
  50. For Lack of a Better Name(server): DNS Explained.
  51. Flask by Example
  52. Faster Python Programs through Optimization
  53. Fan-in and Fan-out: The crucial components of concurrency
  54. Enough Machine Learning to Make Hacker News Readable Again
  55. Dynamics and Control with Python
  56. Django for Web Designers and Front End Developers
  57. Distributed task processing using Celery
  58. Distributed Computing Is Hard, Lets Go Shopping
  59. Decorators: A Powerful Weapon in your Python Arsenal
  60. Data Wrangling for Kaggle Data Science Competitions — An etude
  61. Contribute with me! Getting started with open source development
  62. Computer science fundamentals for self-taught programmers
  63. Character encoding and Unicode in Python
  64. Castle Anthrax: Dungeon Generation Techniques
  65. Cache me if you can: memcached, caching patterns and best practices
  66. By Your Bootstraps: Porting Your Application to Python3
  67. Build your own PiDoorbell! – Learn Home Automation with Python
  68. Building and breaking a Python sandbox
  69. Blending art, technology, and light, Python for interactive and real time
  70. Beyond Defaults: Creating Polished Visualizations Using Matplotlib
  71. Bayesian statistics made simple
  72. Application Deployment State of the Onion
  73. Ansible – Python-Powered Radically Simple IT Automation
  74. An Introduction to Twisted
  75. All Your Ducks In A Row: Data Structures in the Standard Library and Beyond
  76. Advanced methods for creating decorators
  77. Writing RESTful web services with Flask
  78. Which messaging layer should you use if you want to build
  79. What is coming in Python packaging
  80. Upgrade your Web Development Toolchain
  81. Unit Testing Makes Your Code Better
  82. Track memory leaks in Python
  83. The Sorry State of SSL
  84. The Day of the EXE Is Upon Us
  85. Technical on-boarding, training, and mentoring.
  86. Subprocess to FFI: Memory, Performance, and Why You Shouldn’t Shell Out
  87. So You Want to Build an API?
  88. Software Engineering Research for Hackers: Bridging the Two Solitudes
  89. Software Carpentry: Lessons Learned
  90. Smart Dumpster: Employing Python to Report Real-Time Resource
  91. Set your code free: releasing and maintaining an open-source Python project
  92. See Docs Run. Run, Docs, Run!
  93. Sane schema migrations with Alembic and SQLAlchemy
  94. REST is not enough: Using Push Notifications to better support your mobile clients
  95. Quick Wins for Better Website Security
  96. Python packaging simplified, for end users, app developers
  97. Python in the Browser: Intro to Brython
  98. Pushing Python: Building a High Throughput, Low Latency System
  99. Programming an Autonomous 20 Foot Blimp with Python
  100. PostgreSQL is Web Scale (Really)
  101. Postgres Performance for Humans
  102. Performance Testing and Profiling: A Virtuous Cycle
  103. Outreach Program for Women: Lessons in Collaboration
  104. My big gay adventure. Making, releasing and selling an indie game made in python.
  105. Multi-factor Authentication – Possession Factors
  106. Localization Revisited
  107. Lightning talks – Sunday morning
  108. Lightning talks – Saturday evening
  109. Know Thy Neighbor: Scikit and the K-Nearest Neighbor Algorithm
  110. Keynote – Van Lindberg
  111. Keynote – Jessica McKellar
  112. Keynote – Guido Van Rossum
  113. Keynote – Fernando Perez
  114. It’s Dangerous to Go Alone: Battling the Invisible Monsters in Tech
  115. Introduction to SQLAlchemy Core
  116. Introduction to Docker
  117. In Depth PDB
  118. Hitchhikers Guide to Participating in Open Source
  119. Getting Started Testing
  120. Garbage Collection in Python
  121. Games for Science: Creating interactive psychology experiments
  122. Fast Python, Slow Python
  123. Farewell and Welcome Home: Python in Two Genders
  124. Django: The good parts
  125. Diving into Open Data with IPython Notebook & Pandas
  126. Discovering Python
  127. Developing Flask Extensions
  128. Designing Poetic APIs
  129. Designing Django’s Migrations
  130. Deliver Your Software In An Envelope
  131. Data intensive biology in the cloud: instrumenting ALL the things
  132. Closing address – PyCon 2014 (2014/04/13)
  133. Cheap Helicopters In My Living Room
  134. Building the App
  135. A Scenic Drive through the Django Request-Response Cycle
  136. Analyzing Rap Lyrics with Python
  137. Advanced techniques for Web functional testing
  138. 2D/3D graphics with Python on mobile platforms

Now you need some serious time off to view all that!

source: http://pyvideo.org/category/50/pycon-us-2014

“The Expert” is Lauris Beinerts viral comedy sketch

“The Expert” is the title of the short comedy sketch Lauris Beinerts (Google+) posted a last week on his Youtube account. It went viral and is heading for half a million views in under seven days. It is about a “Funny business meeting illustrating how hard it is for an engineer to fit into the corporate world.” The video stars Orion Lee, James Marlowe, Abdiel LeRoy, Ewa Wojcik, and Tatjana Sendzimir, and is written & directed by Lauris Beinerts. It is based on a short story by Alexey Berezin titled “The Meeting” (in Russian) or translated by Google.

“I can do absolutely everything, I’m an expert!”

At LeaseWeb we understand…

At LeaseWeb we understand how hard it is for an expert to operate in the corporate world. So, if you have been in too many meetings like the one so well portrayed above, then you may want to take a look at our career offerings for engineers. Currently we are looking for:

  • Senior Engineer Virtualization and Storage
  • Hosting Engineer Virtualisation and Storage
  • Hosting Engineer Cloud
  • Software Engineer CDN
  • Software Engineer Cloud
  • Infrastructure Security Engineer
  • Infrastructure Engineer Cloud

We employ over 360 people in the Netherlands, Germany, the UK, and the US. Despite the company size, we try to keep things informal around the offices, with few layers of management. The members of the OCOM Group (LeaseWeb, EvoSwitch, FiberRing, DataXenter) hail from over 34 different nationalities, making English the language of choice.

Interested in working for us? Take a look at our careers website!

Text mode 2048 game in C, algorithm explained

2048 is hot! It may be considered the next Flappy Bird. I implemented 2048 in the C programming language, so you can run it on your Linux server on the console (in text mode). You can find the 2048.c project on Github.

2048

Rules

You can move the tiles in four directions using the arrow keys: up, down, left, and right. All numbers on the board will slide into that direction until they hit the wall and if they bump into each other then two numbers will be combined into one if they have the same value. Each number will only be combined once per move. Every move a new number 2 or 4 appears. If you have a 2048 on the board you have won, but you lose once the board is full and you cannot make a move.

Algorithm

I chose to store the field in a two-dimensional array called board that is 4×4.

[2][0][8][2]          [2][4][8][4]
[0][4][0][2]  press   [4][8][4][4]
[4][0][2][2]  up key  [0][0][0][0]
[0][8][2][2]          [0][0][0][0]

Since I want the board to be addressed as board[x][y], the board consists of a set of four columns size four. So the first column is the array [2,0,4,0]. If we press the up button this should become [2,4,0,0]. The function “slideArray()” is reponsible for this. This function will “slide” the numbers in the arrays like this:

[2,0,4,0] => [2,4,0,0]
[0,4,0,8] => [4,8,0,0]
[8,0,2,2] => [8,4,0,0]
[2,2,2,2] => [4,4,0,0]

The algorithm can just use board[x] to point to a column and work directly on that. In pseudocode, this is what the algorithm does:

- walk over the array from the first to the last number
  - for each original number in the array that is not zero
    - look backwards for a target position that does not contain a zero (unless it is position zero)
      - if the target position does not contain the original number use the next position
    - if the target position is different from the original position
      - add the number to the number on the target position
      - replace the original number by zero

The above algorithm executed will do all transformations, it will:

[2,2,2,2] => [4,4,0,0]
[0,8,2,2] => [8,4,0,0]

But there is a problem, it will also do this:

[2,2,4,4] => [8,4,0,0]

The two’s are merged into a four and then the first four is merged into that making it an eight. This is wrong. It should be doing this:

[2,2,4,4] => [4,8,0,0]

This is avoided by adding a “stop” variable that will be initially set to zero, but when a merge has been done it will be set to the merge position plus one. This will make sure any next slide will stop before it merges into this number again, since double merges are not allowed.

To prevent complex programming I use a function rotateBoard that rotates the board 90 degrees counter-clockwise. This allows the moveLeft(board) to be implemented as:

rotateBoard(board);
moveUp(board);
rotateBoard(board);
rotateBoard(board);
rotateBoard(board);

As long as you rotate four times in total everything works as expected. This method is not very efficient, but reduces the complexity of the code.

Compiling and running

Since 2048.c is a single C file it is easy to get running, just execute the following commands:

wget https://raw.githubusercontent.com/mevdschee/2048.c/master/2048.c
gcc -o 2048 2048.c
./2048

This will run on most machines. If not, then execute the following command to install the compiler:

sudo apt-get install build-essential

Further reading

Symfony2 Guzzle bundle for cURL API calling

The LswGuzzleBundle adds Guzzle API call functionality to your Symfony2 application. It is easy to use from the code and is aimed to provide full debugging capabilities. source: Github

On Packagist, you can see that in the past two years we have published 10 LeaseWeb Symfony2 bundles. The latest and tenth addition to the list is the LswGuzzleBundle. It is an alternative for the LswApiCallerBundle, so it also is a bundle that helps you to call API’s using cURL. The difference is that this bundle uses Guzzle as a wrapper for cURL.

Guzzle profiler and debugger

This bundle also adds a debug panel to your Symfony2, where you can find the cURL API calls Guzzle makes in your application. It provides profiling and extensive debugging capabilities to your Symfony2 application. We feel it is an essential bundle if you create an application that does not work with a database, but instead has an API as data store backend. Guzzle allows you to specify the parameters and and available calls in a Guzzle service description. Since this allows the bundle to understand the structure of the call, it will display the parsed parameters and output. This is an unique feature of this bundle.

guzzle_bundle

Open source

The bundle is MIT licensed. You can find it at:

As always there is a readme documentation on GitHub to get you started.

Testing framework using Maven, TestNG and Webdriver

We have set a goal to create a flexible and extendable automated testing framework, which should expand test coverage for as many LeaseWeb applications functionalities as possible. Tests should be also run on CI (Jenkins in our case). We are writing our tests on Java with help of several tools:

  • Webdriver is a driver that contains programming interface for controlling all kinds of possible actions in browser.
  • TestNG is a testing framework. It structures, groups, and launches tests. It also generates testing reports.
  • Maven is a software project management and comprehension tool. It manages all dependencies and different flows for building a project.

Implementation

1. We created a Maven project with next structure:
src/main/java – contains packages with Page Objects. Each package represents separate application and can contain packages with general methods and objects of the framework.
src/main/resources – here we store config files and other resources that are used in our tests.
src/test/java – contains packages with tests. Each package contains tests related only to one application or a test flow that goes through the couple of applications.

This structure is important for a project compiling in Maven.

2. The main goal was to write tests in a readable manner where each step is one single line of code that clearly describes what it does, and at the same time hides all actions that need to be done by Webdriver “under the hood” in our Page Object classes. By this we make it really easy to read and understand what our test does for every team member – from developers to product managers and even sales people.

3. Test run. We decided to use TestNG for several reasons:

a. People are moving from JUnit to TestNG once they start to write integration and end-to-end tests (e.g. with Selenium). On unit tests level you can put up with JUnit deficiencies, but once you start to write integration and end-to-end tests, JUnit stands in your way and TestNG comes to help you out.

b. No need to create your own Thread objects! You can use @Test annotation with ThreadPoolSize, and invocationCount parameters to run parallel tests.

c. Parameterization. A good testing framework should make it simple, and avoid copy\paste coding. Alas, the default JUnit implementation of the parametrized tests is unusable. In short, JUnit expects all test methods to be parameter-less. Because of this constraint, parameterized tests with JUnit looks awkward because parameters are passed by constructor and kept as fields. People who work with JUnit usually test their code with only one set of parameters. I suspect this is because parameterized tests are unusable with JUnit. And this is bad because such tests are really useful. TestNG allows you to use @DataProvider annotation, which allows you to parameterize your tests however you like and improve your test coverage dramatically with less lines of code in your tests.

d. Parameters to passing the tests:

  • Configure your tests so they, for example, invoke GUI methods on different servers/ports?
  • Run tests using MySQL database instead of h2?
  • Run GUI tests with Opera, Firefox, Chrome (or even IE 6)?
  • It is all easy to do with @Parameters annotation in tests.

e. Groups of tests. Each test class (and method) can belong to multiple groups, and with TestNG it’s very easy to run only selected group with @Test (group) annotation. You can easily set in CI which group of tests you want to run – smoke, GUI, API, etc.

f. Test dependencies. It’s not desirable practice with unit tests, but when it comes to integration tests, pretty often you cannot avoid it. This is where TestNG becomes very helpful and easy to use.

g. Reporting. TestNG provides more detailed and helpful reports in comparison with JUnit.

All these settings and tweaks can be specified in textng.xml file, which  is used by TestNG to run tests both on your local dev machine and on CI Server. It will declare which tests should be run or skipped and contains information about groups, packages, and classes. This file is located in the root project folder.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="CAS" verbose="1" >
  <test name="RegressionLogin"   >
  <classes>
	<class name="OrderFlowTests">
      <methods>
        <include name="orderFlow" />
      </methods>
    </class>

    <class name="LoginOppTests" >
   	  <methods>
        <include name="successLoginToOpp" />
      </methods>
 	</class>
  </classes>

 </test>
</suite>

4. Our pom.xml setup.

The beauty of Maven for us, is that it keeps and maintains all dependencies that we use in our project. It makes it much easier to setup the project on a new machine, and also makes a lot easier to run the project on CI. We need at least two plugins to start maven-compiler-plugin и maven-surefire-plugin (in the tags of which we have written our testing.xml). We made few tweaks in the plugin configuration section to show our testing.xml file, and also specify which file naming formats should be considered as tests for TestNG. File is located in to the root project folder.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>NewLeaseWebSite</groupId>
	<artifactId>NewLeaseWebSite</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<dependencies>
		<dependency>
			<groupId>org.testng</groupId>
			<artifactId>testng</artifactId>
			<version>6.3.1</version>
		</dependency>
		<dependency>
			<groupId>org.seleniumhq.selenium</groupId>
			<artifactId>selenium-firefox-driver</artifactId>
			<version>2.37.1</version>
		</dependency>
		<dependency>
			<groupId>org.seleniumhq.selenium</groupId>
			<artifactId>selenium-java</artifactId>
			<version>2.26.0</version>
		</dependency>
	</dependencies>
	<build>
		<testSourceDirectory>src/tests/java</testSourceDirectory>
		<resources>
			<resource>
				<directory>src/main/resources</directory>
				<excludes>
					<exclude>**/*.java</exclude>
				</excludes>
			</resource>
		</resources>
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-compiler-plugin</artifactId>
					<version>3.1</version>
					<configuration>
						<source>1.7</source>
						<target>1.7</target>
					</configuration>
				</plugin>
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-surefire-plugin</artifactId>
					<version>2.16</version>
					<configuration>
						<suiteXmlFiles>
							<suiteXmlFile>testng.xml</suiteXmlFile>
						</suiteXmlFiles>
						<includes>
							<include>**/Test*.java</include>
							<include>**/*Tests*.java</include>
							<include>**/*Tests.java</include>
							<include>**/*Test.java</include>
							<include>**/*TestCase.java</include>
						</includes>
					</configuration>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>
</project>

As a result, we have a test framework that can be easily extended, should not require much effort for maintenance in the future, and can be run on any CI server with several simple setup steps.