happy & lucky numbers

The other day I paired with the guys from @solutiondrive and @niklas_heer, we had a fun evening learing about happy numbers, shared PhpStorm knowledge, tried Codeception etc. Actually we didn’t even finish the “Happy Numbers” Kata, since we only wrote the classifying routine, not the loop generating the output.

On my way home I kept googling and also found out about Lucky Numbers. Lucky numbers are natural numbers, recursively filtered by a sieve that eliminates numbers based on their position (where the second number tells the elimination offsets).

So I immediately came up with another Kata: generating those numbers.
My constraint: no upper limit, i.e. use PHP’s Generator instead

… so I came up with the idea to implement the sieve itself as a Generator, that reads from an injected Generator, filters as needed and yields the result. The first “sieve generator” is fed from another generator that simply yields all natural numbers. The second one is fed from the first one and so on. The generator into generator injection is handled by yet another generator … turn’s out: it works, but doesn’t look so nice.
The outer generator cannot simply inject generators endlessly (since they are actually instanciated), so injection has to be deferred - that however dilutes the self-contained sieve generator :-(

Anyways it was a good exercise on PHP’s generators. I think I’ll give it another try soon, again with generators yet another approach.

funny Math.random behaviour

Playing around with V8’s custom startup snapshots I noticed some funny behaviour regarding Math.random.

It is clear that if you call Math.random() within the custom startup code the generated random numbers are baked into the snapshot and then not so random anymore. If you call Math.random() at runtime, without custom startup code, it just behaves as expected: it generates random numbers. However if you have custom startup code, calling Math.random() early on startup, it correctly generates random numbers during startup but it breaks runtime random number generation causing weird error messages like

TypeError: Cannot read property '4' of undefined

@virgofx raised this issue at the V8 issue tracker.

For the moment I came up with using random numbers from PHP’s Mersenne Twister

$this->v8 = new V8Js('PHP', [], [], true, $blob);
$this->v8->__random = function() { return mt_rand() / mt_getrandmax(); };
$this->v8->executeString('Math.random = PHP.__random; ');

20x performance boost with V8Js snapshots

Recently @virgofx filed an issue on V8Js whether (startup) performance of V8Js could be increased. He wants to do server-side React rendering and noticed that V8 itself needs roughly 50ms to initialize and then further 60ms to process React & ReactServer javascript code. Way too much for server side rendering (on more or less every request).

Up to V8 4.4 you simply could compile it with snapshots and V8Js made use of them. With 4.4 that stopped (V8Js just kept crashing), and I never really cared what they could do nor what the performance hit of this is, I just disabled them.

… turns out there even are three modes:

  • no snapshots at all (what I did then)
  • snapshots support enabled, with external snapshot data (the default)
  • snapshots support enabled, with internal snapshot data (the snapshots are then linked into the library itself)

Those snapshots are created once at compile time and store the state of V8’s heap after it has fully initialized itself. Hence their benefit is that the engine doesn’t fully bootstrap over and over, … it simply restores the snapshot and is (almost) ready to start.

Only the second of those three modes wasn’t supported by V8Js, since it simply didn’t provide the external startup data – and hence V8 failed to start.

Digging deeper into snapshots I found out about custom startup snapshots. V8 since version 4.3 allows extra JavaScript code to be embedded into the snapshot itself. This is you can bake React & ReactServer right into the snapshot so it doesn’t have to re-evaluate the source over and over again.

The performance impact of this is enormous:

performance comparison

The Y-axis shows milliseconds, the blue bar the amount of time needed by V8 to bootstrap, the red bar time needed to evaluate React & ReactServer source code. Timings are averages over 100 samples taken on my Core i5 laptop.

I compiled V8 5.0.104 with snapshot support, hence the blue bar immediately drops from about 60 ms down to about 4 ms. Since the base snapshots doesn’t have React included, the red bare remains at ~90 ms at first.

… including React into the snapshot, the red bar of course is gone, bootstrapping takes a little longer then – but it is many times faster than without snapshots.

V8Js on Heroku

Update Mar 28, 2016: It is no longer necessary (and hence discouraged) to use a forked buildpack. The official PHP buildpack now has support for so-called custom platform repositories, therefore better see here how to use V8Js on Heroku.

After I’ve built my own PHP buildpack with V8Js included it’s now easily possible to push PHP applications onto Heroku that require the extension.

When creating the app on Heroku simply specify the custom buildpack like

heroku create laughinghipster -b https://github.com/stesie/heroku-buildpack-php.git

… where laughinghipster is an arbitrary application name and the last argument the URL to my buildpack on Github.

The pushed repo must include a file named composer.json that requires ext-v8js; either with a particular version or just wildcard:

{
    "require": {
        "slim/slim": "2.*",
        "slim/views": "0.1.*",
        "twig/twig": "1.*",
        "ext-v8js": "*" 
    }
}

Then simply push the application to Heroku, it should detect the dependency on ext-v8js and simply download & install it:

stesie@hahnschaaf:~/Projekte/laughinghipster$ git push heroku master
Zähle Objekte: 1878, Fertig.
Delta compression using up to 4 threads.
Komprimiere Objekte: 100% (1738/1738), Fertig.
Schreibe Objekte: 100% (1878/1878), 4.11 MiB | 108.00 KiB/s, Fertig.
Total 1878 (delta 938), reused 207 (delta 89)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Fetching set buildpack https://github.com/stesie/heroku-buildpack-php.git... done
remote: -----> PHP app detected
remote: -----> No runtime required in 'composer.json', defaulting to PHP 5.6.15.
remote: -----> Installing system packages...
remote:        - PHP 5.6.15
remote:        - Apache 2.4.16
remote:        - Nginx 1.8.0
remote: -----> Installing PHP extensions...
remote:        - v8js (composer.lock; downloaded)
remote:        - zend-opcache (automatic; bundled)
remote: -----> Installing dependencies...
remote:        Composer version 1.0.0-alpha10 2015-04-14 21:18:51
remote:        Loading composer repositories with package information
remote:        Installing dependencies from lock file
remote:          - Installing slim/slim (2.6.2)
remote:            Downloading: 100%
remote:        
remote:          - Installing slim/views (0.1.3)
remote:            Downloading: 100%
remote:        
remote:          - Installing twig/twig (v1.23.1)
remote:            Downloading: 100%
remote:        
remote:        Generating optimized autoload files
remote: -----> Preparing runtime environment...
remote:        NOTICE: No Procfile, using 'web: vendor/bin/heroku-php-apache2'.
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing... done, 90.7MB
remote: -----> Launching... done, v3
remote:        https://laughinghipster.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy.... done.
To https://git.heroku.com/laughinghipster.git
 * [new branch]      master -> master

… and you’re set.

PHP buildpack for V8Js

Update Mar 28, 2016: It is no longer necessary (and hence discouraged) to fork the buildpack and rebuild it completely. The official PHP buildpack now has support for so-called custom platform repositories, and I’ve built one for V8Js meanwhile.

The other day I’ve configured a Dokku instance on my root server and then tried to install a PHP project requiring the V8Js extension. This of course failed for obvious reasons: Heroku’s buildpack for PHP doesn’t provide the V8Js PHP extension.

Luckily the buildpack is available on Github. So it should be possible to have a fork that supports V8Js.

So here we go, how hard can it be? :)

If you’re just interested in using the buildpack I’ve created, feel free to just use my own fork on Github. The master branch references my personal S3 bucket, so Heroku or Dokku just fetch the needed resources from there.

Below you’ll find a step by step guide on how to build such a buildpack on your own:

Step 0: Setting up S3 bucket

The buildpack assumes that the binaries are stored on S3; hence a new S3 bucket (along an IAM user with access on that bucket) needs to be created first.

Step 1: Clone Heroku’s PHP buildpack

First step is to fetch Heroku’s original build pack to a local workspace:

$ git clone https://github.com/heroku/heroku-buildpack-php
$ cd heroku-buildpack-php

Step 2: Create app on Dokku and configure remote

… before any modification, just create an app as a clean base for building the binaries later on:

$ dokku apps:create buildpack-php
$ dokku config:set buildpack-php BUILDPACK_URL="https://github.com/heroku/heroku-buildpack-python"
$ git remote add dokku ssh://dokku@dokku.brokenpipe.de:20022/buildpack-php

Step 3: Configure app

S3/IAM access key + secret need to be provided as environment variables, therefore we simply set them with dokku config:set.

$ dokku ps:scale buildpack-php web=0
$ dokku config:set buildpack-php AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXXXXXXX"
$ dokku config:set buildpack-php AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
$ dokku config:set buildpack-php S3_BUCKET="buildpack-phpv8"
$ dokku config:set buildpack-php S3_PREFIX="dist-cedar-14-master"
$ dokku config:set buildpack-php STACK="cedar-14"
$ dokku config:set buildpack-php WORKSPACE_DIR=/app/support/build

Step 4: Upload the buildpack to Dokku

If the app is configured as needed, just push the unmodified code to see of everything works as expected:

$ git push dokku master

Step 5: Build used libraries

Evey build should have its own dokku run to ensure that nothing else but the declared dependencies can be used plus the packages are as clean as possible:

$ dokku run buildpack-php bob deploy libraries/gettext
$ dokku run buildpack-php bob deploy libraries/icu
$ dokku run buildpack-php bob deploy libraries/libmcrypt
$ dokku run buildpack-php bob deploy libraries/pcre
$ dokku run buildpack-php bob deploy libraries/zlib

Step 6: Build Apache, Nginx & PHP

$ dokku run buildpack-php bob deploy apache-2.4.16
$ dokku run buildpack-php bob deploy nginx-1.8.0
$ dokku run buildpack-php bob deploy php-min
$ dokku run buildpack-php bob deploy composer
$ dokku run buildpack-php bob deploy composer-1.0.0alpha11

The php-min package is always initially installed on every built slug (container) and used to track dependencies & installation candidates. The composer installer needs information on which PHP versions & extensions are available, which are made available by means of JSON manifests, which need to be uploaded seperately.

u26678@1b8458e5519f:~$ bob build php-5.5.30

Fetching dependencies... found 4:
  - libraries/zlib
  - libraries/libmcrypt
  - libraries/icu
  - libraries/gettext
Building formula php-5.5.30 in /tmp/bobE7yqsM:
    -----> Building PHP 5.5.30...
...
    -----> Done. Run 's3cmd --ssl --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --acl-public put /tmp/bobE7yqsM/php-5.5.30.composer.json s3://buildpack-phpv8/dist-cedar-14-master/php-5.5.30.composer.json' to upload manifest.
u26678@1b8458e5519f:~$ s3cmd --ssl --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --acl-public put /tmp/bobE7yqsM/php-5.5.30.composer.json s3://buildpack-phpv8/dist-cedar-14-master/php-5.5.30.composer.json
'/tmp/bobE7yqsM/php-5.5.30.composer.json' -> 's3://buildpack-phpv8/dist-cedar-14-master/php-5.5.30.composer.json'  [1 of 1]
 2026 of 2026   100% in    0s     3.96 kB/s  done
'/tmp/bobE7yqsM/php-5.5.30.composer.json' -> 's3://buildpack-phpv8/dist-cedar-14-master/php-5.5.30.composer.json'  [1 of 1]
 2026 of 2026   100% in    0s     3.53 kB/s  done
Public URL of the object is: http://buildpack-phpv8.s3.amazonaws.com/dist-cedar-14-master/php-5.5.30.composer.json

… the build deploy command automatically creates the manifest and prints the command required to publish it. Copy & paste FTW :-)

… repeat that for php-5.6.16 and php-7.0.1

Last but not least all of those little manifests files need to be collected into a single file named packages.json:

u4725@983895293e5e:~$ support/mkrepo.sh
-----> Fetching manifests...
WARNING: Empty object name on S3 found, ignoring.
's3://buildpack-phpv8/dist-cedar-14-master/ext-v8js-0.4.0_php-5.5.composer.json' -> './ext-v8js-0.4.0_php-5.5.composer.json'  [1 of 7]
 393 of 393   100% in    0s     2.74 kB/s  done
's3://buildpack-phpv8/dist-cedar-14-master/ext-v8js-0.4.0_php-5.6.composer.json' -> './ext-v8js-0.4.0_php-5.6.composer.json'  [2 of 7]
 393 of 393   100% in    0s     3.31 kB/s  done
...
-----> Generating packages.json...
-----> Done. Run 's3cmd --ssl --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --acl-public put packages.json s3://buildpack-phpv8/dist-cedar-14-master/packages.json' to upload repository.
u4725@983895293e5e:~$ s3cmd --ssl --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --acl-public put packages.json s3://buildpack-phpv8/dist-cedar-14-master/packages.json
'packages.json' -> 's3://buildpack-phpv8/dist-cedar-14-master/packages.json'  [1 of 1]
 9913 of 9913   100% in    0s    19.31 kB/s  done
'packages.json' -> 's3://buildpack-phpv8/dist-cedar-14-master/packages.json'  [1 of 1]
 9913 of 9913   100% in    0s    17.36 kB/s  done
Public URL of the object is: http://buildpack-phpv8.s3.amazonaws.com/dist-cedar-14-master/packages.json

… again, mkrepo.sh tells us how to upload the resulting file.

… now we have a buildpack for PHP that is functionally equivalent to Heroku’s version (apart from not having compiled each and every PHP version and extension, that might be available on Heroku).

Step 7: Update bin/compile

bin/compile is the shell script that is executed during the git push to either Dokku or Heroku; it has a variable named S3_URL which needs to point to the S3 bucket created in Step 0.

Step 8: Adding own recipes

As the buildpack clone is usable, now it’s time to add further recipies. In case of V8Js this is the V8 library itself (packages as libraries/v8) and the extension plus a bare version for every major PHP release.

The bare version is the extension alone, without the PHP version included, that it was built against as well as any further dependencies (V8 in that case). The non-bare variant ships all dependencies except for PHP itself.

The recipes themselves are simple shell scripts.

The V8 library as well as the bare package versions don’t need a manifest file, as they are just needed during build time. The extension package itself needs one however, otherwise composer won’t find it … and hence cannot install it. This is in case of the v8js package the manifest file must be uploaded manually + the packages.json file needs to be regenerated.

Dokku on Docker

I’ve used Heroku once back in 2013 and actually liked it a lot as it lets you concentrate on the development part and just pulling in services as needed (and without further work needed).

Contrary I have had a root server at Hetzner for more then a decade now and I don’t want to pay for pet projects hosted on Heroku then (and I’m interested in hosting to some degree at least).

Enter Dokku. Dokku is a very small “platform as a service” thingy, written in around 200 lines of Bash. After all a mini-Heroku based on Docker. Their installation guide assumes that you have a VPS and their bootstrap script converts the VPS into a mini-Heroku, running Dokku on the box itself alongside Nginx as a reverse proxy.

So far so good, but that’s not what I wanted to have as I already have the root server in place which is dockerized heavily (ldap instance, mailgate, web mailer, several blogs, gitlab, reverse proxy, etc.) … hence Dokku itself should go into another Docker container (and the Dokku apps should run Docker-in-Docker – like I’m already doing with the V8Js Jenkins instance).

Googling around I’ve found a promising project over at Github: dokku-in-docker. It is a bit dated (last commit back in Nov 2014) and Dokku itself has gathered quite some pace recently, hence the container didn’t build – and afterall I wanted a recent Dokku version.

Hence I have my own fork now. Simply build it as usual:

docker build -t dokku-in-docker .

then run it like

/usr/bin/docker run --name="dokku.brokenpipe.de" --privileged -d
  -e VHOSTNAME="dokku.brokenpipe.de"
  -e PUBKEY="ssh-rsa AAAA...vkr stesie@hahnschaaf"
  -e VIRTUAL_HOST="*.dokku.brokenpipe.de"
  -v "/opt/docker/dokku.brokenpipe.de/home":"/home/dokku"
  -v "/opt/docker/dokku.brokenpipe.de/docker":"/var/lib/docker"
  -v "/opt/docker/dokku.brokenpipe.de/dokku-services":"/var/lib/dokku/services"
  -p 20022:22
  "dokku-in-docker"
  • the VIRTUAL_HOST environment variable is for the reverse proxy container (jwilder/nginx-proxy) and not dokku itself
  • replace PUBKEY with your pubkey (~/.ssh/id_rsa.pub), dokku doesn’t support multiple users (but you can run several dokku-in-docker containers easily)
  • the first & second volume simply persist apps over container rebuild
  • the third volume persists databases created by dokku postgres:create et al

This way Dokku integrates nicely with the other Docker containers and my approach to have no persistence-needing data in the container itself.

Geierlein 0.8.0 released

Today I’ve released version 0.8.0 of Geierlein, the free Elster client written in HTML5 & JavaScript. Release aspects are, among others

  • bumping the Firefox MaxVersion to 41, to allow using it on the Firefox 41 XUL engine
  • upgrade of the included JavaScript crypto library Forge
  • switching over to RSAES-OAEP encryption scheme (from good old PKCS#7 scheme)

The encryption scheme switch on one hand was necessary, since the API will cease PKCS#7 support in April 2016, while currently supporting both schemes (actually since April 2015, when they opened the door for RSAES-OAEP). Besides that I wanted to make the switch early, since I consider going with better encryption is always the better option.

As Forge didn’t yet support RSAES-OAEP I spent some time implementing it back in August. My pull request on Forge unfortunately is still pending, as Dave is working on an upcoming API change.

Nevertheless Geierlein already ships the patch, so RSAES-OAEP/CMS is used from now on.