Top-5 Things that might bite you

For the full list see

Left-to-right semantics for complicated expressions

$$foo['bar']['baz'] // interpreted as ($$foo)['bar']['baz']
$foo->$bar['baz']   // interpreted as ($foo->$bar)['baz']
$foo->$bar['baz']() // interpreted as ($foo->$bar)['baz']()
Foo::$bar['baz']()  // interpreted as (Foo::$bar)['baz']()

To restore the previous behaviour add explicit curly braces:

${$foo['bar']['baz']}
$foo->{$bar['baz']}
$foo->{$bar['baz']}()
Foo::{$bar['baz']}()

Detection: phan or unit test failures

Removed support for /e (PREG_REPLACE_EVAL) modifier

echo preg_replace('/:-:(.*?):-:/e', '$this->pres->\\1', $text);

Change to:

echo preg_replace_callback(
  '/:-:(.*?):-:/', 
  function($matches) {
    return $this->pres->{$matches[1]}; // Careful!
  },
  $text);

Detection: grep, warnings in logs or unit test failures

$HTTP_RAW_POST_DATA global removed

if (empty($GLOBALS['HTTP_RAW_POST_DATA']) &&
    strpos($_SERVER['CONTENT_TYPE'], 'www-form-urlencoded') === false) {
    $GLOBALS['HTTP_RAW_POST_DATA'] = file_get_contents("php://input");
}

Detection: grep, warnings in logs or unit test failures

session.lazy_write enabled by default

session.lazy_write = 0

Detection: Can cause out-of-band session read timing issues

Invalid octal literals now produce a parse error

echo 05678; // PHP 5.x outputs 375

Parse error: Invalid numeric literal in file.php on line 2		

Detecting parse errors is easy: php -l

Static Analysis

% phan -h
Usage: ./phan [options] [files...]
 -f, --file-list <filename>
  A file containing a list of PHP files to be analyzed

 -r, --file-list-only
  A file containing a list of PHP files to be analyzed to the
  exclusion of any other directories or files passed in. This
  is useful when running Phan from a stored state file and
  passing in a small subset of files to be re-analyzed.

 -l, --directory <directory>
  A directory to recursively read PHP files from to analyze

 -3, --exclude-directory-list <dir_list>
  A comma-separated list of directories for which any files
  included from that directory will not be analysis. Note
  that adding a directory here will not cause its files to
  be parsed.

 -d, --project-root-directory
  Hunt for a directory named .phan in the current or parent
  directory and read configuration file config.php from that
  path.

 -m <mode>, --output-mode
  Output mode from 'text', 'json', 'codeclimate', or 'checkstyle'

 -o, --output <filename>
  Output filename

 -p, --progress-bar
  Show progress bar

 -a, --dump-ast
  Emit an AST for each file rather than analyze

 -e, --expand-file-list
  Expand the list of files passed in to include any files
  that depend on elements defined in those files. This is
  useful when running Phan from a state file and passing in
  just the set of changed files.

 -q, --quick
  Quick mode - doesn't recurse into all function calls

 -b, --backward-compatibility-checks
  Check for potential PHP 5 -> PHP 7 BC issues

 -i, --ignore-undeclared
  Ignore undeclared functions and classes

 -y, --minimum-severity <level in {0,5,10}>
  Minimum severity level (low=0, normal=5, critical=10) to report.
  Defaults to 0.

 -c, --parent-constructor-required
  Comma-separated list of classes that require
  parent::__construct() to be called

 -x, --dead-code-detection
  Emit issues for classes, methods, functions, constants and
  properties that are probably never referenced and can
  possibly be removed.

 -j, --processes <int>
  The number of parallel processes to run during the analysis
  phase. Defaults to 1.

 -z, --signature-compatibility
  Analyze signatures for methods that are overrides to ensure
  compatiiblity with what they're overriding.

 -h,--help
  This help information

% phan -i -b display.php

display.php:416 CompatError expression may not be PHP 7 compatible

echo preg_replace('/:-:(.*?):-:/e', '$this->pres->\\1', $text);

echo preg_replace_callback(
    '/:-:(.*?):-:/', 
    function($matches) {
      return $this->pres->$matches[1]; // Oops!
    },
    $text);

echo preg_replace_callback(
    '/:-:(.*?):-:/', 
    function($matches) {
      return $this->pres->{$matches[1]}; // Ok
    },
    $text);

% git clone https://github.com/Seldaek/monolog.git
% cd monolog
% find . -name '*.php' | grep -v test > filelist.txt
% phan -i -f filelist.txt

./src/Monolog/Handler/ChromePHPHandler.php:178 PhanTypeMismatchReturn Returning type int but headersAccepted() is declared to return bool
./src/Monolog/Handler/ElasticSearchHandler.php:124 PhanTypeMismatchArgumentInternal Argument 3 (previous) is \elastica\exception\exceptioninterface but \runtimeexception::__construct() takes \runtimeexception|\throwable
./src/Monolog/Handler/FirePHPHandler.php:81 PhanTypeMismatchReturn Returning type array but createRecordHeader() is declared to return string
./src/Monolog/Handler/FirePHPHandler.php:153 PhanTypeMismatchArgumentInternal Argument 1 (array_arg) is string but \current() takes array
./src/Monolog/Handler/FirePHPHandler.php:154 PhanTypeMismatchArgumentInternal Argument 1 (array_arg) is string but \current() takes array
./src/Monolog/Handler/FirePHPHandler.php:154 PhanTypeMismatchArgumentInternal Argument 1 (array_arg) is string but \key() takes array
./src/Monolog/Handler/FlowdockHandler.php:70 PhanTypeMissingReturn Method \monolog\handler\flowdockhandler::getdefaultformatter is declared to return \monolog\formatter\formatterinterface but has no return value
./src/Monolog/Handler/GelfHandler.php:55 PhanTypeMismatchProperty Assigning null to property but \monolog\handler\gelfhandler::publisher is \gelf\imessagepublisher|\gelf\publisher|\gelf\publisherinterface
./src/Monolog/Handler/MandrillHandler.php:49 PhanSignatureMismatch Declaration of function send($content, array $records) should be compatible with function send(string $content, array $records) defined in ./src/Monolog/Handler/MailHandler.php:46
./src/Monolog/Handler/NativeMailerHandler.php:117 PhanSignatureMismatch Declaration of function send($content, array $records) should be compatible with function send(string $content, array $records) defined in ./src/Monolog/Handler/MailHandler.php:46
./src/Monolog/Handler/RedisHandler.php:41 PhanTypeMismatchDefault Default value for int $capSize can't be bool
./src/Monolog/Handler/SocketHandler.php:115 PhanTypeMismatchProperty Assigning float to property but \monolog\handler\sockethandler::timeout is int
./src/Monolog/Handler/SocketHandler.php:126 PhanTypeMismatchProperty Assigning float to property but \monolog\handler\sockethandler::writingTimeout is int
./src/Monolog/Handler/SocketHandler.php:218 PhanTypeMismatchArgumentInternal Argument 2 (seconds) is float but \stream_set_timeout() takes int
./src/Monolog/Handler/SocketHandler.php:218 PhanTypeMismatchArgumentInternal Argument 3 (microseconds) is float but \stream_set_timeout() takes int
./src/Monolog/Handler/SocketHandler.php:274 PhanTypeMismatchProperty Assigning resource to property but \monolog\handler\sockethandler::resource is null
./src/Monolog/Handler/StreamHandler.php:65 PhanTypeMismatchProperty Assigning null to property but \monolog\handler\streamhandler::stream is resource|string
./src/Monolog/Handler/StreamHandler.php:86 PhanTypeMismatchProperty Assigning null to property but \monolog\handler\streamhandler::stream is resource|string
./src/Monolog/Handler/StreamHandler.php:105 PhanTypeMismatchProperty Assigning array|string to property but \monolog\handler\streamhandler::errorMessage is null
./src/Monolog/Handler/SwiftMailerHandler.php:43 PhanSignatureMismatch Declaration of function send($content, array $records) should be compatible with function send(string $content, array $records) defined in ./src/Monolog/Handler/MailHandler.php:46
./src/Monolog/Handler/SyslogUdp/UdpSocket.php:38 PhanTypeMismatchProperty Assigning null to property but \monolog\handler\syslogudp\udpsocket::socket is resource

ChromePHPHandler.php:178 PhanTypeMismatchReturn Returning type int but headersAccepted() is declared to return bool

/**
 * Verifies if the headers are accepted by the current user agent
 *
 * @return Boolean
 */
protected function headersAccepted() {
    if (empty($_SERVER['HTTP_USER_AGENT'])) {
        return false;
    }
    return preg_match('{\bChrome/\d+[\.\d+]*\b}', $_SERVER['HTTP_USER_AGENT']);
}

FirePHPHandler.php:154 PhanTypeMismatchArgumentInternal Argument 1 (array_arg) is string but \current() takes array

/**
 * Base header creation function used by init headers & record headers
 *
 * @param  array  $meta    Wildfire Plugin, Protocol & Structure Indexes
 * @param  string $message Log message
 * @return array  Complete header string ready for the client as key and message as value
 */
protected function createHeader(array $meta, $message) {
    $header = sprintf('%s-%s', self::HEADER_PREFIX, join('-', $meta));

    return array($header => $message);
}

/**
 * Creates message header from record
 *
 * @see createHeader()
 * @param  array  $record
 * @return string
 */
protected function createRecordHeader(array $record)
{
    // Wildfire is extensible to support multiple protocols & plugins in a single request,
    // but we're not taking advantage of that (yet), so we're using "1" for simplicity's sake.
    return $this->createHeader(
        array(1, 1, 1, self::$messageIndex++),
        $record['formatted']
    );
}

/**
 * Creates & sends header for a record, ensuring init headers have been sent prior
 *
 * @see sendHeader()
 * @see sendInitHeaders()
 * @param array $record
 */
protected function write(array $record)
{
    if (!self::$sendHeaders) {
        return;
    }

    // WildFire-specific headers must be sent prior to any messages
    if (!self::$initialized) {
        self::$initialized = true;

        self::$sendHeaders = $this->headersAccepted();
        if (!self::$sendHeaders) {
            return;
        }

        foreach ($this->getInitHeaders() as $header => $content) {
            $this->sendHeader($header, $content);
        }
    }

    $header = $this->createRecordHeader($record);
    if (trim(current($header)) !== '') {
        $this->sendHeader(key($header), current($header));
    }
}

Let's deploy it!

Atomic

No performance hit

• No restarts
• No LB removal
• No thundering herd
• Cache reuse

Must be able to serve two versions of the site concurrently!

Requests that begin on DocumentRoot A must finish on A

Set the DocumentRoot to symlink target!

Easy with nginx

fastcgi_param DOCUMENT_ROOT $realpath_root

Apache

Avoid hardcoding full paths

Watch your include_path setting

incpath extension can resolve your include_path for you

Version all static assets

DB Schema changes need special care

How do you manage deploys?

At Etsy we use irc

Channel: #push Topic: <prod> *joe frank|bob
devbot: Swapping symlinks. Your code is about to start taking production traffic
pushbot: joe frank : Your code is live. Time to watch graphs: http://etsy/abcd
Rasmus: .join
*** pushbot has changed the topic on #push to <prod> joe frank|bob Rasmus
frank: .good
*** pushbot has changed the topic on #push to <prod> *joe *frank|bob Rasmus
joe: .done
*** pushbot has changed the topic on #push to <prod> bob Rasmus
pushbot: bob Rasmus: You're up
bob: .in
*** pushbot has changed the topic on #push *bob Rasmus
Rasmus: .in
*** pushbot has changed the topic on #push *bob *Rasmus

pushbot commands

• .join    - join push queue
• .in        - code has been pushed
• .good - your stuff looks good
• .uhoh - your stuff looks bad
• .hold  - there is a problem, hold everything
• .nm     - never mind (leave queue)
• .done - push done

Channel: #push Topic: <princess> bob Rasmus
Jenkins: Starting build #36803 for job qa
Jenkins: Starting build #38784 for job princess
Jenkins: Project qa build #36803: SUCCESS in 6 min 19 sec: http://ci/job/qa/36803/
pushbot: bob Rasmus : qa tests have passed
devbot: [who_tried] Everyone in this push has run Try recently. w00t!
Jenkins: Project princess build #38784: SUCCESS in 1 min 10 sec: http://ci/job/princess/38784/
pushbot: bob Rasmus : princess tests have passed
bob: .good
Rasmus: .good
*** pushbot has changed the topic on #push to <princess> *bob *Rasmus
pushbot: bob Rasmus : everyone is ready, checking on Jenkins...
Jenkins: qa: last build: 36803 (9 min 5 sec ago): SUCCESS: http://ci/job/qa/36803/
Jenkins: princess: last build: 38784 (2 min 54 sec ago): SUCCESS: http://ci/job/princess/38784/

Deploy to Production:

• ssh to deploy host
• dsh to all targets
• rsync files

Channel: #push Topic: <prod> bob Rasmus
devbot: Swapping symlinks. Your code is about to start taking production traffic
pushbot: bob Rasmus : Your code is live. Time to watch graphs: http://etsy/et5cp
Jenkins: Starting build #39452 for job prod
pushbot: bob Rasmus : prod tests have passed
Jenkins: Project prod build #39452: SUCCESS in 30 sec: http://ci/job/prod/39452/
bob: .good
Rasmus: .good
*** pushbot has changed the topic on #push to <prod> *bob *Rasmus
pushbot: bob Rasmus : everyone is ready, checking on Jenkins...
Jenkins: prod: last build: 39452 (1 min 39 sec ago): SUCCESS: http://ci/job/prod/39452/
bob: .done
pushbot: clear
*** pushbot has changed the topic on #push to clear

Graph Everything!

• Statsd
• Ganglia
• Graphite

Log Everything!

• logster
• Supergrep
• Logstash
• Elastic Search

• Commit to master
• Deploy from HEAD
• Branches?
• Branches are in code via feature flags

Blameless post-mortems

PHP 7 in production

PHP 7 Tuning

Opcache

opcache.memory_consumption=2048
opcache.max_accelerated_files=100000
opcache.validate_timestamps=1
opcache.revalidate_freq=2
opcache.save_comments=0
opcache.enable_file_override=0
opcache.enable_cli=0
opcache.max_wasted_percentage=10
opcache.interned_strings_buffer=128
opcache.fast_shutdown=1
opcache.huge_code_pages=1
opcache.optimization_level=0x7FFFBFFF

Huge Pages

$ sysctl -w vm.nr_hugepages=512
vm.nr_hugepages = 512
(Add it to your /etc/sysctl.conf)

$ grep Huge /proc/meminfo
AnonHugePages:      6144 kB
HugePages_Total:     512
HugePages_Free:      300
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

increase realpath_cache_size

realpath_cache_size=128k

If using MySQL, use mysqlnd

Check your command buffer usage

DocumentRoot on tmpfs

$ mount | grep tmpfs
tmpfs on /var/www type tmpfs (rw,relatime,size=12288000k,mode=755)

$ ls -la /var/www
total 5
drwxr-xr-x  5 root   root    160 Feb 23 02:47 .
drwxr-xr-x 26 root   root   4096 Feb  7 19:40 ..
lrwxrwxrwx  1 root   root     14 Feb 23 02:47 current -> /var/www/A
drwxrwxr-x 25 apache apache  640 Feb 11 22:04 A
drwxrwxr-x 25 apache apache  640 Feb 11 22:04 B

Application-level changes?

Remember this?

$a = [];
for($i=0; $i < 100000;$i++) {
    $a[] = ['abc','def','ghi','jkl','mno','pqr'];
}
echo memory_get_usage(true);

// PHP 5.x  109M
// PHP 7.0   42M no opcache
// PHP 7.0    6M with opcache enabled

Use it!

include 'config.php'; // $config = [ ... ]

include 'countries.php'; // $countries = [ 'CA'=>'Canada', ... ]

Hyperthreading and NUMA

• HyperThreading handles extreme loads better
• If you don't have multi-socket servers, turn on HT and move on
• For multi-socket servers, things get interesting

Digital Ocean

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
Stepping:              2
CPU MHz:               1797.917
BogoMIPS:              3595.83
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0,1

Multi-socket bare metal without HT

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    1
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:              2
CPU MHz:               1203.320
BogoMIPS:              5005.24
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-11
NUMA node1 CPU(s):     12-23

Multi-socket bare metal with HT

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:              2
CPU MHz:               1200.000
BogoMIPS:              5004.73
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-11,24-35
NUMA node1 CPU(s):     12-23,36-47

Solutions?

• numactl --interleave=all httpd/php-fpm
• split multi-socket with containers
• BIOS Snoop Mode setting? HS/ES/COD?
• ignore it

Using Gearman?

Check out Driveshaft!

• Manages pools of workers
• Registers jobs with Gearmand for each pool
• Jobs are run by hitting an endpoint over HTTP/S

{
    "gearman_servers_list":
    [
        "localhost"
    ],
    "pools_list":
    {
        "ShopStats":
        {
            "job_processing_uri": "http://localhost/job.php",
            "worker_count": 20,
            "jobs_list":
            [
                "ShopStats"
            ]
        },
        "Newsfeed":
        {
            "job_processing_uri": "http://localhost/job.php",
            "worker_count": 10,
            "jobs_list":
            [
                "Newsfeed"
            ]
        },
        "Regular":
        {
            "job_processing_uri": "http://localhost/job.php",
            "worker_count": 5,
            "jobs_list":
            [
                "Sum3",
                "Sum",
                "Sum2"
            ]
        }
    }
}

Thank You

Report Bugs

Useful bug reports, please!