How do you manage deploys? ok, but how do you manage deploys with hundreds of developers? At Etsy we use --irc-- Slack *joe frank|bob devbot: Swapping symlinks. Your code is about to start taking production traffic pushbot: joe frank : Your code is live. Time to watch graphs: http://etsy/abcd Rasmus: .join *** pushbot has changed the topic on #push to joe frank|bob Rasmus frank: .good *** pushbot has changed the topic on #push to *joe *frank|bob Rasmus joe: .done *** pushbot has changed the topic on #push to bob Rasmus pushbot: bob Rasmus: You're up bob: .in *** pushbot has changed the topic on #push *bob Rasmus Rasmus: .in *** pushbot has changed the topic on #push *bob *Rasmus ]]> .join in #push pushbot commands *.join*    - join push queue *.in*        - code has been pushed *.good* - your stuff looks good *.uhoh* - your stuff looks bad *.hold*  - there is a problem, hold everything *.nm*     - never mind (leave queue) *.done* - push done bob Rasmus Jenkins: Starting build #36803 for job qa Jenkins: Starting build #38784 for job princess Jenkins: Project qa build #36803: SUCCESS in 6 min 19 sec: http://ci/job/qa/36803/ pushbot: bob Rasmus : qa tests have passed devbot: [who_tried] Everyone in this push has run Try recently. w00t! Jenkins: Project princess build #38784: SUCCESS in 1 min 10 sec: http://ci/job/princess/38784/ pushbot: bob Rasmus : princess tests have passed bob: .good Rasmus: .good *** pushbot has changed the topic on #push to *bob *Rasmus pushbot: bob Rasmus : everyone is ready, checking on Jenkins... Jenkins: qa: last build: 36803 (9 min 5 sec ago): SUCCESS: http://ci/job/qa/36803/ Jenkins: princess: last build: 38784 (2 min 54 sec ago): SUCCESS: http://ci/job/princess/38784/ ]]> Deploy to Production: ssh to deploy host dsh to all targets rsync files bob Rasmus devbot: Swapping symlinks. Your code is about to start taking production traffic pushbot: bob Rasmus : Your code is live. Time to watch graphs: http://etsy/et5cp Jenkins: Starting build #39452 for job prod pushbot: bob Rasmus : prod tests have passed Jenkins: Project prod build #39452: SUCCESS in 30 sec: http://ci/job/prod/39452/ bob: .good Rasmus: .good *** pushbot has changed the topic on #push to *bob *Rasmus pushbot: bob Rasmus : everyone is ready, checking on Jenkins... Jenkins: prod: last build: 39452 (1 min 39 sec ago): SUCCESS: http://ci/job/prod/39452/ bob: .done pushbot: clear *** pushbot has changed the topic on #push to clear ]]> Graph Everything! Statsd Grafana Log Everything! Supergrep Logstash Elastic Search mtail Prometheus Commit to master Deploy from HEAD Branches? Branches are in code via feature flags Blameless post-mortems And when things go wrong...