How do you manage deploys?
At Etsy we use --irc-- Slack
Channel: #push Topic: <prod> *joe frank|bob
devbot: Swapping symlinks. Your code is about to start taking production traffic
pushbot: joe frank : Your code is live. Time to watch graphs: http://etsy/abcd
Rasmus: .join
*** pushbot has changed the topic on #push to <prod> joe frank|bob Rasmus
frank: .good
*** pushbot has changed the topic on #push to <prod> *joe *frank|bob Rasmus
joe: .done
*** pushbot has changed the topic on #push to <prod> bob Rasmus
pushbot: bob Rasmus: You're up
bob: .in
*** pushbot has changed the topic on #push *bob Rasmus
Rasmus: .in
*** pushbot has changed the topic on #push *bob *Rasmus
pushbot commands
- .join - join push queue
- .in - code has been pushed
- .good - your stuff looks good
- .uhoh - your stuff looks bad
- .hold - there is a problem, hold everything
- .nm - never mind (leave queue)
- .done - push done
Channel: #push Topic: <princess> bob Rasmus
Jenkins: Starting build #36803 for job qa
Jenkins: Starting build #38784 for job princess
Jenkins: Project qa build #36803: SUCCESS in 6 min 19 sec: http://ci/job/qa/36803/
pushbot: bob Rasmus : qa tests have passed
devbot: [who_tried] Everyone in this push has run Try recently. w00t!
Jenkins: Project princess build #38784: SUCCESS in 1 min 10 sec: http://ci/job/princess/38784/
pushbot: bob Rasmus : princess tests have passed
bob: .good
Rasmus: .good
*** pushbot has changed the topic on #push to <princess> *bob *Rasmus
pushbot: bob Rasmus : everyone is ready, checking on Jenkins...
Jenkins: qa: last build: 36803 (9 min 5 sec ago): SUCCESS: http://ci/job/qa/36803/
Jenkins: princess: last build: 38784 (2 min 54 sec ago): SUCCESS: http://ci/job/princess/38784/
Deploy to Production:
- • ssh to deploy host
- • dsh to all targets
- • rsync files
Channel: #push Topic: <prod> bob Rasmus
devbot: Swapping symlinks. Your code is about to start taking production traffic
pushbot: bob Rasmus : Your code is live. Time to watch graphs: http://etsy/et5cp
Jenkins: Starting build #39452 for job prod
pushbot: bob Rasmus : prod tests have passed
Jenkins: Project prod build #39452: SUCCESS in 30 sec: http://ci/job/prod/39452/
bob: .good
Rasmus: .good
*** pushbot has changed the topic on #push to <prod> *bob *Rasmus
pushbot: bob Rasmus : everyone is ready, checking on Jenkins...
Jenkins: prod: last build: 39452 (1 min 39 sec ago): SUCCESS: http://ci/job/prod/39452/
bob: .done
pushbot: clear
*** pushbot has changed the topic on #push to clear
Graph Everything!
Log Everything!
- • Supergrep
- • Logstash
- • Elastic Search
- • mtail
- • Prometheus
Blameless post-mortems