ETCD suddenly shuts down on AWS ECS

[andriihlova]

Hi guys,

I'm using etcd v3.5.5 deployed on AWS ECS using quay.io/coreos/etcd:v3.5.5 image. Everything worked okay but suddenly etcd stopped with an error (see logs below).

1703850824018,"{""level"":""info"",""ts"":""2023-12-29T11:53:44.018Z"",""caller"":""v3compactor/revision.go:86"",""msg"":""starting auto revision compaction"",""revision"":307681,""revision-compaction-retention"":1000}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850824035,"{""level"":""info"",""ts"":""2023-12-29T11:53:44.035Z"",""caller"":""v3compactor/revision.go:94"",""msg"":""completed auto revision compaction"",""revision"":307681,""revision-compaction-retention"":1000,""took"":""16.756196ms""}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850824035,"{""level"":""info"",""ts"":""2023-12-29T11:53:44.035Z"",""caller"":""mvcc/index.go:214"",""msg"":""compact tree index"",""revision"":307681}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850824036,"{""level"":""info"",""ts"":""2023-12-29T11:53:44.036Z"",""caller"":""mvcc/kvstore_compaction.go:66"",""msg"":""finished scheduled compaction"",""compact-revision"":307681,""took"":""239.983µs"",""hash"":2666683961}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850824036,"{""level"":""info"",""ts"":""2023-12-29T11:53:44.036Z"",""caller"":""mvcc/hash.go:137"",""msg"":""storing new hash"",""hash"":2666683961,""revision"":307681,""compact-revision"":307571}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837632,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.632Z"",""caller"":""osutil/interrupt_unix.go:64"",""msg"":""received signal; shutting down"",""signal"":""terminated""}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837632,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.632Z"",""caller"":""embed/etcd.go:371"",""msg"":""closing etcd server"",""name"":""default"",""data-dir"":""/mnt/efsmount/volumes/etcd"",""advertise-peer-urls"":[""http://localhost:2380""],""advertise-client-urls"":[""http://127.0.0.1:80""]}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837634,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.634Z"",""caller"":""etcdserver/server.go:1465"",""msg"":""skipped leadership transfer for single voting member cluster"",""local-member-id"":""8e9e05c52164694d"",""current-leader-member-id"":""8e9e05c52164694d""}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837663,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.663Z"",""caller"":""embed/etcd.go:566"",""msg"":""stopping serving peer traffic"",""address"":""127.0.0.1:2380""}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837664,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.664Z"",""caller"":""embed/etcd.go:571"",""msg"":""stopped serving peer traffic"",""address"":""127.0.0.1:2380""}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc
1703850837664,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.664Z"",""caller"":""embed/etcd.go:373"",""msg"":""closed etcd server"",""name"":""default"",""data-dir"":""/mnt/efsmount/volumes/etcd"",""advertise-peer-urls"":[""http://localhost:2380""],""advertise-client-urls"":[""http://127.0.0.1:80""]}",hatch-duo-prd-milvus_etcd/milvus_etcd/fbb356c64527447dadaed1ccd1e119dc

Sorry maybe I'm missing something. Could you please help with this? Do I need to upgrade ETCD or maybe change some configs ?

[jmhbnz]

Hey @andriihlova - Thanks for your question. Based on the following log line it appears as though the etcd process was terminated from an external source:

1703850837632,"{""level"":""info"",""ts"":""2023-12-29T11:53:57.632Z"",""caller"":""osutil/interrupt_unix.go:64"",""msg"":""received signal; shutting down"",""signal"":""terminated""}"

This error means the etcd process received the SIGTERM signal, refer to this explanation of how signals work.

Note you mention that you are running on ECS. ECS is known to send the SIGTERM signal for application graceful shutdown (https://aws.amazon.com/blogs/containers/graceful-shutdowns-with-ecs) but there could be other reasons why this signal was sent.

[andriihlova]

Great thanks, we'll check our ECS setup.