![]() ![]() To prevent duplicate entries we included a timestamp with each change. In the event of node failure or misconfiguration it was possible for multiple bridge nodes to be live simultaneously. This bridge service was deployed on Kubernetes within our infrastructure, consumed the Kyoto Tycoon update feed, and wrote the batched changes every 500ms. Our bridge replicated from Kyoto Tycoon, sending write and delete commands to our Quicksilver root node. To make this possible we decided to build a bridge - QSKTBridge. Once we had evaluated its performance and scalability, we would gradually phase out reads and writes to Kyoto Tycoon, until only Quicksilver was handling production traffic. Our goal in deploying Quicksilver was to run it in parallel with our then existing KV distribution tool, Kyoto Tycoon. Ultimately many, many, things did go wrong, and every bit of failure tolerance put into the system proved to be worth its weight in gold because none of this was visible to customers. We needed a way to deploy our new service seamlessly, and to roll back that deploy should something go wrong. We needed to deploy it to production into a network which was designed to be fault tolerant and in which downtime was unacceptable. Building it, however, was only a small part of the challenge. In a previous post, we discussed why it was built and what it was replacing. ![]() Quicksilver is the data store responsible for storing and distributing the billions of KV pairs used to configure the millions of sites and Internet services which use Cloudflare. For other systems, such as the core distributed data store which keeps millions of websites online, it’s a bit more of a challenge. For some systems this can be rather easy, spin up a new web server or load balancer, redirect traffic and you’re done. One of the great arts of software engineering is making updates and improvements to working systems without taking them offline. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |