The Vitess blog

The Vitess blog


Updates and insights from the Vitess team.

Consistent Lookup Vindex: Achieving Data Consistency without 2PC
Vindex # Vitess uses Vindexes (short for Vitess Index) to associate rows in a table with a designated address known as Keyspace ID. This allows Vitess to direct a row to its intended destination, typically a shard within the cluster. Vindexes play a …
Summer 2023: Fuzzing Vitess at PlanetScale
My name is Arvind Murty, and from May to July of 2023, I worked on Vitess via an internship with PlanetScale. I was first introduced to Vitess when I was in high school as a potential open-source project for me to work on. I had been interested in …
Announcing Vitess 19
Announcing Vitess 19 # We're thrilled to announce the release of Vitess 19, our latest version packed with enhancements aimed at improving scalability, performance, and usability of your database systems. With this release, we continue our commitment …
Announcing Vitess 18
Vitess 18 is now Generally Available, with a number of new enhancements designed to improve usability, performance and MySQL compatibility. MySQL Compatibility Improvements # Foreign Keys # In the past, foreign keys had to be managed outside Vitess. …
Announcing Vitess 17
We are pleased to announce the general availability of Vitess 17! Major Themes in Vitess 17 # In this release of Vitess, several significant enhancements have been introduced to improve the compatibility, performance, and usability of the system. GA …
Vitess Security Audit Results
The Vitess Maintainer team is pleased to announce the results of a recent third-party security audit of the Vitess code base. Vitess had previously been audited in 2019. Given the amount of time that has passed, and the magnitude of change during …
Backup & Restore Performance
The performance of backups and restores is a business requirement for Vitess users and an ongoing concern for Vitess maintainers. For sufficiently large databases, if we can't take backups fast enough, we risk missing daily SLAs in a production …
schemadiff: Vitess In-memory Schema Diffing, Normalization, Validation and Manipulation
Introducing schemadiff, an internal library in Vitess that has been one of its best-kept secrets until now. At its core, schemadiff is a declarative, programmatic library that can produce a diff in SQL format of two entities: tables, views, or full …
Vtctld Server API
We are thrilled to discuss VtctldServer, the new gRPC API for performing cluster management operations with vtctld components. This is the (near-) culmination of long, steady migration that began back in Vitess v9 (!!!) and went GA in Vitess v15, so …
Connection pooling in Vitess
Note: The content was originally published at PlanetScale Connection pooling is a commonly used technique in modern applications to manage database connections efficiently. It involves creating a cache of database connections that the application …
Announcing Vitess 16
We are pleased to announce the general availability of Vitess 16! Documentation improvements # In this release the maintainer team has decided to put an emphasis on reviewing, editing, and rewriting the website documentation to be current with the …
Vitess at FOSDEM 2023
Some of the Vitess maintainers traveled to Brussels, Belgium to attend and speak at pre-FOSDEM MySQL Days and at FOSDEM. pre-FOSDEM MySQL Days # When: Feb 2-3 Who: Deepthi Sigireddi, Harshit Gangal, Shlomi Noach What: VTOrc - How Vitess achieves …
Introducing VTAdmin
VTAdmin is now generally available for use! VTAdmin provides both a web client and API for managing multiple Vitess clusters, and is the successor to the now-deprecated UI for vtctld. What is VTAdmin? # VTAdmin is made up of two components: VTAdmin …
Introducing VDiff V2
Vitess is a solution that allows you to infinitely scale MySQL while providing clients and apps with a single logical view of the fleet of MySQL instances comprising any number of Keyspaces and Shards. Vitess also provides the cluster and data …
Announcing Vitess 15
Vitess 15 is now generally available, with a number of new enhancements designed to make Vitess easier to use, more resilient, and easier to scale! VTOrc release # VTOrc, a Vitess-native cluster monitoring and recovery component, is now GA. VTOrc …
VTOrc: Vitess-native Orchestrator
There was an idea. An idea to make Vitess self-reliant. An idea to get rid of the friction between Vitess and external fault-detection-and-repair tools. An idea that gave birth to VTOrc… Both VTOrc and Orchestrator are tools for managing MySQL …
VTGR: Vitess with Group Replication
Update June 12, 2023: This component has been deprecated in v17 and will be removed in v18! We recommend that you instead use VTOrc with the semi_sync durability policy. Introduction # MySQL group replication is a new replication mechanism that was …
Announcing Vitess 14
We are pleased to announce the general availability of Vitess 14. Major Themes # In this new release, major improvements have been made in several areas of Vitess, including usability and reliability. Online DDL is now GA. Gen4 planner is the new …
Vitess at KubeCon+CloudNativeCon EU 2022
Kubecon+CloudNativeCon EU 2022 # When: May 16-20 Who: Deepthi Sigireddi, Harshit Gangal, Manan Gupta What: Vitess project booth, maintainer talk, office hours Event Details # May 16-17 were co-located events. We were there for the main conference …
LFX Spring Term 2022 Experience
My name is Kushal Kumar and I was selected as a LFX mentee for the spring term 2022 under the organization CNCF:Vitess. It was a great learning experience. The 3 month journey was an exciting one where I got to learn about parsers, queries in sql and …
Cut-over logic in vitess schema migrations
Vitess supports managed, non-blocking schema migrations based on VReplication, aptly named vitess migrations. Vitess migrations are powerful, revertible, and failure agnostic. They take an asynchronous approach, which is more lightweight on the …
Announcing Vitess 13
The Vitess maintainers are pleased to announce the general availability of Vitess 13. Major Themes # In this release, Vitess maintainers have made significant progress in several areas, including query serving and cluster management. Compatibility # …
Vitess Schema Tracking
What is Schema Tracking? # In a distributed relational database system, like Vitess, a central component is responsible for serving queries across multiple shards. For Vitess, it is VTGate. One of the challenges this component faces is being aware of …
Log4j RCE 0-day Mitigation
Background # A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was …
Rails that scales - Powered by Vitess
Past - Frameworks without scale # Over the past couple of decades, there has been a steady rise in the complexity of the development stacks that the developers across the globe have been using. The web has advanced from being just HTML files, to also …
LFX experience
I was a part of the LFX Fall Intern program working on Vitess for 2021 and it was a great learning experience for me. I got introduced to Kubernetes in my previous internship through which I came to know about CNCF and then about the LFX program …
Connecteam - our Vitess story
Connecteam is a SaaS company that provides an employee management solution for deskless teams. Over the last couple of years, we’ve been growing tremendously and we recently started to face one of the hardest technical challenges: horizontal scaling. …
Why write a new planner
Query planning is hard # Have you ever wondered what goes on behind the scenes when you execute a SQL query? What steps are taken to access your data? In this article, I'll talk about the history of Vitess's V3 query planner, why we created a new …
Announcing Vitess 12
On behalf of the Vitess maintainers, I am pleased to announce the general availability of Vitess 12. Major Themes # In this release, Vitess Maintainers have made significant progress in several areas, including Gen4 planner, VTAdmin, and other …
Examining query plans in MySQL and Vitess
Originally posted at Andres's blog. Traditional query optimizing is mostly about two things: first, in which order and from where to access data, and then how to then combine it. You have probably seen the tree shapes execution plans that are …
Announcing Vitess 11
On behalf of the Vitess maintainers, I am pleased to announce the general availability of Vitess 11. Major Themes # In this release, Vitess Maintainers have made significant progress in several areas, including Benchmarking, VTAdmin, Schema Tracking, …
Announcing Vitess Arewefastyet
Benchmarking is a critical technique for delivering high performance software. The basic idea behind benchmarking is measuring and comparing the performance of one software version against another. Over the years, many benchmarking techniques have …
Online DDL: why FOREIGN KEYs are not supported
This post explains the inherent problem of running online schema changes in MySQL, on tables participating in a foreign key relationship. We'll lay some ground rules and facts, sketch a simplified schema, and dive into an online schema change …
A new Protocol Buffers generator for Go
Although the main interface between applications and a Vitess database is through the MySQL protocol, Vitess is a large and complex distributed system, and all the communication between the different services in a Vitess cluster is performed through …
Announcing Vitess 10
On behalf of the Vitess maintainers, I am pleased to announce the general availability of Vitess 10. Major Themes # In this release, Vitess Maintainers have continued to focus on compatibility. It is still the most critical component of Vitess being …
Code generation in Vitess
Cross posting link Golang is a wonderful language. It's simple, and most of the time not confusing or surprising. This makes it easy to jump into library code and start reading and quickly understand what's going on. On the other hand, coming from …
Online DDL in Vitess
Vitess introduces a new way to run schema migrations: non-blocking, asynchronous, scheduled online DDL. With online DDL Vitess simplifies the schema migration process by taking ownership of the operational overhead, and providing the user a simple, …
Announcing Vitess 9
On behalf of the Vitess maintainers team, I am pleased to announce the general availability of Vitess 9. Major Themes # In this release, we have focused on making Vitess more stable after the successful release of Version 8. There have been no major …
Kubecon+CloudNativeCon NA 2020
The Vitess team had a successful presence at Kubecon + CloudNativeCon North America 2020. This year's event was in a virtual format. It ran from Nov 17-20. We had 2 talks, a project booth and 3 office hours sessions. In addition, we had mentions in 2 …
Django with Vitess
Django is a popular framework for Python application developers. It includes packages which make tasks like authorization and content administration easier. Django supports a number of databases including MySQL which makes it possible to run a Django …
Vitess Operator for Kubernetes
Introduction # In this blog, I would like to explore Vitess Operator for Kubernetes. This post demonstrates the sample implementation of Vitess in Kubernetes topology. I also explore common DBA tasks by demonstrating how they are handled in the …
Streaming Vitess at Bolt
Previously posted on link at Nov 3, 2020. Traditionally, MySQL has been used to power most of the backend services at Bolt. We've designed our schemas in a way that they're sharded into different MySQL clusters. Each MySQL cluster contains a subset …
Announcing Vitess 8
On behalf of the Vitess maintainers team, I am pleased to announce the general availability of Vitess 8. Major Themes # In this release, we have continued to make important improvements to the Vitess project with over 200 PRs in several areas. Some …
Announcing Vitess 7
On behalf of the Vitess maintainers team, I am pleased to announce the general availability of Vitess 7. Major Themes # Improved SQL Support # We continued to progress towards (almost) full MySQL compatibility. The highlights in Vitess 7 are replica …
Vitess: The Cross Cell Connection
This post explains about how VTGate handles cross-cell operations and how to configure CellAlias for cross-cell read operations. If you are new to Vitess, it is recommended to read this blog post to get more familiar with the various components and …
Announcing Vitess 6
I am excited to announce the general availability of Vitess 6, the second release to follow our new accelerated release schedule. While only 12 weeks have elapsed since the previous release, it feels like a few key investments have started to pay …
Life of a Vitess Cluster
This post goes into the details of what goes on behind the scenes when a cluster is brought up, for example using helm or the local installation guide. This can be used both as a learning tool and troubleshooting guide. We assume that you have …
Announcing the Release of Vitess 5.0
On behalf of the Vitess maintainers team, I am pleased to announce the immediate release of Vitess 5.0. Vitess 5.0 is largely a clean up release, where we have both deprecated and removed some minor features, and improved both the installation …
Vitess Graduation Retrospective
Last November, Vitess became the eighth CNCF project to reach graduation, joining a host of amazing projects such as Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, and Jaeger. To contextualize this milestone, I picked some tidbits from …
Vitess 4.0 has been released!
Head to the release notes for an overview of functionality added or changed, as well as important changes for those upgrading from earlier releases. We wanted to use this post as an opportunity to reflect on three aspects of Vitess’ development, as …
Why does Vitess recommend 250GB per MySQL server?
Vitess has an opinionated approach to database scalability. Some of those opinions have minimal controversy such as how durability should be provided via replication, but the one I find interesting is the 250GB per MySQL server recommendation. Is …
The Benefits of Unsharded Vitess
For many large companies seeking help with horizontal scaling, Vitess' value proposition is easily understood; running stateful workloads at astronomical scale is a hard problem that Vitess has boldly solved in the past. However, for businesses that …
2019 MySQL Community Contributor Award Program
Building the Vitess community has been our pride and joy. Being able to contribute to the MySQL community, even more so. Vitess' Sugu Sougoumarane has been nominated by the MySQL group for Oracle's 2019 MySQL Community Contributor Award Program, …
Vitess at GoSV
When Vitess started at YouTube back in 2010, it was designed to work in Google’s Borg. As a result of that interesting bit of history (and the Google origin story), Vitess wound up being one of the earliest adopters of (a) Go and (b) Kubernetes. At …
Vitess Security Audit Results
We are happy to announce that Vitess recently underwent a security assessment, funded by CNCF / The Linux Foundation. In February 2019, the team from Cure53 performed tests in the following areas: system complexity cloud infrastructure source code …
Choosing a Primary Vindex
When we talk to Vitess, it appears as if we are talking to a single unit, so what sets up this seemingly magical feature? This is achieved through a VSchema, the beautiful mind that sends all of your queries to the correct shard. When Vitess shards …
Introducing Vitess 3.0
We are pleased to announce Vitess 3.0, a major upgrade from Vitess 2.2. Every major release of Vitess brings new features and improvements for our community, and also sometimes introduces new ways of doing things. Vitess 3.0 is built using “pure” go …
Vitess Weekly Digest - Nov 12 2018
We continue the digest from the Slack discussions for Sep 22 2018 to Oct 5 2018. We've fallen slightly behind on this, but will catch up again soon. Enable VtGateExecute in vtctld # Arsalan [Sep 22nd] Hi, I want to query vitess on vtgate but I have …
Vitess Weekly Digest - Sep 24 2018
This week, we continue the digest from the Slack discussions for Sep 1 2018 to Sep 21 2018. As of this post, we are fully caught up on our backlog. Tracking progress of resharding # jk [Sep 5th] In the SplitClone Stage of resharding, how can i get …
Vitess Weekly Digest - Sep 10 2018
This week, we continue the digest from the Slack discussions for Aug 3 2018 to Aug 31 2018. Secondary Vindexes # raj.veerappan [Aug 3 9:27 AM] how do secondary vindexes work? would they result in further sharding? weitzman [9:32 AM] If you have an …
Vitess Weekly Digest - Aug 24 2018
This week, we continue the digest from the Slack discussions. Update stream # Jian [Jul 25th at 1:27 PM] hi there, I'm new to Vitess, now I'm following the user-guide from vitess.io to explore vitess, in update stream section, I notice they have …
Vitess Weekly Digest - Aug 5 2018
This week, we kick off our new weekly blog updates — bringing you the best of Vitess questions and topics on our Slack discussions. The goal is to show the most interesting topics and requests so those of you just getting started can see highlights …
Custom Sharding With Vitess
Vitess supports a variety of predefined sharding algorithms that can suit different needs. This is achieved by associating a Vindex with your main sharding column. A Vindex essentially provides a mapping function that converts your column value to a …
Vitess releases version 2.1
The Vitess project is proud to announce the release of version 2.1. This version comes packed with new features that improve usability, availability and resilience of the overall system. The release coincides with the Percona Live 2017 Conference, …
Distributed Transactions in Vitess
With Vitess introducing sharding and allowing you to create cross-shard indexes, distributed transactions become unavoidable for certain workloads. Currently, Vitess only supports best-effort distributed transactions. So, it’s possible that a …
Vitess V2: Now with more V3
Starting with Vitess v2.0.0-beta.2, the VTGate V3 API can route complex single-shard queries (containing joins, subqueries, aggregation, sorting, and any combination thereof) as well as perform cross-shard joins. That means you no longer need to tell …
Percona Live featured talk with Sugu Sougoumarane – Vitess: The Complete Story
Cross-posted from Percona Blog. Welcome to the next installment of our talks with Percona Live Data Performance Conference 2016 speakers! In this series of blogs, we’ll highlight some of the speakers that will be at this year’s conference, as well as …
Vitess 2.0 is now beta!
That means we've accomplished all our planned overhauls of client APIs and backward-incompatible protocol changes. See the release notes for what's new. We're now working closely with several users who are evaluating Vitess and providing feedback on …
Cloud Native MySQL Sharding with Vitess and Kubernetes
Cross-posted on Google Cloud Platform Blog. Cloud native technologies like Kubernetes help you compose scalable services out of a sea of small logical units. In our last post, we introduced Vitess(an open-source project that powers YouTube's main …
Scaling MySQL in the cloud with Vitess and Kubernetes
Cross-posted on Google Cloud Platform Blog. Your new website is growing exponentially. After a few rounds of high fives, you start scaling to meet this unexpected demand. While you can always add more front-end servers, eventually your database …