I created a dead simple Conflict Free Replicated Datatype (CRDT) backend and sync engine using plain old Rust and React. This turned out to be a fun experiment in using the windsurf IDE and chat oriented programming (CHOP).
Not familiar with CHOP? It’s a modern way to put together an application through a combination of prompting and coding. The basic flow is to use your code base as context, and then ask an LLM what to implement. You spend your time writing prompts, a prompt is like a requirement you would write for a junior developer. Reading and integrating code is still needed because it hardly ever works perfectly the first time. There are new IDE’s and plugins that lean into this workflow, and for today’s blog I use the Windsurf IDE.
The big goal with this project is twofold. First, I wanted to test out CHOP coding using windsurf. Second, I wanted to create a sync engine which allows me to quickly build single page apps that share state between multiple clients.
What is a CRDT backend and sync engine? A sync engine is a core piece of technology in creating local first software. Previously I looked in to using sqlite compiled to wasm as a state store, but I found this to be quite complex. So I’ve taken a completely different approach with this implementation and built the most simple but useful API possible.
To synchronize any type of state I just need the following three API methods
api/bootstrap this returns ALL the models.api/transactions . After this API it will increment a transaction counter on the server which indicates other clients should update their own state.api/transactions?from=99. A regular interval will poll the backend to get the latest set of transactions.I chose Rust for this backend to better test out CHOP programming. I am no professional rust developer, just a mid coder who has gone through the rustlings and tutorial a couple times. The value in using an LLM based workflow here is that it can help me learn the frameworks and language by building something from scratch.
For keeping the state on the backend I used Sqlite, and the LLM chose actix and tokio as the web framework du jour.
The experience of chopping my way through creating this project was fairly seamless. It took a few tries to get things compiling and I had to do a bit of manual testing with curl to verify things are working as expected. The overall implementation is so simple that I don’t feel like this is any great feat of engineering. The coolest part really is just what I was able to achieve in relatively short amounts of time just by prompting and having a tutorial level knowledge of the language.
I am using sqlite as the database but the data is not relational. My data layer is designed to work with any type of model, so I’m using JSON columns. The schema consists of just two tables, sync_history and model_data. We only do simple queries on this database like getting all models of a type, or getting a list of transactions for a client to apply.
fn new(path : \&str) \-\> Result\<Self\> {
let conn \= Connection::open(path)?;
// Create tables
conn.execute(
"CREATE TABLE IF NOT EXISTS sync\_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
actions TEXT NOT NULL
)",
\[\],
)?;
conn.execute(
"CREATE TABLE IF NOT EXISTS model\_data (
id TEXT PRIMARY KEY,
model\_name TEXT NOT NULL,
data TEXT NOT NULL,
created\_at INTEGER NOT NULL,
updated\_at INTEGER NOT NULL
)",
\[\],
)?;
Ok(Database { conn: Mutex::new(conn) })
}
const { items, createItem, updateItem, deleteItem } \= useItems('Todo');
A component can work with state through the handy useItems API. You just provide it a name for the type of data you are mutating, and it will seamlessly figure out all the conflict resolution and synchronisation in the background automatically.
A simple hook to return the latest data and create, update, delete methods is the best way to interface with a CRDT.
The best libraries expose a narrow interface and a deep amount of usefulness. The above API does this extremely well. Under the hood we have a polling sync engine, optimistic updates and a local database implementation.
The sync engine will poll for updates from the backend. When you apply a change it immediately gets updated in the local database and then queued for sending to the backend when it is online.
One common challenge in CRDT algorithms is how you manage two updates to the same model. We take the simple solution here and just choose the last write as the winner. Yea, it’s possible that this isn’t great in the long run, but I reckon it’s more of an application specific concern if you care about that or not. Everyone knows by now that you shouldn’t be representing bank transactions in CRDT’s anyway!
You can access this little bit of software from my caprover instance at https://todo-rust.zealot.ersin.nz/ .
Caprover is my boring choice in deployment for all my little side hacks. It’s a cool platform that I will write about later in the year, but for now – just know it’s my preferred way to spin up a docker container behind https and nginx.
I am very familiar with CapRover/Docker and github actions, but this is my first time building and deploying a rust program from a github action. For completeness I CHOP’d my way through it. CI/CD is something i love to use, but hate to build so it’s a welcome proposition to have AI do it for me. The overall structure is something pretty standard, a build image plus a deployment image and pushing it all up to GHCR at the end.
Are we learning when we use AI Chat to generate all our code? Yeah — nahh – yea?. I ended up sending a lot of my time reading code. You learn when you read code, but I also think there is benefit to building muscle memory by actually writing the code. I’m more familiar with rust/actix than before. But the fundamentals of Rust are probably not as advanced as spending the same time on writing code from scratch.
The future of learning to code is not just writing prompts for an AI. The fundamentals of syntax, and language semantics are still going to be important going forward. Yet we still need to be pragmatic and deliver value if we are to be professionals. What this means for people learning new languages is that you should balance your time between CHOPing big projects and also still focusing on the low level fundamentals. Remember you can also use the AI as a guide to explain what’s going on semantically, then experiment on your own terms with the compiler.
The sync engine seems solid, but it needs testing under the fire of production use. I am more convinced of the idea that small interfaces and deep implementations are the most useful shape of abstractions. I am not sold on the conflict resolution or polling approach, but it really just needs more applications to be built out to test this.
Some things might not be great with this type of sync engine. A lot of relational data is one. A high number of clients and edits would also be quite challenging for this approach. Needing to be extremely real time like a game engine would also fall over.
But overall, this was a good experiment and I’m already thinking of
As a senior+ engineer, LLMs are powerful tools for prototyping in unfamiliar languages. The new tools are powerful, but not an alternative for learning the actual language and semantics.
You can build a functional CRDT without much fuss if you keep it simple. This might be pragmatic and easy to achieve for your application. Getting rid of the complex parts like conflict algorithms and relational data models allowed us to do this.
The best abstractions are small, with deep and complex implementations. These kind of abstractions are the ones you can use across more places, and replace more easily when they are highly depended on. In our simple program here, we have a simple backend interface and a simple react API interface. As a result, it’s something that can have bits swapped out easily.