ZKstack CLI: Rebuilding the Developer Toolchain for ZKsync's Modular L2 Stack

ZKsync Era is an Ethereum Layer 2 that uses zero-knowledge proofs to batch transactions and post validity proofs back to L1. Matter Labs extended this into ZKstack: an open-source framework that lets teams deploy their own ZK chains sharing the same proving infrastructure and bridging to Ethereum.

Running a ZK chain requires orchestrating dozens of components: an L2 server, a prover pipeline, L1 smart contracts, an external node, a contract verifier, and supporting infrastructure like databases and monitoring. Before this project, spinning up this stack meant running dozens of TypeScript scripts in sequence, editing environment variables by hand, and debugging failures caused by ordering dependencies and missing configuration.

The existing zk CLI managed all of this, but it was breaking under the weight of ZKstack's growth:

• Slow and fragile tooling: The TypeScript CLI was slow to execute and hard to maintain. As ZKstack expanded, the gap between what the tool could handle and what teams needed widened.
• Silent misconfiguration: Components pulled configuration from environment variables, which meant configs couldn't be versioned, diffed, or reviewed in PRs. A wrong value wouldn't surface until the component crashed at runtime, never at setup time.
• Hardhat to Foundry migration: Deployment scripts for system contracts were written as Hardhat TypeScript tasks. Migrating to Foundry meant rewriting deployment logic in Solidity while preserving exact contract deployment behavior.
• Full feature parity required: The existing tool covered ecosystem lifecycle, chain management, prover orchestration, contract verification, external node configuration, consensus setup, observability, and CI. The rewrite had to replace all of it, dropping features was not an option.

We started from first principles: model the domain correctly, then build the CLI around that model.

The core architectural insight was separating the ecosystem (L1 contracts, shared config, cross-chain coordination) from the chain (an individual L2 with its own state keeper, prover config, and genesis). This separation did not exist in the original tooling and it shaped every design decision.

We chose Rust for the CLI for three reasons. First, the ZKsync components are written in Rust, so the CLI could share types and config structs directly. Second, Rust's type system enforced correct configuration at compile time rather than failing at runtime. Third, performance: the existing TypeScript tooling had speed limitations that Rust resolved.

Rather than building a monolith, we structured the CLI as a Cargo workspace with distinct crates for the CLI binary, typed configuration, shared utilities, and domain types. This structure lets us replace environment variables with typed, file-based configuration (YAML, TOML, JSON) that could be versioned, diffed, and validated before execution. We migrated components incrementally, one at a time, so each change was independently deployable and testable.

Phase 1: Foundation

The initial contribution established the zk_toolbox project with the core command structure, ecosystem creation and initialization, chain creation, and basic Docker container management. The config layer was built on alloy-rs, with typed deserialization for YAML, TOML, and JSON configuration files.

Alongside the Rust CLI, we set up CI infrastructure from the start, running ecosystem initialization tests on every commit. We added rustfmt and clippy enforcement and integrated Foundry into the Docker build environment so that Solidity contracts could be compiled and tested within the same pipeline.

Phase 2: Prover Infrastructure and Developer Tooling

The prover subsystem involved multiple components and required coordinating several pieces. We built it bottom-up:

• Prover init: handled the full initialization flow: object storage configuration, proof generation settings, and setup key management.
• Setup key generation: (prover generate-sk) automated the cryptographic key generation step of the proving pipeline.
• Prover run: wired up all six prover components (witness generators, provers, compressors) behind a single command.

In parallel, we built developer experience features: the zkup installation script with platform detection and PATH management, Grafana monitoring integration, observability prompts, the contract verifier workflow, and the update command for managing version migrations.

Phase 3: Testing and CI Infrastructure

CI coverage included multi-chain integration tests, state revert and recovery tests, custom token scenarios, fee validation, and upgrade tests, all running through the same CLI developers use locally.

We also replaced sequential CI wait logic with a parallel process collector that reports all process statuses instead of failing on the first error, and added non-interactive chain creation so CI could run without prompts.

Phase 4: Architecture Refinement and CLI Redesign

The final phase focused on two architectural changes.

• CLI unification: We merged the two separate binaries into a single zkstack CLI, unifying commands like zk_inception ecosystem create and zk_supervisor test under one namespace.
• Chain/ecosystem decoupling: was done incrementally. Previously, every chain required a full ecosystem to exist first. We reworked commands so a chain could be created, configured, and run independently, with its own config paths and container setup. Ecosystem-level commands still worked as before, but were no longer a prerequisite.

We also added shell autocompletion generating completion scripts for bash, zsh, and fish, health check improvements with three new monitoring components, Docker-based server running, and the build-transactions subcommand that outputs unsigned transaction JSON for teams using cold storage or multisig signing.

Delivered across ~90 PRs over 8 months (March – November 2024), merged directly into the zksync-era repository.

• Tooling adoption: zkstack became the standard interface for ZKsync chain deployment and testing, replacing the legacy TypeScript scripts entirely. The CLI is used by Matter Labs' internal teams and external teams deploying ZK chains on ZKstack.
• Configuration reliability: The migration from environment variables to typed, file-based configuration eliminated an entire class of silent misconfiguration bugs. Configuration files are version-controlled, diffable, and validated at load time.
• CI coverage: The test infrastructure covers multi-chain scenarios, state reverts, snapshot recovery, custom base tokens, fee calculations, and chain upgrades, all running through the same CLI that developers use locally.
• Architecture flexibility: The chain/ecosystem decoupling enabled a workflow that did not previously exist: teams can now create and initialize a single chain with explicit L1 config paths, without deploying an entire ecosystem.

Developer onboarding

The setup workflow was consolidated into a straightforward sequence.

All code was contributed directly to the zksync-era open-source repository and continues to be maintained as part of the ZKstack ecosystem. The project was a close collaboration with Matter Labs' engineering team, with regular feedback cycles shaping the tool into what their developers and external teams use today.