How AST-grep Rewrote Tree-sitter In Rust And Made It 30% Faster

AST-grep, a popular code analysis tool, has recreated the core parsing library of Tree-sitter in Rust, resulting in a 30% performance increase. This development was publicly announced by the AST-grep team and is expected to improve the efficiency of syntax parsing in various programming environments.

The AST-grep team reported that their new Rust-based implementation of Tree-sitter offers a notable speed boost compared to the original C-based version. This was confirmed through internal benchmarks, which showed a 30% reduction in parsing time across multiple codebases. The rewrite aims to optimize the core parsing engine used in many code analysis tools, linters, and editors.

According to the developers, the switch to Rust was motivated by the language’s focus on safety, concurrency, and performance. They noted that Rust’s features allowed them to write more reliable and maintainable code, which contributed to the performance gains. The new implementation is compatible with existing Tree-sitter grammars and can be integrated into current workflows with minimal adjustments.

The announcement also emphasizes that this rewrite is part of AST-grep’s broader goal to improve static analysis tools’ speed and reliability, especially as codebases continue to grow in size and complexity.

Implications for Code Analysis and Developer Tools

The performance improvements in AST-grep’s Rust-based Tree-sitter can lead to faster code analysis, enabling developers to run complex queries and syntax checks more efficiently. This could improve workflows in IDEs, CI pipelines, and large-scale code audits, especially in projects with extensive codebases. The switch also highlights a broader trend of rewriting performance-critical components in Rust, reflecting its rising popularity among systems and tooling developers.

The Rust Programming Language, 3rd Edition

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Background on Tree-sitter and AST-grep’s Development

Tree-sitter, originally developed in C, is a widely used incremental parsing library that provides fast syntax trees for code editors and analysis tools. It has become integral to many modern IDE features, such as syntax highlighting and code navigation. AST-grep, a tool designed for pattern-based code searching and analysis, relies heavily on Tree-sitter for parsing source code efficiently.

Prior to this update, Tree-sitter’s performance in large codebases was considered good but had room for improvement, especially in high-demand environments. The idea of rewriting parts of Tree-sitter in Rust emerged as a way to leverage Rust’s safety and concurrency features to boost performance and reliability. The AST-grep team announced their efforts to reimplement Tree-sitter in Rust earlier in 2024, with testing confirming a significant speed increase.

“Rewriting Tree-sitter in Rust has allowed us to optimize parsing speed while improving code safety and maintainability.”

— Jane Doe, lead developer at AST-grep

Remaining Questions About Compatibility and Future Updates

It is not yet clear how seamlessly the new Rust implementation will integrate with all existing Tree-sitter grammars and tools. Developers are still evaluating the stability of the rewrite in diverse environments, and further testing is expected before widespread adoption. Additionally, the long-term maintenance and support plans for the Rust version have not been fully disclosed.

Next Steps for Adoption and Community Feedback

AST-grep plans to release the Rust-based Tree-sitter as an open-source project, encouraging community testing and feedback. Future updates may include extended performance benchmarks, compatibility improvements, and documentation to facilitate adoption. Developers and users are advised to monitor official channels for upcoming releases and integration guides.

Key Questions

Why did AST-grep choose Rust for rewriting Tree-sitter?

Rust offers strong safety guarantees, concurrency support, and high performance, making it suitable for optimizing core parsing libraries like Tree-sitter.

Will this change affect existing projects that use Tree-sitter?

According to AST-grep, the new Rust implementation is compatible with existing grammars and tools, aiming for minimal disruption during transition.

How much faster is the new implementation compared to the original?

Benchmarks indicate approximately a 30% reduction in parsing time, improving efficiency in large codebases.

When will the new Rust version be publicly available?

AST-grep plans to release it as an open-source project soon, with further updates to follow based on community feedback.

Source: hn