Feature: Source Code & Script Analysis (Reviving Tree-sitter Integration)

[EdoardoAllegrini]

Description

This PR introduces the ability for capa to analyze source code and scripted languages (Python, JavaScript, C#, HTML, etc.).

This revives the foundational work done by @adamstorek in PR #1080. That PR was stalled primarily due to the brittleness of manually compiling C-bindings across different operating systems.

I have fully rebased the original feature onto the current master and entirely overhauled the underlying Tree-sitter architecture to use modern, pre-compiled OS-native PyPI wheels.

Key Upgrades since #1080:

• No Local C-Compiler Required: Migrated from the deprecated Language.build_library() to pre-compiled native wheels (tree-sitter-python, tree-sitter-c-sharp, etc.).
• Cross-Platform Out of the Box: Works seamlessly on Windows, macOS, and Linux via standard pip install.
• Fully Rebased: Resolved all conflicts with the modern capa feature extraction engine.
• Cleaned up and Updated code logic: Refactored the Tree-sitter initialization sequence to support the modern Language API, updated the AST query bindings to match newer parser syntax, and ensured strict type hinting aligns with current capa standards.

GSoC Context & Next Steps

This PR also serves as the foundational technical contribution for my Google Summer of Code application. By unblocking the Tree-sitter integration now, my proposed GSoC project can focus entirely on expanding the language support, optimizing the extraction performance, and writing comprehensive rule sets for these new scripted environments. I welcome any feedback from the maintainers/mentors (e.g., @mike-hunhoff) on this implementation!

Fixes #1092
Supersedes #1080

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances capa's analytical capabilities by enabling it to process source code and scripted languages. The core of this enhancement is the re-implementation of Tree-sitter integration, which was previously stalled due to compilation complexities. By adopting modern, pre-compiled Python wheels, the project has overcome these hurdles, making the feature robust and cross-platform compatible. This strategic update not only expands the types of files capa can analyze but also lays the groundwork for future development in this area.

Highlights

• Tree-sitter Integration: Introduced the ability for capa to analyze source code and scripted languages (Python, JavaScript, C#, HTML) by integrating Tree-sitter.
• Cross-Platform Compatibility: Migrated from manual C-bindings to pre-compiled OS-native PyPI wheels for Tree-sitter, eliminating the need for local C-compilers and ensuring seamless operation on Windows, macOS, and Linux via standard pip install.
• Codebase Modernization: Fully rebased the original Tree-sitter feature onto the current master branch, resolving conflicts and updating code logic to align with modern capa feature extraction and type hinting standards.
• GSoC Foundation: This pull request serves as the foundational technical contribution for a Google Summer of Code application, enabling future work on expanding language support, optimizing extraction performance, and developing comprehensive rule sets for new scripted environments.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• CHANGELOG.md
  • Added Tree-Sitter Script Analysis to new features.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is an excellent and substantial contribution, reviving and modernizing the tree-sitter integration for source code analysis. The shift to pre-compiled wheels is a major improvement for usability and portability. The overall architecture is well-designed, with clear separation of concerns between parsing, feature extraction, and language-specific logic. The new feature is well-integrated into capa's existing framework.

I've found a couple of critical bugs in the Python language toolkit that need to be addressed, along with a performance issue related to regex compilation and a minor data quality issue in the C# signatures. My detailed comments provide suggestions for fixing these.

Overall, this is a fantastic piece of work that significantly expands capa's capabilities.