The Filecoin System Test Matrix is a dashboard with a detailed list of Venus features and behaviors and a mapping between those features and test suites, systems, and subsystems that those features are related to. The main goal of this project is to close the discrepancy between the specification and the actual state of implementation. The System Test Matrix can be used to evaluate the state of Venus at a glance and help new contributors understand how the Filecoin network works.
1. A catalog of unique expected behaviors of a Venus node. This catalog is in YAML format and acts as a specification of how different subsystems and subsystems of Venus should behave in given situations with different inputs and is a starting point for writing tests. The Bloxico team provided the base catalog of Lotus behaviors and supported the Venus team with extending the catalog with Venus specific behaviors, and guided the process of exploring and understanding how the architecture of Venus can be mapped to STM systems and subsystems. The quality of this catalog highly impacts the quality of the end results of STM. The Bloxico team was expected to provide a starting point and support for the Venus team.
2. Integration of Go test crawler with the Venus codebase. The Test Crawler scans the repository for tests and parses test annotations (written in our custom format) that map test functions to test behaviors from the previous step. It outputs a JSON report. The Bloxico team supported the Venus developers with adding test annotations to existing test files and helped out with building the habit of annotating future test files, so we can keep the System Test Matrix up to date with each new release. With support from Bloxico test engineers, the Venus developers, for the purpose of documenting the rationale behind existing tests and what behaviors were already tested, annotated all of the test files and produced test crawler reports that are now used to visualize the state of Venus.
We deployed a new Venus-specific dashboard to venus.systemtestmatrix.com. This dashboard should help the Venus developers visualize the current state of Venus, and it’s rendered using the behavior catalog and test crawler output from the previous steps. The dashboard provides a UI for browsing, searching, and visualizing all of the existing behaviors, tests, subsystems, and components of Venus.
The goal was to do a detailed analysis of the existing test suites and craft a dashboard of test cases (not metrics, at this stage) that breaks down test scenarios by subsystems and components. Each test case should carry a status indicating if it’s not implemented, in progress, succeeding, or failing (per branch) which will show the current state of Lotus tests, that can be used as a baseline to evaluate future improvements.
Increase the integration test assets coverage using the current integration testing kit developed by the Lotus maintainers, in coordination with PL team as main Protocol Labs approvers of the tests Bloxico will create and improve.
Improve and craft new Acceptance & E2E tests that will be used to evaluate new Lotus releases.
Improve the current state of tooling used by the QA team by proactively identifying valuable additions/improvements, proposing them to the PL team, and implementing them based on agreed prioritization.