Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
268 lines (195 loc) · 14.7 KB

CONTRIBUTING.md

File metadata and controls

268 lines (195 loc) · 14.7 KB
Raw

Contributing to Framework Tracker

Framework Tracker is part of the e18e.dev community. Want to get involved? Head to our Discord at https://chat.e18e.dev. If you're keen to contribute to the project and get stuck straight into code, we have an open list of good first issues in our GitHub repo: Good First Issues. We keep this list updated and add new issues every week. It's one of the easiest ways to get involved and help push the project forward.

Where to Start

Framework Tracker has a few different areas which you can contribute to:

  • Improving Metrics: Suggest new metrics or improve the way we collect and display existing ones.

  • Documentation: Enhance our documentation site by fixing typos, improving explanations, or adding new sections.

  • UI/UX Improvements: Help us improve the documentation site by suggesting design improvements, fixing bugs, or adding new features.

  • Adding New Frameworks: Help us expand our list of tracked frameworks by adding new ones. See the Adding a New Framework section below for details, as each new framework adds significant maintenance burden.

Project Structure

This is a monorepo managed with pnpm workspaces hosting multiple packages. However, some of the packages are not part of the workspace.

Included in the workspace are the documentation site, built with Astro, and the stats generator, which is a tool for collecting and processing the metrics for each framework.

framework-tracker/
├── packages/
│   ├── docs/            # Astro-based documentation site
│   └── stats-generator/ # Tool for collecting framework metrics

Outside of the workspace, we have individual packages for each framework's starter project and app project. These are used to run the actual measurements for each framework.

framework-tracker/
├── starter-*/       # Metaframeworks configured using default set up
├── app-*/           # Metaframeworks configured for run time tests

Any project marked with starter in the name should be a direct setup of the meta-framework's recommended default configuration. For most of these, we have followed the official getting started guide and used the CLI to set up the project with the recommended path. We often use these to measure dev time performance metrics like build times, dependency counts, and CI performance. It's hard to compare meta-frameworks, so sometimes it's easier to compare the default setup of each framework, as this is what most users will start with and gives a good baseline for comparison. Full details on each starter set up can be found in our docs site methodologies page.

Any project marked with app in the name is a more complex setup on which we often run runtime performance tests. These are often more customized and have more features added on top of the default starter set up. This is because we want to test the performance of each framework under more real world conditions and with more complex features implemented. For example, we might add a blog page with dynamic routing, or a dashboard page with client side interactivity. This allows us to test the performance of each framework under more realistic conditions and see how they perform as the complexity of the app increases.

Having both a starter and app project for each framework allows us to get a more comprehensive view of the performance of each framework across different use cases and levels of complexity and create more fair comparisons.

How Metrics Work

Please note that metrics collection is in early stages and the process is likely to change as we iterate on it. The current process is as follows but is subject to change:

We currently run scripts in CI using GitHub Actions or if the test is more complex we run it locally in container and push the results to the repo.

CI Metrics Collection

The current flow for collecting metrics is as follows:

  1. A PR is merged which triggers the CI Pipeline: generate-stats which uses functions from packages/stats-generator to run measurements for each framework
  2. The CI Pipeline reads the framework config from .github/frameworks.json and runs measurements based on each framework's app and starter config
  3. Repository benchmarks run on Depot; see Default Benchmark Setup for the shared environment and test-specific exceptions.
  4. The collected metrics are passed into the final step which runs the scripts from packages/stats-generator
  5. The stats-generator reads frameworks.json and generates stats only for the configured measurements
  6. Stats are then saved into each package and passed into packages/docs

Framework Configuration

All frameworks are configured in .github/frameworks.json. Each entry specifies what measurements to run:

{
  "name": "astro",
  "displayName": "Astro",
  "frameworkPackage": "astro",
  "starter": {
    "package": "starter-astro",
    "buildScript": "build:astro",
    "buildOutputDir": "dist",
    "measurements": [
      { "type": "install", "runFrequency": 5 },
      { "type": "build", "runFrequency": 5 },
      { "type": "dependencies" }
    ]
  },
  "app": {
    "package": "app-astro",
    "buildScript": "build:app-astro",
    "buildOutputDir": "dist",
    "measurements": [{ "type": "ssrRequestThroughput" }]
  }
},

Container Metrics Collection

Coming soon but will also pull from .github/frameworks.json

Versioning

We track all major and minor versions of meta-framework. When updating we also update all other packages in the repo to the latest as of the date of the meta-framework release.

Right now we update manually but the full tests are ready for sharing we will automate this part

First-party direct dependencies

Every tracked starter includes first-party-dependencies.json: a small package manifest containing the direct dependencies that belong to the framework's own project or ecosystem. It uses the same dependencies and devDependencies sections as the starter's package.json, with the same version ranges. For example:

{
  "dependencies": {
    "astro": "7.1.6"
  },
  "devDependencies": {
    "@astrojs/check": "0.9.10"
  }
}

Include only direct package entries from the starter's package.json; do not include transitive packages. The manifest separates a starter's direct framework packages from its other direct packages. It does not classify the transitive dependency tree.

When changing a starter's dependencies, review this file in the same PR. Add a new first-party package, remove one no longer declared, and leave third-party packages out. Confirm ownership from the package's source project rather than assuming matching names or scopes are first-party.

The validation workflow checks that every entry remains in the same dependency section with the same version as the starter's package.json.

When an upgrade changes the build tool, adapter, starter setup, or benchmark behavior, update the methodology in the same PR using the checklist below.

Keeping the Methodology Up to Date

Update the methodology source in the same PR when adding a meta-framework or changing a project's setup, benchmark behavior, or test environment. Describe what the committed configuration actually runs, including changes introduced by version upgrades.

  • Default Benchmark Setup: Update shared settings when Node, pnpm selection, runner hardware, browser configuration, or dependency installation changes. Put an exception that applies to only one test in that test's section.
  • Framework Build Tools and Adapters: Add or update the framework's UI library, build tool, and runtime adapter/server in the table. Keep table entries short; use the dot points below it for presets, custom server entries, and differences between starter and runtime packages. Mark a missing starter or runtime app explicitly rather than implying it is benchmarked.
  • Dev Time → Project Setups: Record the CLI command and selected options, or documented setup steps, for a new starter. Update them when the starter setup changes, including any changes made after generation.
  • Run Time → Framework Specific Notes: Explain framework-wide rendering choices and why they are needed. Keep build tool and adapter details in the shared table rather than repeating them here.
  • Individual benchmark sections: Document new or changed routes, data, rendering and navigation behavior, measurement tools, repetitions, aggregation, and environment exceptions where relevant. Explain departures from a framework's default behavior and how they affect the comparison.
  • Links and versions: Keep the home, Dev Time, and Run Time pages linked to Default Benchmark Setup. Update affected methodology links if headings change. Exact dependency versions belong in project manifests and lockfiles; routine version updates do not need to be duplicated in the setup table.

Check the documentation against the affected project configuration, benchmark code, and CI workflow. Run pnpm build:docs and check formatting for the edited files. In the PR description, explain any change that affects comparison with previous results.

Adding a New Framework

Adding a new framework increases the maintenance burden, so please open an issue to discuss it before starting work. If approved, follow these steps:

  1. Create the starter package: Add a new directory in packages/ (e.g., packages/starter-my-framework). Set it up using the framework's official CLI or getting started guide with the recommended defaults. The starter should not be added to the pnpm workspace — it has its own independent package.json and lockfile. Pin the core framework dependency to an exact version (e.g., "my-framework": "2.0.0" not "^2.0.0"). Add first-party-dependencies.json with the starter's direct first-party packages.

  2. Create the app package (optional): If runtime performance testing is planned, add an app-* package (e.g., packages/app-my-framework) with a more complex setup that includes features like dynamic routing or client-side interactivity. Make sure the framework dependency is pinned to the same exact version as the starter package.

  3. Add an entry to .github/frameworks.json: Configure the framework's measurements:

    {
  "name": "my-framework",
  "displayName": "My Framework",
  "frameworkPackage": "my-framework",
  "focusedFramework": false,
  "starter": {
    "package": "starter-my-framework",
    "buildScript": "build:my-framework",
    "buildOutputDir": "dist",
    "measurements": [
      { "type": "install", "runFrequency": 5 },
      { "type": "build", "runFrequency": 5 },
      { "type": "dependencies" }
    ]
  }
}

    Set focusedFramework to false for new additions unless the framework is a priority for tracking.

  4. Update the methodology: Follow Keeping the Methodology Up to Date. Add the framework to the build tools and adapters table, document its starter setup, and explain any runtime or test-specific differences.

  5. Test locally: Make sure the framework builds successfully by running the build script from inside the package directory.

  6. CI for sync-version and validate-stats will automatically run on the new framework once it's added to frameworks.json.

  7. Submit a PR: Open a pull request with the new packages, configuration, and methodology updates. Once merged, the CI will automatically pick up the new framework and raise a PR with new metrics.

Adding a Code Comparison Example

The Code Comparison page is driven by markdown files in packages/docs/src/content/code-comparison/<example>/<framework>.md. One file per framework, named after the framework's slug, which is its packages/docs/src/content/devtime/starter-*.json filename without the prefix. Anything else under that folder fails pnpm type-check, including a stray file at the wrong depth or with the wrong extension. A framework with no file gets a placeholder tab, so a partial example still ships.

Each file holds optional frontmatter, optional prose, and one fenced code block per file the task needs:

---
docs: https://svelte.dev/docs/kit/routing
---

```svelte title="src/routes/about/+page.svelte"
<h1>About</h1>
```

The rules the page depends on:

  • title= is the path relative to the starter package root, and the block is the complete file at that path. A reader must be able to write every block verbatim into a fresh copy of packages/starter-<framework> and have the example work with no other edits and no added dependencies. Replace a file the starter already ships rather than showing a diff of it.
  • Keep snippets minimal. No layouts, styling, metadata, or test ids.
  • Prose is for a step the framework needs beyond adding a file. One or two sentences.
  • docs is optional and must be a working URL to that framework's guide for the task.

Adding a new example means adding the folder plus a ## heading and a <CodeComparison example="..." /> line in packages/docs/src/content/docs/code-comparison.mdx. Nothing else changes, so examples can be contributed in parallel.

Verify a snippet before opening the PR, because nothing in CI can. Copy the starter to a scratch directory, pnpm install, write every block into it at its title= path, run pnpm dev, and request the routes the example adds.

Run pnpm format too. Prettier reformats astro, vue, ts, and tsx fences to this repo's style. It leaves svelte fences alone, since prettier-plugin-svelte is not installed here, so format those by hand to match.

Getting Started

To get the project running locally:

  1. Prerequisites: Make sure you have Node.js (v24+) and pnpm installed.

  2. Clone the repo:

    git clone https://github.com/e18e/framework-tracker.git
cd framework-tracker

  3. Install workspace dependencies:

    pnpm install

    This installs dependencies for the workspace packages (docs and stats-generator). The starter-* and app-* packages are not part of the workspace and have their own lockfiles.

  4. Run the docs site locally:

    pnpm dev:docs

  5. Linting and formatting:

    pnpm lint:all        # Run linting across workspace and framework packages
pnpm format          # Format code with Prettier
pnpm type-check:all  # Run type checking across workspace and framework packages