Introduction

THOR (The Hand Book of Open-source Repositories) is a website that provides world-class guidance on open-source contributions. This is very near and dear to my heart. Ever since I was 18 years old in 2019, I watched my friend get selected for G-soc. I struggled to understand how to go about it and I was always wishing I had a medium that would provide a very clear understanding of how to go about open-source. This buggled my mind for 7 years, until I finally sat down to build a website to help other people in the future that would want to understand open-source and how to go about it.

Project setup

I already had Nodejs installed on my computer, which is really all you need to use playwright and passmark. We were provided API keys through the mail, keep the API KEY safe inside your env (add .env to .gitignore)

OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here

In three easy steps, you can setup your project.

1. You run this command. It will create a new folder called my-hackathon-tests with everything Playwright needs:

npm init playwright@latest my-hackathon-tests

Answer the prompts.

1. Move into your project folder

cd my-hackathon-tests

1. Install passmark and dotenv

npm install passmark dotenv

Here is a look at my folder structure, I have three test files in my project.

MY-HACKATHON-TESTS/
├── .sixth/
├── node_modules/
├── test-results/
├── tests/
│   ├── 01-homepage.spec.js
│   ├── 02-guest-flow.spec.js
│   └── 03-navbar.spec.js
├── .env
├── .gitignore
├── package-lock.json
├── package.json
└── playwright.config.js

Testing strategy

Writing tests in passmark is as easy as ABC literally, you write your descriptions in plain english sentences and the AI tries to understand it and do it for you. Although, the shortcoming in passmark is that the AI takes screenshots of your website and makes it's assertions which can burn through credits / tokens.

In my website, I had a flow that was repetitive, essentially, the user lands on the homepage and clicks a button to be taken to the dashboard, I did not want to keep repeating myself writing descriptions and assertions for that flow, so I created a helper function called navigateAsGuest

Now, here is where the shortcomings of passmark comes in because it takes screenshots of your website, repeating this flow of the user landing on the homepage and clicking a button can burn through credit / token very fast, so I landed on a split-mechanism approach, where I used playwright for navigation steps and only used passmark to make assertions. This was done because playwright has direct access to the DOM and can do DOM manipulations which saves on compute credit.

What I Tested

Homepage test: This test was done in a a build-driven development testing strategy because I am the main / lead developer of the website, I know what will be seen on the landing page (a logo, a "continue as guest" button, a large banner image) so I tested for the presence of these using a mixture of playwright and passmark, but I will not advice anyone to do it this way because in tech, we should follow a test-driven development style.

// 01-homepage.spec.js
test("Homepage loads with key elements", async ({ page }) => {
  test.setTimeout(120_000); // give it 2 minutes total

  await page.goto("https://ifeanyi-thor.netlify.app/");
  await page.waitForSelector("text=THOR"); // wait for specific text, not networkidle

  await runSteps({
    page,
    userFlow: "Verify homepage elements",
    steps: [],
    assertions: [
      { assertion: "The heading 'THOR' is visible on the page" },
      { assertion: "The subtitle 'The Handbook of Open-source Repositories.' is visible on the page" },
      { assertion: "There is a hammer/logo icon image visible on the page" },
      { assertion: "There is a large illustrated image of an astronaut cat visible on the left side" },
    ],
    test,
    expect,
  });
});

Guest flow test: This is a tricky test because you have to test for edge cases and user pattern is weird and hard to imagine, but here we are testing for when a user enters the website without authenticating with GitHub, we check that they are not able to use the following features: leave a comment, the markdown playground, but they can still use the sidebar to navigate throughout the documentation.

// 02-guest-flow.spec.js
test("Sidebar link 'introduction' loads the correct page", async ({ page }) => {
  test.setTimeout(60_000);
  await navigateAsGuest(page);
  await page.getByText("introduction").click();
  await page.waitForLoadState("networkidle"); // wait for content

  await runSteps({
    page,
    userFlow: "Verify introduction page content",
    steps: [],
    assertions: [
      { assertion: "The main content area shows the Introduction page content" },
    ],
    test,
    expect,
  });
});

Nav bar test: Here, I checked if the navigation links worked by correctly taking a user to the page they are meant to go to. The dark mode toggle button Oh my Gosh, this stressed me the most because I was trying to test that the dark / light mode toggle button works. So I ended up having to directly manipulate the DOM in the test instead of fighting the button.

// 03-navbar.spec.js
test("Playground nav link shows Authentication Required modal", async ({ page }) => {
  test.setTimeout(60_000);
  await navigateAsGuest(page);
  await page.getByRole("link", { name: "Playground" }).click();
  await page.waitForSelector("text=Authentication Required", { timeout: 10000 });

  await runSteps({
    page,
    userFlow: "Verify auth modal",
    steps: [],
    assertions: [
      { assertion: "A modal is visible on the page" },
      { assertion: "The modal contains the heading 'Authentication Required'" },
      { assertion: "The modal contains the text 'Please sign in with GitHub to use this feature'" },
      { assertion: "A 'Sign in with GitHub' button is visible inside the modal" },
    ],
    test,
    expect,
  });
});

We have three test files, with a single command we can run them

npx playwright test --project chromium

What I learned

Run tests sequentially

Without this, Playwright runs tests in parallel by default which caused our tests to interfere with each other.

// playwright.config.js
export default defineConfig({
  workers: 1,
})

Use passmark only for assertions

// wrong — costs credits just to click a link
steps: [{ description: "Click the Docs link" }]

// right — let Playwright click, let Passmark only assess
steps: []

Direct DOM manipulation when the UI fights you

// when clicking the actual button is unreliable
await page.evaluate(() => {
  document.documentElement.setAttribute("data-theme", "dark");
  localStorage.setItem("theme", "dark");
});

waitForLoadState("networkidle")

// waits for ALL network requests to finish
await page.waitForLoadState("networkidle");

Resources

Github: https://github.com/MasterIfeanyi/thor-passmark-tests

THOR: https://ifeanyi-thor.netlify.app

Conclusion

I have written unit tests before this hackathon but it was a long time ago back in 2022 and this is the first time since 2022 that I am writing unit tests again. Oh, this really reminded me of how much I have missed it. This was the first time I saw that I can create multiple test files and they would all still run in parallel or sequentially. Seven years ago I watched my friend get into GSoC and had no roadmap to follow. I built THOR so nobody else has to feel that way. This hackathon pushed me to do something I hadn't done since 2022, write tests and it reminded me that shipping a product is only half the job. The other half is proving it works. Passmark made that second half genuinely enjoyable. If you have built something you care about, test it. Future you will be grateful.