Conversation
|
In Beta browsers there are sometimes A/B tests which can skew the test results. For Chrome, I found https://chromium.googlesource.com/chromium-variations/ and so For Firefox, something similar exists, but I haven't done the research yet how to prevent it from happening there. |
queengooborg
left a comment
There was a problem hiding this comment.
filterVersions is also used in the find-missing-reports script, which notifies us if there is any critical report files missing. We should add a new argument that allows us to specify if we want to include beta releases.
One of my other concerns is that we have no way of tracking if a report came from a beta or stable version of the browser, which could potentially skew our results when running the update-bcd script. It would also make it harder to know if we need to re-run the collector on the stable version of the browser and obtain support results from a stable release. If we could update the filename saved by the Selenium script to clarify it's for a beta release, that would be helpful.
See: #1957 PS: Another way would be to amend the result export with the browser release data, i.e. adding a key diff --git a/scripts/selenium.ts b/scripts/selenium.ts
index 202d1b4c..0ad5f105 100644
--- a/scripts/selenium.ts
+++ b/scripts/selenium.ts
@@ -658,8 +658,9 @@ const run = async (
filename.replace(browser, `${browser}-beta`);
}
log(task, `Downloading ${filename} ...`);
- const report = await (await fetch(downloadUrl)).buffer();
- await fs.writeFile(path.join(RESULTS_DIR, filename), report);
+ const report = await (await fetch(downloadUrl)).json();
+ report['release'] = bcdBrowsers[browser].releases[version];
+ await fs.writeFile(path.join(RESULTS_DIR, filename), JSON.stringify(report));
}
} finally {
driver.quit().catch(() => {}); |
I've been manually collecting results on beta browsers but ideally we want to collect results automatically. I think this PR should do that.
This will help to reduce manual BCD PRs that people usually submit during the beta phase of browsers. See also https://github.com/mdn/community-meetings/blob/main/23-12-11/minutes.md#bcd where Ruth asked about this.