diff --git a/docs/tutorial.md b/docs/tutorial.md index 77620560e7..7404486af9 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -8,7 +8,7 @@ We will run two runs using the `mmlu` scenario on the `openai/gpt2` model. The ` `helm-run` is a command line tool for running benchmarks. -To run this benchmark using the HELM command-line tools, we need to specify **run entries** that describes the desired runs. For this example, the run entries are `mmlu:subject=anatomy,model=openai/gpt2` (for anatomy) and `mmlu:subject=philosophy,model=openai/gpt2` (for philosophy). +To run this benchmark using the HELM command-line tools, we need to specify **run entries** that describe the desired runs. For this example, the run entries are `mmlu:subject=anatomy,model=openai/gpt2` (for anatomy) and `mmlu:subject=philosophy,model=openai/gpt2` (for philosophy). We will now use `helm-run` to execute the runs. Run this command: @@ -27,7 +27,7 @@ The meaning of the arguments are as follows: - The environment directory is `prod_env/` by default and can be set using `--local-path`. Credentials for making API calls should be added to a `credentials.conf` file in this directory. - The output directory is `benchmark_output/` by default and can be set using `--output-path`. -After running this command, navigate to the `benchmark_output/runs/my-suite/` directory. This should contain a two sub-directories named `mmlu:subject=anatomy,model=openai_gpt2` and `mmlu:subject=philosophy,model=openai_gpt2`. Note that the names of these sub-directories is based on the run entries we used earlier, but with `/` replaced with `_`. +After running this command, navigate to the `benchmark_output/runs/my-suite/` directory. This should contain two sub-directories named `mmlu:subject=anatomy,model=openai_gpt2` and `mmlu:subject=philosophy,model=openai_gpt2`. Note that the names of these sub-directories are based on the run entries we used earlier, but with `/` replaced with `_`. Each output sub-directory will contain several JSON files that were generated during the corresponding run: @@ -53,11 +53,11 @@ This reads the pre-existing files in `benchmark_output/runs/my-suite/` that were - `groups.json` contains a serialized list of `Table`, each containing information about groups in a group category. - `groups_metadata.json` contains a list of all the groups along with a human-readable description and a taxonomy. -Additionally, for each group and group-relavent metric, it will output a pair of files: `benchmark_output/runs/my-suite/groups/latex/_.tex` and `benchmark_output/runs/my-suite/groups/json/_.json`. These files contain the statistics for that metric from each run within the group. +Additionally, for each group and group-relevant metric, it will output a pair of files: `benchmark_output/runs/my-suite/groups/latex/_.tex` and `benchmark_output/runs/my-suite/groups/json/_.json`. These files contain the statistics for that metric from each run within the group. ## Using `helm-server` -Finally, the `helm-server` command launches a web server to visualize the output files of `helm-run` and `helm-benchmark`. Run: +Finally, the `helm-server` command launches a web server to visualize the output files generated by `helm-run` and processed by `helm-summarize`. Run: ```sh helm-server --suite my-suite