Thetrainline Bot

Ruby coding challenge: create a scraper for Thetrainline.com.

Requirements

• I used Ruby 3.4.7. Not tested in previous versions.
• Bundler.

How to

Install the library with bundler install

Then, to run the Scraper you can use the bin/run script: bin/run <from> <to> <date> <time>

Run the tests with rspec.

Make sure you check the 100% code coverage with open coverage/index.html.

Available trips

• Madrid - München - 2025-12-01 and 2025-12-02 - departures from 6am to 23:59. bin/run Madrid Munchen 2025-12-01 05:00

• Paris - Bruxelles - 2025-12-01 - departures from 6am to 9am. bin/run Paris Bruxelles 2025-12-01 05:00

• Wien - Barcelona - 2025-12-01 - departures from 6am to 9am. bin/run Wien Barcelona 2025-12-01 05:00

Base directory for files

By default, the Thetrainline module searches for files in the 'static' folder of the project. However you can declare another path with the Environment Variable STATIC_FOLDER.

Static local website

In this challenge it wasn't require to code any anti-bot feature, we are allowed to download a static local copy of the website, for simplicity.

These files can't run any Javascript code, so you can't change your Selected Trip. This means that we can't expand any trip information (to see the whole products and agencies detailed list). The same thing happens with the fares, where the fare name can only be accessed when choosing the fare type (standard or plus).

My first versions ignored this, but then I decided to save multiple .html files per request. Each of these files will have one trip expanded and one fare. Meaning that almost every trip will have 2 files (standard fare and plus).

The bot now search for the files matching the given datetime, then iterating them, getting the full information and the fares.

Different web interfaces

Thetrainline.com seems to have different user interfaces depending on the trip you are requesting.

Ashchurch For Tewkesbury -> Ash has a different UI than London -> Paris, and different than Madrid -> München (the one I am implementing).

However, most of my searches have the same UI, and that's the one I am implementing.

Transformation

This is the information we need:

# Segment
{
      :departure_station => "Ashchurch For Tewkesbury",
           :departure_at => #<DateTime: 2025-04-26T06:09:00+00:00 ((2456774j,22140s,0n),+0s,2299161j)>,
        :arrival_station => "Ash",
             :arrival_at => #<DateTime: 2025-04-26T09:37:00+00:00 ((2456774j,34620s,0n),+0s,2299161j)>,
       :service_agencies => ["thetrainline"],
    :duration_in_minutes => 208,
            :changeovers => 2,
               :products => ["train"],
                  :fares => [...] # See below
}

# Fare
{
                       :name => "Advance Single",
             :price_in_cents => 1939,
                   :currency => "GBP",
                             ...
},

And this is what the website has for every trip:

- Date.
- Departure at.
- Departure station.
- Arrival at.
- Arrval station.
- Duration.
- Changeovers.
- Service agencies.
- Products.
- Fares (Standard | 1st Class): name, price in cents, currency and is_cheapest.

Is cheapest?

This is a property I added because I felt it would be useful for the user.

Improvements

If we wanted to escalate this further, I would:

• Add a Logger: I tried to add as many exceptions as possible, but in this kind of libraries, it's very useful to have a lot of verbosity throught a logger.
• Utils could be split in multiple modules depending on what's helping: datetime, strings, files... It would be useful for new parsers.
• YARD Documentation. We could generate documentation automatically by just documening our methods. This is specially helpful to avoid type issues.
• Automated tests in pre-commit with Git Hooks (I've always used Overcommit).
• Create hash fixtures for matching expected values to avoid duplicating code.