With Rust how do you perform platform testing before publishing to crate.io?

Question

I'm working on a framework which I'm testing on my Mac . Eventually, I'll want to publish to crate.io . I'd like for that to not blow up as a result of poor platform testing. Is there a means to test on all, or at least most, of the current deployment platforms without direct access to those platforms? For example, I don't have access to a Windows box.

Answer 1

If you host your code on github you can set up github actions to build and test your code on multiple platforms.

I have two sets of actions that run on my code.

• one runs the tests, and clippy and checks rust fmt on Linux only for normal pushes and pull requests
• the other runs when setting a release branch and creates a release, runs tests and builds and uploads release binaries for windows, Linux and macOS.

You can see the full files in all their glory here .

But combining these and simplifying means that to get testing on every push and pull request you would create a file at .github/workflows/testing.yml like this (untested):

name: Run Tests

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build_matrix:
    name: Run tests for ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        name: [linux, windows, macos]
        include:
          - name: linux
            os: ubuntu-latest
          - name: windows
            os: windows-latest
          - name: macos
            os: macos-latest
    steps:
    - uses: actions/checkout@v1

    - uses: actions-rs/toolchain@v1
      with:
        profile: minimal
        toolchain: nightly
        override: true

    - name: Test
      run: cargo test

Answer 2

I actually do this in one of my projects, altbinutils . I use Github Actions. What I test are:

• tests for each platform
• builds for each platform
• coverage testing
• format testing

I separate workflows of each platforms to different configuration files because (i) I don't want to have all failed when one fails and (ii) I actually want to present them separately as Shields badges on my README.md to provide comprehensive detail.

You can see the workflows under .github/workflows directory, but again, let me share the configurations here because it might change in the future.

Build Workflows

Build workflows ensure that your code will compile in the target platform but not that it works correctly .

I have three separate workflows for builds. These are:

• build.linux.yml
• build.windows.yml
• build.macos.yml

The contents are:

# build.linux.yml
name: build_linux

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [ubuntu-latest]
        rust: [stable, beta]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: build

# build.windows.yml
name: build_windows

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [windows-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: build

# build.macos.yml
name: build_macos

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [macos-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: build

If you look carefully, you can see I set up stable and beta Rust matrix to ensure my code won't break in the future and only on Linux platform since doing that on others are kind of redundant.

Test Workflows

Test workflows ensure that your code behaves correctly.

I have three three separate workflows for each platform. Those are:

# test.linux.yml
name: test_linux

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [ubuntu-latest]
        rust: [stable, beta]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: test
          args: --lib --workspace

# test.windows.yml
name: test_windows

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [windows-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: test
          args: --lib --workspace -- --nocapture

# test.macos.yml
name: test_macos

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [macos-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: test
          args: --lib --workspace -- --nocapture

The extra thing you can see in these configurations are --workspace flag, which will test all your workspace if you have submembers for your project. In my case, I have separate apps live under subdirectories, which are actually members of the workspace.

Also note that for Windows and MacOS, I have also added -- --nocapture flag, which will actually print out all logs and print statements on those platforms. I feel quite confident about Linux but, for the other platforms, I am not.

Coverage Workflow

Coverage workflow runs tests and calculates the percentage of the lines of code that your tests actually hit. Then it pushes the statistics to Codecov so that you can see comprehensive details.

I have only one workflow for that, which runs on Linux. Only one is enough because coverage on other platforms will lead to (roughly) the same results.

The workflow configuration file is:

# coverage.yml
name: coverage

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [ubuntu-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - name: Generate Coverage
        run: |
          cargo install cargo-tarpaulin
          cargo tarpaulin --verbose --all-features --workspace --timeout 120 --out Xml
      - name: Upload Coverage
        uses: codecov/codecov-action@v1
        with:
          fail_ci_if_error: true

Format Workflow

Format workflow tests if the code is well formatted on new pull requests. You can use it if you think someone else will contribute code to your codebase and you want to keep it clean and readable as much as possible.

The workflow configuration file is:

# format.yml
name: format

on: [push, pull_request]

jobs:
  Test:
    strategy:
      matrix:
        os: [ubuntu-latest]
        rust: [stable]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: ${{ matrix.rust }}
          override: true
      - uses: actions/cache@v2
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions-rs/cargo@v1
        with:
          command: fmt
          args: -- --check

Presenting the Results

You can present the results as badges using Shields . I present them in a Markdown table as below:

|             | Build                                 | Test                                | Format                  | Coverage                    |
| ----------- | ------------------------------------- | ----------------------------------- | ----------------------- | --------------------------- |
| **Linux**   | ![linux build][linux_build_badge]     | ![linux test][linux_test_badge]     | ![format][format_badge] | ![coverage][coverage_badge] |
| **Windows** | ![windows build][windows_build_badge] | ![windows test][windows_test_badge] | -                       | -                           |
| **MacOS**   | ![macos build][macos_build_badge]     | ![macos test][macos_test_badge]     | -                       | -                           |

<!-- START BADGES -->

<!-- linux -->
[linux_build_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/build_linux/master?logo=linux&logoColor=white&style=flat-square
[linux_test_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/test_linux/master?logo=linux&logoColor=white&style=flat-square&label=test
[format_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/format/master?logo=linux&logoColor=white&style=flat-square&label=format
[coverage_badge]: https://img.shields.io/codecov/c/github/erayerdin/altbinutils/master?style=flat-square&token=71P6IZY43W&logo=linux&logoColor=white

<!-- windows -->
[windows_build_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/build_windows/master?logo=windows&logoColor=white&style=flat-square
[windows_test_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/test_windows/master?logo=windows&logoColor=white&style=flat-square&label=test

<!-- macos -->
[macos_build_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/build_macos/master?logo=apple&logoColor=white&style=flat-square
[macos_test_badge]: https://img.shields.io/github/workflow/status/erayerdin/altbinutils/test_macos/master?logo=apple&logoColor=white&style=flat-square&label=test

<!-- END BADGES -->

You can actually present the status of one of your workflow (to show if it succeeded or failed) by changing the URL below to your needs:

https://img.shields.io/github/workflow/status/[your-username]/[your-reponame]/[name-attr-of-config-file]/[branch]

This will generate a badge for you so that people can be sure your code is working or not.