How to run XCUITests in Microsoft AppCenter from Azure DevOps

Question

I've posted issues in the respective Microsoft Github repositories, but given that they redirected me to Stackoverflow, I'm gonna re-post the question here.

My question is: How to run the automated XCUITests in Microsoft AppCenter from Azure DevOps?

I studied the documentation on how to manually build the app from the console and then upload it to AppCenter (which works). Now I would like to use the official Azure App Center Test task , which should do a similar thing. Unfortunately, these two documentations are significantly different and I have no idea what information I have to provide to that step in order to make it work.

The biggest difference I noticed, is that the AppCenter documentation uses "build-for-testing" and a DerivedData directory where it builds stuff that gets uploaded, whereas the AppCenterTest task requests an IPA, a build directory and a Test IPA path. How to obtain these artifacts?

I tried something like this:

- task: Xcode@5
  inputs:
    actions: 'clean build test'
    configuration: 'Debug'
    sdk: 'iphoneos'
    xcWorkspacePath: 'MyProject/MyProject.xcworkspace'
    scheme: 'MyProject'
    packageApp: true
    exportPath: '$(build.artifactStagingDirectory)/debug'
    signingOption: 'manual'
    signingIdentity: '$(APPLE_CERTIFICATE_SIGNING_IDENTITY)'
    provisioningProfileUuid: '$(APPLE_PROV_PROFILE_UUID)'
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 11'
    publishJUnitResults: true

- task: Xcode@5
  inputs:
    actions: 'build-for-testing'
    configuration: 'Debug'
    sdk: 'iphoneos'
    xcWorkspacePath: 'MyProject/MyProject.xcworkspace'
    scheme: 'MyProject'
    packageApp: false
    args: '-derivedDataPath $(build.artifactStagingDirectory)/DerivedData'

- task: AppCenterTest@1
  inputs:
    appFile: '$(build.artifactStagingDirectory)/DerivedData/Build/Products/Debug-iphoneos/MyProject.app'
    artifactsDirectory: '$(build.artifactStagingDirectory)/AppCenterTest'
    frameworkOption: 'xcuitest'
    xcUITestBuildDirectory: '$(ProjectDir)/Build/Products/Debug-iphoneos'
    xcUITestIpaFile: '$(build.artifactStagingDirectory)/DerivedData/Build/Products/Debug-iphoneos/MyProjectUITests-Runner.app'
    credentialsOption: 'serviceEndpoint'
    serverEndpoint: 'MyCustomer AppCenter Deployment'
    appSlug: 'MyCustomer/MyProject-iOS'
    devices: 'iphoneDevices'
    localeOption: 'en_US'
    skipWaitingForResults: true

but I'm getting an error similar to ##[error]Error: Cannot find any file based on /Users/runner/runners/2.163.1/work/1/a/DerivedData/Build/Products/Debug-iphoneos/MyProject.app

Does anyone have a full working example and could provide an example YAML file that:

• Builds a native iOS app with XCode that has at least one UI test
• Uploads the app to AppCenter and runs that UI test on a real device

Answer 1

Below is from the XCUItest example in Microsoft document.

In your pipeline, before the App Center Test task, you will need an Xcode task step with an action of build-for-testing to build the test runner app needed for testing. In that task, under Arguments in the Advanced section, specify a derived data path, typically using -derivedDataPath DerivedData. You will also need an .ipa file for your application. You can build the .ipa file in the same Xcode build-for-testing task by checking the Create app package option, or in a separate Xcode build step or in a Bash script step.

According to above, You can try setting packageApp=true and specify a archivePath the directory where created archives should be placed. Please check here for more information about xcode task .

For below example, the .ipa file will be created and saved to $(build.artifactStagingDirectory)/achive

- task: Xcode@5
  inputs:
    actions: 'build-for-testing'
    configuration: 'Debug'
    sdk: 'iphoneos'
    xcWorkspacePath: 'MyProject/MyProject.xcworkspace'
    scheme: 'MyProject'
    packageApp: true
    archivePath: "$(build.artifactStagingDirectory)/achive"
    args: '-derivedDataPath $(build.artifactStagingDirectory)/DerivedData'

Then in App Center Test task , you need to set appFile = $(build.artifactStagingDirectory)/achive/{myapp}.ipa . and xcUITestBuildDirectory to the XCUITest bundle location ( DerivedData/Build/Products/Debug-iphoneos/ ). For below example:

- task: AppCenterTest@1
  inputs:
    appFile: '$(build.artifactStagingDirectory)/achive/{myapp}.ipa'
    artifactsDirectory: '$(build.artifactStagingDirectory)/AppCenterTest'
    frameworkOption: 'xcuitest'
    xcUITestBuildDirectory: '$(build.artifactStagingDirectory)/DerivedData/Build/Products/Debug-iphoneos'
    runTests: true
    username：
    password:
    credentialsOption: 'serviceEndpoint'
    serverEndpoint: 'MyCustomer AppCenter Deployment'
    appSlug: 'MyCustomer/MyProject-iOS'
    devices: 'iphoneDevices'
    localeOption: 'en_US'
    skipWaitingForResults: true

Hope above helps!

Answer 2

We ended up with the following build script that actually works. Please note, that you can't combine steps arbitrarily, as xcode build-for-testing does not allow providing signing certificates and thus cannot be combined with packageApp: true

# Xcode
# Build, test, and archive an Xcode workspace on macOS.
# Add steps that install certificates, test, sign, and distribute an app, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/xcode

trigger:
- master

pool:
  vmImage: 'macos-latest'

steps:
########################################
# Install development certificates     #
########################################
- task: InstallAppleCertificate@2
  inputs:
    certSecureFile: 'MyCompany-distribution.p12'
    certPwd: $(P12password)
    keychain: 'temp'

- task: InstallAppleProvisioningProfile@1
  inputs:
    provisioningProfileLocation: 'sourceRepository'
    provProfileSourceRepository: 'Signing/Distribution_Profile.mobileprovision'

########################################
# Build App and test it locally        #
########################################
- task: Xcode@5
  inputs:
    actions: 'clean build test'
    configuration: 'Debug'
    sdk: 'iphoneos'
    xcWorkspacePath: 'TestApp-iOS/TestApp-iOS.xcodeproj'
    scheme: 'TestApp-iOS'
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 11'
    publishJUnitResults: true

###############################################
# Run automated UI Tests on physical devices  #
###############################################
- task: Xcode@5
  inputs:
    actions: 'build-for-testing'
    configuration: 'Debug'
    sdk: 'iphoneos'
    xcWorkspacePath: 'TestApp-iOS/TestApp-iOS.xcodeproj'
    scheme: 'TestApp-iOS'
    packageApp: false
    args: '-derivedDataPath $(build.artifactStagingDirectory)/DerivedData'

- task: Xcode@5
  inputs:
    actions: 'build'
    configuration: 'Release'
    sdk: 'iphoneos'
    xcWorkspacePath: 'TestApp-iOS/TestApp-iOS.xcodeproj'
    scheme: 'TestApp-iOS'
    packageApp: true
    exportPath: '$(build.artifactStagingDirectory)/release'
    signingOption: 'manual'
    signingIdentity: '$(APPLE_CERTIFICATE_SIGNING_IDENTITY)'
    provisioningProfileUuid: '$(APPLE_PROV_PROFILE_UUID)'
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 11'

- task: AppCenterTest@1
  inputs:
    appFile: '$(Build.ArtifactStagingDirectory)/release/TestApp-iOS.ipa'
    artifactsDirectory: '$(Build.ArtifactStagingDirectory)/AppCenterTest'
    frameworkOption: 'xcuitest'
    xcUITestBuildDirectory: '$(build.artifactStagingDirectory)/DerivedData/Build/Products/Debug-iphoneos'
    credentialsOption: 'serviceEndpoint'
    serverEndpoint: 'MyCompany AppCenter Deployment'
    appSlug: 'MyCompany/Moments-iOS'
    devices: 'MyCompany/iphone11'
    localeOption: 'en_US'
    skipWaitingForResults: true
    showDebugOutput: true