## Testing the engine Pull requests submitted to the [engine repository](https://github.com/flutter/engine) should be tested to prevent functional regressions. This guide describes how to write and run various types of tests in the engine. ## C++ - core engine If you edit `.cc` files in https://github.com/flutter/engine/tree/main, you're working on the core, portable Flutter engine. ### Unit tests C++ unit tests are co-located with their header and source files. For instance, `fml/file.h` and `fml/file.cc` have a `fml/file_unittest.cc` in the same directory. When editing C++ files, look for its `_unittest.cc` sibling or create one if there isn't one present. The engine repo has a unified build system to build C, C++, Objective-C, Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and [Ninja](https://ninja-build.org/). Individual `_unittest.cc` files are referenced by the `BUILD.gn` build rule located in the folder or in an ancestor folder. You can run the C++ unit tests with: ``` testing/run_tests.py --type=engine ``` from the `flutter` directory, after building the engine variant to test (by default `host_debug_unopt`). To use a different variant (e.g. if you use an Apple Silicon Mac), run: ``` testing/run_tests.py --type=engine --variant=host_debug_unopt_arm64 ``` Behind the scenes, those tests in the same directory are built together as a testonly executable when you build the engine variant. The `run_tests.py` script executes them one by one. C++ unit tests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. #### Google Tests C++ unit tests in the core engine uses the [Google Test](https://google.github.io/googletest/primer.html) C++ testing framework to facilitate C++ test discovery, assertions, etc. Since the engine is portable, these unit tests are compiled and run directly on and for your host machine architecture. It's best practice to test only one real production class per test and create mocks for all other dependencies. ## Java - Android embedding If you edit `.java` files in the https://github.com/flutter/engine/tree/main/shell/platform/android directory, you're working on the Android embedding which connects the core C++ engine to the Android SDK APIs and runtime. ### Robolectric JUnit tests For testing logic within a class at a unit level, create or add to a JUnit test. Existing Java unit tests are located at https://github.com/flutter/engine/tree/main/shell/platform/android/test and follow the Java package directory structure. Files in the `shell/platform/android/io/flutter/` package tree can have a parallel file in the `shell/platform/android/test/io/flutter/` package tree. Files in matching directories are considered [package visible](https://docs.oracle.com/javase/tutorial/java/javaOO/accesscontrol.html) as is the case in standard Java. When editing production files in `shell/platform/android/io/flutter/`, the easiest step to add tests is to look for a matching `...Test.java` file in `shell/platform/android/test/io/flutter/`. See the [Java unit test README](https://github.com/flutter/engine/blob/main/shell/platform/android/test/README.md) for details. The engine repo has a unified build system to build C, C++, Objective-C, Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and [Ninja](https://ninja-build.org/). Because it doesn't use the more common Gradle build system (which can't build C++ for instance), the tests and its dependencies can't be directly built and run inside Android Studio like a standard Android project. Instead, the engine provides the script: ``` testing/run_tests.py --type=java ``` to easily build and run the JUnit tests. This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your [GN flags](../contributing/Compiling-the-engine.md#compiling-for-android-from-macos-or-linux) if you haven't built the engine for a while. Behind the scenes, it invokes GN and Ninja to build a single .jar file containing the test runner and dependencies. Then it uses the system `java` runtime to execute the .jar. JDK v8 must be set as your `$JAVA_HOME` to run the Robolectric tests. See [Setting-up-the-Engine-development-environment#using-vscode-as-an-ide-for-the-android-embedding-java](../dev/Setting-up-the-Engine-development-environment.md) for tips on setting up Java code completion and syntax highlighting in Visual Studio when working on the engine and tests. JUnit tests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. #### Robolectric [Robolectric](http://robolectric.org/) is a standard Android testing library to mock the Android runtime. It allows tests to be executed on a lightweight Java JVM without booting a heavy Android runtime in an emulator. This allows for rapid test iterations and allows our tests to run better on CI systems. All engine JUnit tests are Robolectric tests. This means all `android.*` imports are mocked by Robolectric. If you need to modify how Android components (such as an [android.view.View](https://developer.android.com/reference/android/view/View.html) or an [android.app.Activity](https://developer.android.com/reference/android/app/Activity.html)) behave in the test, see other tests for examples or see docs at http://robolectric.org/ on how to interact with shadows. #### Mockito [Mockito](https://site.mockito.org/) is also a standard Android testing library used to mock non-Android dependencies needed to construct and test interactions with your your under-test production class. It's best practice to test only one real production class per test and mock all other dependencies with mockito. The Mockito library is an available test dependency when writing Robolectric tests. ### Component integration tests Component tests test the interaction of multiple embedding Java classes together but they don't test all production classes end-to-end. In the Android embedding case, we test groups of Java classes by their function in `...ComponentTest.java` files that are also in the `shell/platform/android/test/io/flutter/` directory. C++ engine parts via JNI are not tested here. Component tests are also Robolectric JUnit tests and are invoked together with unit tests when running: ``` testing/run_tests.py --type=java ``` JUnit component tests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. ### End-to-end tests End-to-end tests exercise the entire Android embedding with the C++ engine on a real Android runtime in an emulator. It's an integration test ensuring that the engine as a whole on Android is functioning correctly. The project containing the Android end-to-end engine test is at https://github.com/flutter/engine/tree/main/testing/scenario_app/android. This test project is build similarly to a normal Flutter app. The Dart code is compiled into AOT and the Android part is compiled via Gradle with a dependency on the prebuilt local engine. The built app then installed and executed on an emulator. Unlike a normal Flutter app, the Flutter framework on the Dart side is a lightweight fake at https://github.com/flutter/engine/tree/main/testing/scenario_app/lib that implements some of the basic functionalities of `dart:ui` Window rather than using the real Flutter framework at `flutter/flutter`. The end-to-end test can be executed by running: ``` testing/scenario_app/run_android_tests.sh ``` Additional end-to-end instrumented tests can be added to https://github.com/flutter/engine/tree/main/testing/scenario_app/android/app/src/androidTest/java/dev/flutter/scenarios. If supporting logic is needed for the test case, it can be added to the Android app under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/android/app/src/main/java/dev/flutter/scenarios or to the fake Flutter framework under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/lib. As best practice, favor adding unit tests if possible since instrumented tests are, by nature, non-hermetic, slow and flaky. End-to-end tests on Android are run on presubmit for flutter/engine PRs. ## Objective-C - iOS embedding If you edit `.h` or `.mm` files in the https://github.com/flutter/engine/tree/main/shell/platform/darwin/ios directory, you're working on the iOS embedding which connects the core C++ engine to the iOS SDK APIs and runtime. ### XCTest unit tests For testing logic within a class in isolation, create or add to a XCTestCase. The iOS unit testing infrastructure is split in 2 different locations. The `...Test.mm` files in https://github.com/flutter/engine/tree/main/shell/platform/darwin/ios contain the unit tests themselves. The https://github.com/flutter/engine/tree/main/testing/ios/IosUnitTests directory contains an Xcode container project to execute the test. See the [iOS unit test README](https://github.com/flutter/engine/blob/main/testing/ios/IosUnitTests/README.md) for details on adding new test files. The engine repo has a unified build system to build C, C++, Objective-C, Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and [Ninja](https://ninja-build.org/). Since GN and Ninja has to build the C++ dependencies that the Objective-C classes reference, the tests aren't built by the Xcode project in https://github.com/flutter/engine/tree/main/testing/ios/IosUnitTests. Instead, the engine provides the script: ``` testing/run_tests.py --type=objc ``` to easily build and run the XCTests. - Add the `--ios-variant ios_debug_sim_unopt_arm64` argument when using an arm64 Mac simulator (built with `--simulator-cpu=arm64`). This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your GN flags ([See compiling for ios from macos](../contributing/Compiling-the-engine.md#compiling-for-ios-from-macos)) if you haven't built the engine for a while. Behind the scenes, it invokes GN and Ninja to build the tests and dependencies into a single `.dylib`. Then it uses Xcode and the Xcode project at `testing/ios/IosUnitTests` to import and execute the XCTests in the `.dylib`. If you get an `AssertionError: libios_test_flutter.dylib doesn't exist` error, you may need to manually run the ninja command that is printed to the terminal. e.g. `ninja -C $FLUTTER_ENGINE/out/ios_debug_sim_unopt_arm64 ios_test_flutter` See [Setting-up-the-Engine-development-environment#editor-autocomplete-support](../dev/Setting-up-the-Engine-development-environment.md) for tips on setting up C/C++/Objective-C code completion and syntax highlighting when working on the engine and tests. To debug the XCTests, you can open the Xcode project at `testing/ios/IosUnitTests/IosUnitTests.xcodeproj` and run the tests (such as via ⌘U). Note you cannot modify the test source and build the tests in Xcode for reasons mentioned above. If you modify the test, you need to run `testing/run_tests.py` again. XCTests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. #### XCTest [XCTest](https://developer.apple.com/documentation/xctest) is the standard way of creating unit tests in Xcode projects. Since iOS has x86 simulators and since we can build x86 engines, we can execute the XCTests directly on macOS in a headless simulator using the real iOS SDK. #### OCMock [OCMock](https://ocmock.org) is a standard iOS testing library used to mock dependencies needed to construct and test interactions with your under-test production class. It's best practice to test only one real production class per test and mock all other dependencies with OCMock. The OCMock library is available as a test dependency when writing XCTests for the engine. ### End-to-end tests End-to-end tests exercise the entire iOS embedding with the C++ engine on a headless iOS simulator. It's an integration test ensuring that the engine as a whole on iOS is functioning correctly. The project containing the iOS end-to-end engine test is at https://github.com/flutter/engine/tree/main/testing/scenario_app/ios. This test project is build similarly to a normal debug Flutter app. The Dart code is bundled in JIT mode and is brought into Xcode with a `.framework` dependency on the prebuilt local engine. It's then installed and executed on a simulator via Xcode. Unlike a normal Flutter app, the Flutter framework on the Dart side is a lightweight fake at https://github.com/flutter/engine/tree/main/testing/scenario_app/lib that implements some of the basic functionalities of `dart:ui` Window rather than using the real Flutter framework at `flutter/flutter`. The end-to-end test can be executed by running: ``` testing/scenario_app/run_ios_tests.sh ``` Additional end-to-end instrumented tests can be added to https://github.com/flutter/engine/tree/main/testing/scenario_app/ios/Scenarios/ScenariosTests. If supporting logic is needed for the test case, it can be added to the Android app under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/ios/Scenarios/Scenarios or to the fake Flutter framework under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/lib. As best practice, favor adding unit tests if possible since end-to-end tests are, by nature, non-hermetic, slow and flaky. End-to-end tests on iOS are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. ## Dart - dart:ui If you edit `.dart` files in https://github.com/flutter/engine/tree/main/lib/ui, you're working on the 'dart:ui' package which is the interface between the C++ engine and the Dart Flutter framework. ### Unit tests Dart classes in https://github.com/flutter/engine/tree/main/lib/ui have matching unit tests at https://github.com/flutter/engine/tree/main/testing/dart. When editing production files in the 'dart:ui' package, add to or create a test file in `testing/dart`. To run the Dart unit tests, use the script: ``` testing/run_tests.py --type=dart ``` Behind the scenes, it invokes the engine repo's unified [GN](https://gn.googlesource.com/gn/) and [Ninja](https://ninja-build.org/) build systems to use a version of the Dart SDK specified in the `DEPS` file to create a `sky_engine` Dart package. Then it compiles and runs each `_test.dart` file under `testing/dart`. To debug the test, open `src/out/ios_debug_sim_unopt/scenario_app/Scenarios.xcodeproj` in Xcode and hit CMD+U. Dart unit tests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. _See also: [Flutter Test Fonts](https://github.com/flutter/flutter/blob/master/docs/contributing/testing/Flutter-Test-Fonts.md)_ ### Framework tests Dart tests in the `flutter/flutter` framework repo are also executed on top of the `dart:ui` package and underlying engine. These tests are executed during pre-submit on our CI system when submitting PRs to the `flutter/engine` repository. Assuming your `flutter` and `engine` working directories are siblings, you can run the framework tests locally using the following command from the root of your `flutter` repository: ```bash (cd packages/flutter; ../../bin/flutter test --local-engine=host_debug_unopt --local-engine-host=host_debug_unopt) ``` ## Web engine Web tests are run via the `felt` command. More details can be found in [lib/web_ui/README.md](https://github.com/flutter/engine/blob/main/lib/web_ui/README.md#hacking-on-the-web-engine).