Test Execution Internals

This document explains tests.cmd internals and helps troubleshoot test execution issues.

Overview

The test execution chain:

tests.cmd → Bazel → IdeaUltimateRunTestsBuildTarget → TestingTasksImpl → JUnit 5

Key components:

• tests.cmd: Shell script that invokes Bazel with test parameters
• Bazel target: //build:local_idea_ultimate_run_tests_build_target
• TestingOptions: Parses -Dintellij.build.test.* system properties
• TestingTasksImpl: Orchestrates classpath assembly and JVM setup
• JUnit runners: Execute tests in a forked JVM process

Common Issues & Solutions

Tests Not Found

Symptoms:

• "No tests found" message
• Test class exists but isn't executed

Causes & Solutions:

1. Pattern mismatch - Simple class names don't work (see Pattern Matching below):

   # WRONG - simple name won't match FQN
   -Dintellij.build.test.patterns=MyTest
   
   # CORRECT - use wildcard or FQN
   -Dintellij.build.test.patterns=*MyTest
   -Dintellij.build.test.patterns=com.example.MyTest
   

2. Test not in classpath - The test class must be in a module that's part of the test classpath. Check if the module is included in the build.

3. Test class not recognized - Ensure class name ends with Test or has JUnit annotations.

OutOfMemoryError

Solution: Increase heap size:

./tests.cmd \
  --module <module> \
  --test MyTest \
  -Dintellij.build.test.jvm.memory.options=-Xmx8g

Bazel Build Fails

Symptoms:

• Build errors before tests run
• Missing dependencies

Solutions:

1. Check module dependencies - Ensure test module has required dependencies in .iml file.

2. Verify BUILD.bazel is synced - Run ./build/jpsModelToBazel.cmd after changing .iml files.

Note: Bazel incremental builds are always correct. Do not use bazel clean - it won't help.

Test Discovery Issues

Symptoms:

• Wrong tests executed
• Tests filtered unexpectedly

Debug steps:

1. Check test groups - If using groups, verify testGroups.properties configuration.

2. Check bucketing - For parallel execution, tests are distributed by hash:

   # See which bucket a test falls into
   -Didea.test.runners.count=4
   -Didea.test.runner.index=0  # Run only bucket 0
   

3. Check class filters - TestCaseLoader applies pattern matching before test execution.

Debug Mode

Enable debug mode to attach a debugger:

./tests.cmd \
  --module <module> \
  --test MyTest \
  -Dintellij.build.test.debug.enabled=true \
  -Dintellij.build.test.debug.port=5005 \
  -Dintellij.build.test.debug.suspend=true

Then attach debugger to port 5005.

Quick Reference

Detailed Reference

Execution Flow Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│  1. COMMAND LINE                                                            │
│     ./tests.cmd --module <module> --test MyTest                             │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  2. SHELL SCRIPT                                                            │
│     tests.cmd → community/build/run_build_target.sh                         │
│     Maps --module/--test to -D properties, wraps as --jvm_flag=<arg>        │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  3. BAZEL                                                                   │
│     bazel run //build:local_idea_ultimate_run_tests_build_target            │
│     (defined in build/BUILD.bazel)                                          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  4. BUILD TARGET ENTRY POINT                                                │
│     IdeaUltimateRunTestsBuildTarget.main()                                  │
│     → UltimateProjectTestingTasks.runTests()                                │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  5. TEST OPTIONS PARSING                                                    │
│     UltimateProjectTestingOptions (extends TestingOptions)                  │
│     Reads all -Dintellij.build.test.* system properties                     │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  6. TEST EXECUTION ORCHESTRATION                                            │
│     TestingTasksImpl.runTests()                                             │
│     - Builds test classpath                                                 │
│     - Prepares JVM arguments and system properties                          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  7. FORKED TEST PROCESS                                                     │
│     TestingTasksImpl.runJUnit5Engine()                                      │
│     Spawns new JVM with bootstrap classpath                                 │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
┌───────────────────────────────────┐ ┌───────────────────────────────────────┐
│  8a. JUNIT 5 TESTS                │ │  8b. JUNIT 3/4 TESTS (Legacy)         │
│  JUnit5TeamCityRunner.main()      │ │  JUnit5TeamCityRunner.main()          │
│  - Uses JUnit Platform Launcher   │ │  - Uses JUnit Platform Launcher       │
│  - ClassNameFilter                │ │  - ClassNameFilter                    │
│  - PostDiscoveryFilter            │ │  - PostDiscoveryFilter                │
└───────────────────────────────────┘ └───────────────────────────────────────┘
                    │                               │
                    └───────────────┬───────────────┘
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  9. TEST DISCOVERY & FILTERING                                              │
│     TestCaseLoader                                                          │
│     - Loads test classes from classpath roots                               │
│     - Applies pattern/group filters                                         │
│     - Handles bucketing for parallel execution                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  10. TEST EXECUTION                                                         │
│      JUnit Platform executes tests                                          │
│      TCExecutionListener reports results to TeamCity                        │
└─────────────────────────────────────────────────────────────────────────────┘

Key Classes Reference

Entry Points & Orchestration

Test Options

JUnit 5 Test Runners

Test Discovery & Loading

Bucketing & Distribution

Test Module Hierarchy

Entry Points by Product

Note: Product entry points (RustRover, RubyMine, CLion) inherit intellij.idea.ultimate.tests.main as the default, but to run product-specific tests, use the dedicated test module via --module. See TESTING.md for the correct module per product.

CI-Defined Test Modules

From intellij-teamcity-config/.teamcity/src/ijplatform/KnownModules.kt:

Module Configuration

Default mainModule is set in:

• UltimateProjectTestingOptions.kt:36 - Ultimate: intellij.idea.ultimate.tests.main
• CommunityRunTestsBuildTarget.kt:28 - Community: intellij.idea.community.main.tests

Ultimate Test Module Tree (Simplified)

intellij.idea.ultimate.tests.main
├── intellij.idea.ultimate.tests
├── intellij.idea.ultimate.tests.kotlin
├── intellij.platform.tests
├── intellij.java.tests
└── ... (hundreds of test modules)

Separate hierarchies (NOT in .main):
├── intellij.idea.ultimate.tests.kotlin.k2
│   └── intellij.devkit.kotlin.fir.tests
├── intellij.idea.ultimate.tests.devBuildTests
└── kotlin.fir-all-tests (K2/FIR tests)

TestingOptions Properties

TestingOptions Properties

Most options use the intellij.build.test.* prefix. Bucketing uses idea.test.* prefix.

// Test selection (mutually exclusive, in priority order)
testConfigurations  // -Dintellij.build.test.configurations=<config>
testPatterns        // -Dintellij.build.test.patterns=<pattern>
testGroups          // -Dintellij.build.test.groups=<group>

// Test execution
mainModule          // -Dintellij.build.test.main.module=<module>
attemptCount        // -Dintellij.build.test.attempt.count=<n>

// JVM configuration
jvmMemoryOptions    // -Dintellij.build.test.jvm.memory.options=<opts>
customRuntimePath   // -Dintellij.build.test.jre=<path>

// Debugging
isDebugEnabled      // -Dintellij.build.test.debug.enabled=<bool>
debugPort           // -Dintellij.build.test.debug.port=<port>
isSuspendDebugProcess // -Dintellij.build.test.debug.suspend=<bool>

// Bucketing (parallel execution) - NOTE: uses idea.test.* prefix
bucketsCount        // -Didea.test.runners.count=<n>
bucketIndex         // -Didea.test.runner.index=<n>

// Coverage
enableCoverage      // -Dintellij.build.test.coverage.enabled=<bool>
coveredClassesPatterns // -Dintellij.build.test.coverage.include.class.patterns=<patterns>

Bazel Target Configuration

The test target is defined in build/BUILD.bazel:

java_binary(
  name = "local_idea_ultimate_run_tests_build_target",
  runtime_deps = [":build"],
  main_class = "IdeaUltimateRunTestsBuildTarget",
  data = ALL_ULTIMATE_TARGETS + [BAZEL_TARGETS_JSON_ULTIMATE],
  jvm_flags = [
    "-Dintellij.build.console.exporter.to.temp.file=true",
    "-Dintellij.build.console.messages.verbose=false",
    "-Dintellij.build.clean.output.root=false",      # Reuse compiled classes
    "-Dintellij.build.use.compiled.classes=true",    # Skip recompilation
    "-Dintellij.build.bazel.targets.json.file=$(rlocationpath %s)" % BAZEL_TARGETS_JSON_ULTIMATE,
  ],
  add_opens = INTELLIJ_ADD_OPENS,
)

Test Process JVM Configuration

TestingTasksImpl.prepareEnvForTestRun() configures the forked test JVM:

// Key system properties set for test process:
"idea.home.path"     → projectHome
"idea.config.path"   → tempDir/config
"idea.system.path"   → tempDir/system
"java.io.tmpdir"     → tempDir

// JVM options:
"-XX:+HeapDumpOnOutOfMemoryError"
"-XX:HeapDumpPath=<snapshotsDir>/intellij-tests-oom-<timestamp>.hprof"
"-Xms750m -Xmx1024m"  // or custom from jvmMemoryOptions
// Plus --add-opens for module access

Passing JVM Args to Test Process

There are two mechanisms for passing JVM arguments to the test JVM process:

1. Memory Options (intellij.build.test.jvm.memory.options)

For JVM memory settings like heap size, use the dedicated property:

./tests.cmd --module <module> --test <pattern> -Dintellij.build.test.jvm.memory.options="-Xmx4g -Xms2g"

Multiple options are space-separated within quotes. These options are added to the beginning of the JVM arguments via VmOptionsGenerator.generate().

Implementation (see TestingTasksImpl.kt, runJUnit5Engine method):

val customMemoryOptions = options.jvmMemoryOptions?.trim()?.split(Regex("\\s+"))?.takeIf { it.isNotEmpty() }
jvmArgs.addAll(
  index = 0,
  elements = VmOptionsGenerator.generate(
    customVmMemoryOptions = if (customMemoryOptions == null) mapOf("-Xms" to "750m", "-Xmx" to "1024m") else emptyMap(),
    additionalVmOptions = customMemoryOptions ?: emptyList(),
    // ... other parameters omitted
  ),
)

2. Pass-through System Properties (pass.* prefix)

To pass arbitrary system properties to the test JVM, use the pass. prefix. The prefix is stripped before passing to the test process:

./tests.cmd --module <module> --test <pattern> -Dpass.my.custom.property=value -Dpass.some.flag=true

Results in test JVM receiving:

• -Dmy.custom.property=value
• -Dsome.flag=true

Implementation (see TestingTasksImpl.kt, prepareEnvForTestRun method):

for ((key, value) in System.getProperties()) {
  key as String
  if (key.startsWith("pass.")) {
    systemProperties.put(key.substring("pass.".length), value as String)
  }
}

This is a TeamCity convention for passing properties to nested processes.

Combined Example

./tests.cmd \
  --module <module> \
  --test MyTest \
  -Dintellij.build.test.jvm.memory.options="-Xmx4g" \
  -Dpass.my.test.flag=enabled \
  -Dpass.debug.level=verbose

Important: Properties without pass. prefix are consumed by the build scripts, NOT passed to the test JVM.

Test Discovery Flow

Step-by-Step Pattern Matching

┌─────────────────────────────────────────────────────────────────────────────┐
│  PATTERN INPUT                                                              │
│  -Dintellij.build.test.patterns=*MyTest                                     │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  PATTERN COMPILATION (TestClassesFilter.compilePattern)                     │
│  filter.replace("$","\\$").replace(".","\\.");replace("*",".*")             │
│  "*MyTest" → ".*MyTest"                                                     │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  CLASS DISCOVERY (ClassFinder)                                              │
│  Scans JARs for *Test.class files, extracts FQN:                            │
│  org/example/MyTest.class → "org.example.MyTest"                            │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  PATTERN MATCHING (PatternListTestClassFilter.matches)                      │
│  pattern.matcher(className).matches()  ← FULL STRING MATCH                  │
│  ".*MyTest".matches("org.example.MyTest") → true                            │
└─────────────────────────────────────────────────────────────────────────────┘

Pattern Transformation Table

Key Implementation Details

1. Pattern Compilation (TestClassesFilter.compilePattern()):

   filter = filter.replace("$", "\\$").replace(".", "\\.").replace("*", ".*");
   return Pattern.compile(filter);
   

2. Pattern Matching (PatternListTestClassFilter.matches()):

   return ContainerUtil.exists(patterns, pattern -> pattern.matcher(className).matches());
   

   • Uses matches() NOT find() - requires ENTIRE string to match
   • className is ALWAYS the fully qualified name (FQN)

3. ClassNameFilter (JUnit 5) - Fast pre-filter on class names

   • Calls TestCaseLoader.isClassNameIncluded(className)
   • Applied to EVERY class in classpath (must be fast)

4. PostDiscoveryFilter (JUnit 5) - Post-discovery filter

   • Calls TestCaseLoader.isClassIncluded(className)
   • Checks bucketing (which runner should execute this test)

5. TestCaseLoader.fillTestCases() (JUnit 3/4) - Scans classpath roots

   • Uses ClassFinder to find all *Test.class files
   • Calls isPotentiallyTestCase() which calls filter's matches()
   • Applies group-based filtering from testGroups.properties

Why Simple Class Names NEVER Work

Root Cause: Pattern.matches() requires the ENTIRE string to match.

// Pattern: "MyTest" → Regex: "MyTest"
// className: "org.example.MyTest"
"MyTest".matches("org.example.MyTest")  // Returns FALSE

// Pattern: "*MyTest" → Regex: ".*MyTest"  
".*MyTest".matches("org.example.MyTest")  // Returns TRUE

This applies to ALL modules (default and non-default). Always use:

• FQN: org.example.MyTest
• Or wildcard: *MyTest

Related Documentation

• TESTING.md - How to run tests via tests.cmd (quick start, parameters, examples)
• Writing Tests - How to write tests (framework, @TestApplication, fixtures, EDT)