Methodology Overview: Setup And Configuration
Vendors are responsible for setup of their tool. For detection evaluations, all vendors are required to turn off protections, preventions, etc., to the extent possible to ensure the evaluation can be executed. Otherwise, vendors are asked to configure their solutions in a way that would be available to their customers and representative of a realistic deployment. MITRE does not make judgments on the realism of the configuration chosen. Any change to the default tool configuration is noted and reported to MITRE, including rule configuration, alert sensitivities, and cloud configurations, among other possibilities. The vendor provides version numbers as well as a description of configuration changes for MITRE to include in the final report.
Tool configuration and updates are not allowed after the evaluation phase begins, barring misconfigurations that limit the ability for MITRE to complete the evaluation. After the initial evaluation has completed, configuration changes are permitted for detection evaluations with the explicit permission of the MITRE team. Retests are constrained to the standard execution phase duration. MITRE withholds the right to include or exclude configuration change results based on the practical nature of the enhancement, as determined by our team. For any changes to the configuration, we note this in the results.
If the range requires additional modifications to enable the vendor to successfully deploy its tool for evaluation, MITRE notes these modifications in the final report.