To evaluate the performance of CNV callers, we utilized a publicly available benchmark dataset from the study by Fang et al. [32], based on the highly aneuploid HCC1395 triple-negative breast cancer cell line and its matched normal control (HCC1395 BL B-lymphocytes). This dataset has a validated ground truth for somatic variants, established through orthogonal methods including microarrays, PCR, and single-cell sequencing. The final benchmark set comprises 1,788 structural variants (717 deletions, 230 duplications, 551 insertions, 133 inversions, 146 translocations, and 11 breakpoints). The data consist of 21 Whole-Genome Sequencing (WGS) technical replicates (SRA: SRP162370) generated across multiple sequencing centers, allowing for the assessment of tool robustness against technical variability. We downloaded the raw FASTQ files for analysis for the following samples: SRR7890850 (WES_LL_T_1), SRR789 0851 (WES_LL_N_1), SRR7890874 (WES_IL_N_1), SRR78 90876 (WES_FD_T_2), SRR7890877 (WES_FD_T_3), SRR7890 878 (WES_FD_N_3), SRR7890881 (WES_FD_ N_2), SRR7890883 (WES_IL_T_1), SRR7890918 (WES_EA _T_1), SRR7890919 (WES_EA_N_1).

We processed the raw sequencing data uniformly. Read quality was assessed using FastQC (v0.12.0) [34]. Adapter trimming and quality filtering were performed with fastp (v0.23.4) [35] using a sliding window of 4 bp (required quality Q20), removing reads shorter than 70 bp. Filtered reads were aligned to the GRCh38 reference genome using BWA-MEM (v0.7.17) [36].

Tools were selected for comparison based on the following criteria. Each tool had to be specifically designed for somatic CNV detection, be widely adopted within the research community as evidenced by a citation count exceeding two hundred, and have been actively maintained by its developers since at least 2018. Based on these criteria, four tools were selected: CNVkit [28], Sequenza [29], Facets [30], and ASCAT [31]. All tools were run with default parameters as recommended by their developers. A comparative summary of the key technical features, requirements, and methodological approaches of the four evaluated tools is provided in Table 1.

We wrote a custom Python script to standardize and compare the different output formats of the selected CNV calling tools. While each tool employs a distinct output structure, all reports contain the fundamental information required for analysis: chromosomal location and estimated copy number value. For comparison, the reference genome was partitioned into consecutive 100-base pair bins. Each bin was annotated with a copy number value based on its overlap with the CNV segments called by a given tool.

Bins overlapping a reported deletion were assigned a value of 1, while those overlapping an amplification were assigned the reported value (e.g., 3 or higher). Bins with no overlapping CNV call or falling outside exonic regions were assigned a null value.

The comparative analysis was performed by merging the binned data from any two result sets intended for comparison, such as a tool's output versus the validated benchmark or the outputs of two different tools. All rows containing a null value in either of the two compared columns were removed from the merged dataset, so the analysis only covered regions where both sources provided a call. For the evaluation of specific variant types, the dataset was further filtered prior to metric calculation. Rows with a copy number greater than two were isolated for the analysis of amplifications, while rows with a copy number less than two were isolated for the analysis of deletions. This binned dataset was used for all subsequent evaluations.

Tool performance was evaluated using sensitivity, specificity, reproducibility, and concordance metrics, calculated separately for deletions and amplifications. Sensitivity, or the true positive rate, was defined as the proportion of benchmark variants correctly identified by a tool (TP / [TP + FN]). Specificity, representing the true negative rate, was calculated as the proportion of true negative calls correctly identified (TN / [FP + TN]). Reproducibility was assessed as the proportion of bins with identical copy number calls across multiple technical replicates of the same sample, reflecting the technical consistency of a single tool. Concordance was measured as the proportion of bins with identical copy number calls generated by different tools for the same sample, indicating the level of consensus between methods. All metrics were derived from the standardized binned dataset to ensure a uniform comparison.

Evaluation against the orthogonally validated ground truth from the HCC1395 cell line (5 technical replicates: FD2, FD3, EA, IL, LL) showed clear differences between the tools (Fig. 1). CNVkit demonstrated the highest recall for deletions, with a mean recall value of 0.930, indicating good detection of true loss-of-copy events. However, its amplification recall was lower (mean recall = 0.538), suggesting a variant-type bias in its algorithm. Facets and Sequenza performed best for detecting amplifications, achieving high mean recall values of 0.851 and 0.851, respectively. Facets had the most balanced and stable profile overall, maintaining robust recall for both deletions (mean = 0873) and amplifications (mean = 0.851) with consistently low variability (std ~ 0.043–0.017), suggesting more reliable detection across alteration types. ASCAT had low sensitivity, especially for amplifications (mean = 0.454, std = 0.066), while its performance for deletions (mean = 0.732, std = 0.105) was also suboptimal compared to the leading tools. This poor detection of validated events makes ASCAT an outlier in this evaluation.

A consistent trend across all tools was the superior detectability of deletions compared to amplifications, with the maximum observed recall being 0.930 versus 0.851, respectively. This suggests algorithmic or methodological difficulties in identifying gain-of-copy events from exome data.

Regarding precision, all tools showed high specificity for both variant types, well above their recall values (Fig. 2). Facets and CNVkit led in precision for deletions (means of 0.994 and 0.965, respectively), though CNVkit's higher variability (std = 0.038) indicated less consistent reliability. For amplifications, CNVkit, Sequenza, and Facets all achieved near-perfect precision (means > 0.996), indicating that their amplification calls are generally reliable, even if their sensitivity to detect all such events varies. High precision across tools indicates that false positives are not the main problem; rather, the challenge is capturing all true events, especially amplifications.

While sensitivity and specificity measure accuracy against a known ground truth, they do not capture the consistency of a tool's output across technical replicates. A good algorithm should produce reproducible calls regardless of technical sequencing variations. To assess this reproducibility, we analyzed five distinct whole-exome sequencing samples (FD2, FD3, EA, IL, LL) derived from the same HCC1395 cell line. For each tool, we compared CNV calls between pairs of these technical replicates (e.g., FD2 vs. FD3) to quantify the proportion of overlapping genomic bins with identical ploidy assignments. This metric reflects the tool's stability and technical reliability in repeated experiments.

For deletions, all tools demonstrated high and consistent reproducibility, with mean scores ranging from 0.824 (ASCAT) to 0.941 (Sequenza) (Fig. 3). Sequenza and Facets showed particularly strong agreement across replicates for this variant type. Reproducibility for amplifications, however, varied considerably between tools. (mean = 0.541, std = 0.035), meaning its copy number gain calls were unstable across replicates. Sequenza and Facets, on the other hand, maintained high reproducibility for amplifications (means of 0.969 and 0.974, respectively), with low variability. ASCAT showed intermediate but robust performance for amplifications (mean = 0.843). This shows that some tools (Sequenza, Facets) produce stable call sets regardless of technical noise, while others, particularly CNVkit, are inconsistent for amplification detection, which limits their use in settings that require reliable results.

Beyond absolute accuracy, consensus between different bioinformatic tools is an important metric, since agreement between independent methods increases confidence in a call set. To quantify this, we calculated pairwise concordance - defined as the proportion of identical CNV calls - for every possible pair of tools across all samples. For each tool, its overall concordance was characterized by the distribution of its pairwise agreement values with all other tools, serving as a measure of its overall agreement with other tools.

Inter-tool concordance differed substantially and was also affected by variant type (Fig. 4). For deletions, Facets demonstrated the highest median concordance (0.768 ± 0.086), closely followed by ASCAT (0.731 ± 0.090), CNVkit (0.712 ± 0.125), and Sequenza (0.710 ± 0.089). This relatively tight clustering indicates a stronger consensus among tools on deletion calls. For amplifications, concordance was lower and more variable across all tools. CNVkit and ASCAT showed the poorest agreement with the field, with mean concordance values of 0.508 (± 0.066) and 0.521 (± 0.089), respectively. Sequenza and Facets achieved higher consensus for amplifications (0.698 ± 0.200 and 0.691 ± 0.200, respectively), though the higher standard deviations show that their agreement with other tools was inconsistent and depended on the specific pairwise comparison.

The analysis revealed that inter-tool concordance is not random but follows distinct, reproducible patterns based on algorithmic methodology, forming clear “strategic alliances” between tools (Fig. 5). For deletions, the highest and most stable agreement was consistently observed between Sequenza and ASCAT (mean concordance across samples: ~0.775) and between CNVkit and Facets (mean concordance: ~0.833) or data processing for detecting copy number losses. Conversely, the lowest agreement for deletions was repeatedly found between CNVkit/ASCAT and Facets/ASCAT pairs, showing that these tools interpret the data differently.

For amplifications, the pattern was more pronounced. The pair Sequenza/Facets demonstrated near-perfect concordance (mean: 0.965 ± 0.018), indicating very similar calling of copy number gains by these two tools. CNVkit showed persistently poor agreement with all other tools for amplifications, with its concordance values never exceeding 0.565 and dropping as low as 0.352 with ASCAT in the LL sample. Similarly, ASCAT's agreement with other tools for amplification was consistently low. CNVkit and ASCAT are thus outliers in amplification detection, while Sequenza and Facets form a high-consensus pair.