Question about “duplicate in list” error and memory usage when running qpDstat #121
Description
Dear Author,
We are currently running qpDstat from the AdmixTools package using the following command:
/data/01/user157/software/AdmixTools/bin/qpDstat -p parDstat_alltest > test.out
Our parameter file is as follows:
indivname: snp.eigenstrat.ind
snpname: snp.eigenstrat.lower.chr.snp
genotypename: snp.eigenstrat.chr.geno
poplistname: test.popfilename.txt
printsd: YES
f4mode: NO
blgSize: .01
The .ind, .geno, and .snp files are as follows:
We have 14 species within a single genus, and for each triplet of ingroup species, we used a species from another genus (M_un) as the outgroup.
In the popfilename.txt, we therefore listed all possible three-species combinations (each with the same outgroup) and included all three permutations for each triplet, as shown in the attached example (see screenshot).
However, during execution we encountered the following error:
fatalx:
(loadlist) duplicate in list: A_ca
/slurmState/slurmSpool/slurmd/job1133823/slurm_script: line 10: 3990729 Aborted (core dumped)
Could you please let us know what might be causing this “duplicate in list” error?
We checked our population list file, but we are sure that no duplicate population names is within a single line.
In addition, our dataset includes 278,518,514 SNPs, which seems to require very large memory.
We currently allocate 2 CPUs and 250 GB of memory, but in some runs qpDstat still exceeds memory limits.
Do you have any recommendations on the typical memory requirements or efficient settings for datasets of this size (e.g., adjusting blgSize or using a subset of SNPs)?
Thank you very much for your time and for maintaining such an important tool for population genetic analysis.
Best regards,
Na Wan