Automated qpGraph: possibilities for the consideration of technically savvy Users?

[durmang]

David (of Eurogenes fame) recently posted something of considerable interest:

https://eurogenes.blogspot.com/2020/...ploration.html

Rather significant, because it makes this type of analysis quite accessible for the rest of us. I myself dived in immediately... but alas, my laptop does not have enough juice! (I did have a desktop (a few months back) that would've been perfect for this)

I know that this forum is home to multiple individuals with a solid grasp of qpGraph (Kale, Chad, David/Generalissimo, etc), but I'm also quite aware that manually exploring topologies is one hell of a pain (and perhaps somewhat problematic from a conceptual standpoint).

With this no longer being an issue, can the experts here say whether any among them could attempt a comprehensive qpGraph topology that involves a substantive sampling of the aDNA record (with respect to Paleolithic/Mesolithic/Neolithic samples)?

When I was attempting this myself, my goal was an automated graph involving:

Archaic Humans: Neanderthal, Denisovan

Africans: Shum_Laka_8000BP, Taforalt

Broadly Eurasian: Ust_Ishim

Broadly "West Eurasian": Kostenki14, Sunghir, GoyetQ116-1, Villabruna, Yana, MA1

ENA: Tianyuan, LAO_Hoabinhian, MNG_North_N

Near Eastern: Natufians, Pinarbasi_HG, CHG (Kotias + Satsurblia), Iran_N (Ganj_Dareh), Sarazm_EN, Shahr_I_Sokhta_BA2 (IVC-related)

If anyone with the means could attempt this, that'd be pretty awesome.

[Kale]

You don't know how bad I'm wanting to do this! I just can't figure out how to get it up and running.
Right now I'm stuck at why does qpgraph need block_lengths.rds if F2 statistics are already computed?

Basically I have 1500 lines of this (output from qpfstat)... Linux command-line qpgraph understands this, how do I make R-qpgraph understand?

 

Neanderthal.DG Neanderthal.DG 119.340
Neanderthal.DG Mbuti.DG 61.162
Neanderthal.DG Dinka.DG 61.136
Neanderthal.DG Mende.DG 61.376
Neanderthal.DG Mota.SG 61.233
Neanderthal.DG South_Africa_2000BP.SG 61.447
Neanderthal.DG Cameroon_ShumLaka_8000BP.DG 61.200
Neanderthal.DG Taforalt 62.099
Neanderthal.DG Loschbour.DG 62.937
Neanderthal.DG Onge.DG 62.926
Mbuti.DG Mbuti.DG 158.395
Mbuti.DG Dinka.DG 151.343
Mbuti.DG Mende.DG 148.330
Mbuti.DG Mota.SG 147.789
Mbuti.DG South_Africa_2000BP.SG 140.026

[durmang]

You don't know how bad I'm wanting to do this! I just can't figure out how to get it up and running.

We're undoubtedly in the same boat! It's an exciting enhancement of qpGraph (and I have so many ideas that I'm chomping at the bit to explore).

I'm gonna keep at it, until I figure out a way. But my problem is a bit more basic than yours: my computer goes black and becomes unresponsive, when I try to get the f2 blocks. Lol

In the meantime, my hope is that some kind soul finds enough mercy in their heart to do my work for me (by attempting an admixture graph involving the populations that I've listed).

[Mnemonics]

You don't know how bad I'm wanting to do this! I just can't figure out how to get it up and running.
Right now I'm stuck at why does qpgraph need block_lengths.rds if F2 statistics are already computed?

Basically I have 1500 lines of this (output from qpfstat)... Linux command-line qpgraph understands this, how do I make R-qpgraph understand?

 

Neanderthal.DG Neanderthal.DG 119.340
Neanderthal.DG Mbuti.DG 61.162
Neanderthal.DG Dinka.DG 61.136
Neanderthal.DG Mende.DG 61.376
Neanderthal.DG Mota.SG 61.233
Neanderthal.DG South_Africa_2000BP.SG 61.447
Neanderthal.DG Cameroon_ShumLaka_8000BP.DG 61.200
Neanderthal.DG Taforalt 62.099
Neanderthal.DG Loschbour.DG 62.937
Neanderthal.DG Onge.DG 62.926
Mbuti.DG Mbuti.DG 158.395
Mbuti.DG Dinka.DG 151.343
Mbuti.DG Mende.DG 148.330
Mbuti.DG Mota.SG 147.789
Mbuti.DG South_Africa_2000BP.SG 140.026

Are you trying parse_qpDstat_output on your own precomputed dstats?

[Kale]

But my problem is a bit more basic than yours: my computer goes black and becomes unresponsive, when I try to get the f2 blocks. Lol

Mine does that too, your problem is you haven't given up on trying to do that yet like I have

Are you trying parse_qpDstat_output on your own precomputed dstats?

Those aren't dstats, that is the output of qpfstat, which on the Linux version is what qpgraph operates on. You give it that and a proposed graph and whammo it's off to the races.
neither parse_qpDstat_output, parse_qp3pop_output, nor parse_qpf4ratio_output work on that file and there is no parse_qpFstat_output command
read_f2 doesn't work, it's looking for the weirdo .rds generated things.

[K33]

Can this qpGraph run thru R create "Ghost" populations (ie, Basal Human, Archaic)? Or does it only create branches to/from available samples?

Also the Reich dataset gives you .geno, .ind, and .snp files after unzipping... is the .geno file the one you feed into qpgraph? Are there any primers on how to use this Windows qpGraph/admixtools software package?

I have used the R program before, but only for G25 ADMIXTURE back before the online G25 tools were available...

[Kale]

Can this qpGraph run thru R create "Ghost" populations (ie, Basal Human, Archaic)? Or does it only create branches to/from available samples?

Also the Reich dataset gives you .geno, .ind, and .snp files after unzipping... is the .geno file the one you feed into qpgraph? Are there any primers on how to use this Windows qpGraph/admixtools software package?

I have used the R program before, but only for G25 ADMIXTURE back before the online G25 tools were available...

Yes it can create ghosts, and the reich dataset needs? (or at least definitely should) be converted to packedancestrymap format.
Info for how to run the new admixtools is here...
https://uqrmaie1.github.io/admixtools/

Step0) Download it
Step0.5) Invoke it's mighty power
library(admixtools)

Step 1) get all of your f2 stats...
extract_f2('C:/whatever/is/your/directory/with/dataset', 'C:/wherever/you/want/the/f2/to/go', pops = c('pop1, 'pop2.DG', 'pop3.SG'), maxmem = 1000)
You must create the directory you want the f2's to go, it won't make it by itself.
Maxmem is however much memory you can give it, 1000 works well for me.

Step 2a) let it start from scratch
opt_results = find_graphs('C:/wherever/you/told/the/f2/to/go', pops = c('pop1, 'pop2.DG', 'pop3.SG'), outpop = NULL, numrep = 1, numgraphs = 50, numgen = 20, numsel = 5, numadmix = 3, numstart = 1, stop_after = 1800, verbose = TRUE)
winner = opt_results %>% top_n(1, -jitter(score))
winner$score[[1]]
write_dot(winner$edges[[1]])
plot_graph(winner$edges[[1]])

Step2b) or give it a starter graph to work with
opt_results = find_graphs('C:/wherever/you/told/the/f2/to/go', outpop = NULL, numrep = 1, numgraphs = 50, numgen = 20, numsel = 5, numadmix = 12, numstart = 1, keep = c("best"), initgraphs = 'C:/wherever/you/have/a/graphfile.txt', stop_after = 1800, verbose = TRUE)
winner = opt_results %>% top_n(1, -jitter(score))
winner$score[[1]]
write_dot(winner$edges[[1]])
plot_graph(winner$edges[[1]])

[TuaMan]

Knowing nothing about R, what would be the ideal version for this new Admixtools? Since R is Windows compatible, that means we can just download a version of R as its own standalone application and run it directly out of our Windows PC/Laptop right?

EDIT: I see on github the author mentions R version 3.5 or higher, but I know from Linux newer distributions aren't always necessarily across the board better for certain applications.

[Kale]

Yep, just download an R binary for Windows and good to go. I was using 3.4.3 to run nmonte, which being less than 3.5 meant I needed a new one, I just got the newest (4.0.3) and it seems to be working fine.

[TuaMan]

Downloaded latest R, Rtools40, and Admixtools itself. When I run the command to install the interactive app:

devtools::install_github("uqrmaie1/admixtools", dependencies = TRUE)

I get this message:

Skipping install of 'admixtools' from a github remote, the SHA1 (c312fe0a) has not changed since last install.
Use `force = TRUE` to force installation

Is this an expected message? Am I going to blow something up if I try forcing the install?