Nebula Genomics

[francisberrigan]

Thank you @Jatt1,

finally managed to do this today, as it took five days for the huge bam file to be uploaded to dropbox and google drive so that i could see which link(s) would work with both ydna-warehouse and yfull (as the bam file is far too big at 230GB for direct upload)

(and this only after i upgraded my cable internet service to 1GBdownload/100MBupload and also found download extensions for my browsers so i didn't have to restart the downloads/uploads each time there was any disconnect or annoying windows update restart) -- this discovery means i'll be switching to fibre optic connection for incredible upload speeds over my current cable connection (especially, unbelievably, now that fibre turns out be much less expensive and far more reliable...)

-- i also uploaded to sequencing.com and they gave me a substantial discount on the main app after i found a spelling mistake in their uploading tool (!)

-- impute.me allowed me to upload straight to the site with the nebula .vcf.gz file which apparently is the same for dante results, though i only learned this after a conversation with the site's author, lassefolkersen, who was incredibly gracious (as that availability is not mentioned on the site), and of course donated to the site the equivalent $15 CDN that promethease.com requires instead of the mere $5 that impute suggests (considering impute seems to have near exponential greater functionality)

as an adoptee and a single father, having genetic health tools is invaluable over either nonexistent/minimal/incorrect (beliefs versus facts) family health history

it will still be a few days because of the file size before i can starting to look at those sites' tools/apps/algorithms.

[aaronbee2010]

So in case you haven't checked in recently, @aaronbee2010, here is an update/follow-up:

1. solved: embarrassing unawareness on my part -- as you know wgs fastq files are supposed to be paired end reads!
2. solved: i installed the complete python package from the windows store, restarted, and everything worked -- so that might help others
3. solved: don't need the java workaround (but don't forget to mind the gap on your previous/future posts!)

4. just found and am seeing if the bam analysis kit v2 might be a workaround but not sure if you think it is as robust as wgsextract
5: do you think uploading to yfull will help with paternal ancestry above all the other consumer dna data i've already enrolled in (along with connected gedcoms) as my new birth father is also very interested in our results
6. i'm not clear if converting cram --> bam --> 23andme v5 means that a great deal of information is filtered out/downgraded and whether if i upload the eventual wgsextract export to my existing dna membership sites if it's even worth it?

7. i found the site and the file https://ftp.ncbi.nlm.nih.gov/genomes...28_GRCh38.p13/ (updated reference genome) -- is there a reason not to use this for the export (compatibility/etc.)?
8. my wgextract export file (combined all) had LESS segments, cM, and overlap connections than using the "create a superkit" tier 2 tool on gedcom (which combined all my other site dna exports) -- how would that be possible?

thanks for all your previous posts and in advance for any responses here
and i wish i would have read your advice before i upgraded from 30x

p.s. i too am very irritated the nebula-->ftdna partnership has so far been vaporware though i've seen far too much of ftdna's greedy agenda to be surprised by the negotiation/integration delays

Francis Berrigan
R --> M207 --> M173 --> M343 --> L754 --> L389 -> P297 --> M269 --> L23 --> L51 --> L151 --> P312 --> Z290 --> L21 --> DF13 --> Z39589 --> S1026 --> Z16886 --> R-Z16903 (but only up to bigy700)
U5 --> U5b1 --> U5b1h

Sorry. I saw your message a few days ago but I've been very busy with work (I work 7 days a week and 6 of those days (Mon-Sat) are shifts that start from 6:30 am onwards - generally when I get home, I'm too tired to do anything productive) hence this reply taking a while.

Firstly, congratulations on connecting with both birth families of yours. It's always nice to see people in your situation find the information they're looking for and it made for a nice read too

I was aware of the 0000 gap from the post about the CRAM-to-BAM conversion, however it's an issue I wasn't able to correct (despite what common intuition may suggest) when typing out the message surprisingly. At the very least I should've made a footnote about it, so my apologies there!

In regards to the numbered questions you have, here are my answers for each one:

4. Can I ask for clarity regarding what you wish to use this program for? Since you seem to have solved the WGSExtract program hanging, that program should be used to generate a simulated 23andMe v5 (or any other vendor/chip version file or even a combination of these) file. I believe the BAM Analysis Kit (BAK) performs a different set of functions (the Y-STR determination seems to be the main one in my opinion). While the BAK states that it can calculate admixture, GEDmatch/Admixture Studio are much better suited to something like this, and require the simulated file from WGSExtract. This also means you only need to perform a variant-calling pipeline once, as opposed to the BAK requiring you to perform this every time you want to calculate admixture if I'm not mistaken. Hopefully @teepean could provide more clarity on that, and I apologise to him in advance if I've stated something incorrect.

To answer your update, I think it's a robust program but it appears to me that it might not be suitable for what you're trying to achieve (making that simulated file).

5. Very small numbers of SNPs can be present in a WGS but not a BY700 (due to much higher length (not depth) of coverage) and vice versa (due to anything except a 23 million base-pair region of the Y chromosome being filtered out during sample preparation, preventing your autosomal reads from mapping to the Y-DNA reference, or reads from one part of your Y chromosome being allocated to the wrong part of the Y-DNA reference).

Having said this, the vast majority of SNPs found in one file are also found in the other, so there's a considerable chance that you won't gain anything from uploading your Nebula data, so I would consider whether or not it's worth the money to you.

As far as mtDNA goes, if you already have your mtFull Sequence uploaded to YFull (assuming this is the test you've purchased from FTDNA for your maternal ancestry) then you won't gain anything at all (unlike Y-DNA) from the Nebula BAM file being uploaded to their mtDNA tree, although this does come with the Y-DNA analysis of your Nebula BAM file. Again, it comes down to whether or not you want to upload your Nebula BAM to YFull for Y-DNA purposes.

6. For CRAM > BAM, there's no loss of information at all. When going from BAM to 23v5, only around 640'000 genomic positions are evaluated (around 0.02% of the human genome) then written to a text file. Information regarding the quality (i.e. base quality, MAPQ, read depth etc.) of the genotype in each position that's present in the BAM file does not make it to the 23v5 file. This does result in a colossal loss of information from BAM > 23v5, however your BAM is 230GB (and that's compressed, mind you!) and the 23v5 file produced is probably 20MB at most (without compression), so there's that.

This loss of information would only be a problem for you if you were to delete any BAM/CRAM files on your system after producing the 23v5 file though. As far as those "DNA membership sites" you mentioned go, everything depends on the files they accept. Is it okay if I ask you what websites these are?

7. Since simulated 23v5 (and other chip vendors/versions) only contain the primary chromosome sequences and the mitochondrial sequence, the additional sequences present in GRCh38.p13 aren't required by WGSExtract, so don't worry there. WGSExtract includes a few reference genomes within its folders, which are suitable enough for its purpose.

8. This issue isn't unique to you, it's on GEDmatches end. Here's an article explaining it: http://www.beholdgenealogy.com/blog/?p=2963

If you have any more questions, I'll be happy to answer them as well. Thanks for being patient!

[TigerMW]

I agree the integration delays for the Nebula/FTDNA partnership have been slow. Keep in mind it is not necessarily evil or "greedy" capitalists at work. I'm not sure if it was highly profitable or not but a tremendous amount of resources were redirected to support pandemic testing. This is what the government encouraged, rightfully, I think.

[francisberrigan]

Thanks for all the information!

4. I was hoping, after paying $1000 CDN total for the nebula 100x (and money is tight during covid quarantine), that it would give me a great deal more information to upload to gedmatch and my consumer sites, but you point out that cram --> bam (no loss) --> 23andme v5 (downgraded to .1% of genome/snps) -- so there's not much difference there. Hopefully won't matter after nebula and ftdna sort out their partnership (though this doesn't help me uploading full cram/bam to my other consumer sites or gedmatch). Will check out Admixture Studio though, thanks!

The .1% thing makes sense now though, as we are all of 99.9% African descent, with only the local environment variable .1% difference which is why the consumer dna sites focus entirely on that minuscule amount to allow their users/paying customers to live in perpetuated belief systems as opposed to scientific fact (don't want them to have to face reality and truth!).

5. I have just now uploaded my full bam to yfull, ydna-warehouse, sequencing.com, and impute.me (due to delays with file-size uploading) so I'll see what the results are in the next few days (apparently they all get very few 100x nebula files and the size is slowing everything down) -- though not a bam file but the nebula vcf.gz file at impute

6. As I'm in Communications, I found the birth mother within 5 minutes (social media/traditional media trail), but it took 16 months to find the birth father -- as mentioned in an earlier post reinventing the intersecting spider web technique -- which is why i kept adding consumer dna sites... by the end I had tested with 23andme, Ancestry, FTDNA (family, mito, up to bigy700 -- which is the evil greedy capitalist part as they should just be offering 30x wgs which offers exponentially greater connections/information for only $299 total as opposed to milking/monetizing every step/tier/channel for christmas bonuses), LivingDNA, and National Genographic (which closed down last July), and had uploaded to Gedmatch, WeGene, MyHeritage, DNAland, Genesis, Ysearch and Mitosearch (before they closed down), Promethease, and WikiTree.

I also used Ancestry for my new half-brother to confirm father-side (paid that test for him because at first he thought my showing up out of nowhere after 50 years might have been some sort of grift (!) and because he had already built a family tree there) as well as my new Aunt to confirm mother-side (as she was also shocked I had survived birth secretly on the opposite coast 4200km away) and then confirmed my new birth father there as well (then added them to promethease, ftdna, myheritage, gedmatch, etc.).

7. Thanks, I had found the latest reference genome, and am always wanting to use that, and contacted the government agency hosting the files (they had a mistake in one of the files) and they were kind enough to also tell me they will be updating the latest reference genome (in that link I shared) as per: "Datasets resource will be adding support for the alt/patch sequences in an upcoming release" so we can use that updated reference with WGSExtract hopefully soon!

8. Thanks for info on why nebula --> gedmatch upload didn't make a difference! Still bizarre to me that the "super kit" (combining all my consumer dna tests) at tier 2 with gedmatch provided more matches and connections than the WGSExtract combined export based on a wgs bam.

9. Yes, one last question, that great code snippet you made for cram --> bam conversion using the one reference genome, do you have/use a similar snippet for converting the raw .fna file (latest reference genome) to a big zip export/output .gz file using bgzip.exe (so I can use with the WGSExtract program) -- I do not have the python knowledge to do this. Had already unzipped the zipped .fna file because WGSExtract doesn't like non big zip .gz files, but the bgzip.exe does not just open in windows environment and requires a similar command prompt snippet to work its magic.

Truly appreciate your generous feedback.

[TigerMW]

P.S. @TigerMW the sociopathic capitalist standard-operating procedures at FTDNA, 23andme, MyHeritage, and Ancestry are trivially obvious compared to the much more evil racist/bloodline/eugenics agendas behind the origins of several of the consumer sites and their specific tribal/cult beliefs that are completely at odds with the scientific proof offered by the Human Genome Project that there is only one human race. And imagine the business/income fallout of telling inbred hillbilly flatearther/antivaxxer/qanon/5gconspiracy/altfacts/white wing/trump voting 'mericans that their DNA tests prove they are all actually 99.9% African with only 0.1% local environment variable "ethnic" flavouring.

P.P.S. Although they'd probably just start proudly calling themselves Point One Percenters.

It’s okay. I apologize if I offended you. I think we’ll all be okay with a little patience and goodwill.

[francisberrigan]

And in case anyone is interested in nebula 100x bam stats according to WGSextract (which put my mind at easy regarding actually receiving the 100x upgrade):

wgsextract bam stats.png

I'm guessing the M avg read depth is actually 112x or 102x and a glitch?

There might be technical reasons, but disappointing the X and Y avg read depths are the lowest...

[aaronbee2010]

Thanks for all the information!

4. I was hoping, after paying $1000 CDN total for the nebula 100x (and money is tight during covid quarantine), that it would give me a great deal more information to upload to gedmatch and my consumer sites, but you point out that cram --> bam (no loss) --> 23andme v5 (downgraded to .1% of genome/snps) -- so there's not much difference there. Hopefully won't matter after nebula and ftdna sort out their partnership (though this doesn't help me uploading full cram/bam to my other consumer sites or gedmatch). Will check out Admixture Studio though, thanks!

The .1% thing makes sense now though, as we are all of 99.9% African descent, with only the local environment variable .1% difference which is why the consumer dna sites focus entirely on that minuscule amount to allow their users/paying customers to live in perpetuated belief systems as opposed to scientific fact (don't want them to have to face reality and truth!).

5. I have just now uploaded my full bam to yfull, ydna-warehouse, sequencing.com, and impute.me (due to delays with file-size uploading) so I'll see what the results are in the next few days (apparently they all get very few 100x nebula files and the size is slowing everything down) -- though not a bam file but the nebula vcf.gz file at impute

6. As I'm in Communications, I found the birth mother within 5 minutes (social media/traditional media trail), but it took 16 months to find the birth father -- as mentioned in an earlier post reinventing the intersecting spider web technique -- which is why i kept adding consumer dna sites... by the end I had tested with 23andme, Ancestry, FTDNA (family, mito, up to bigy700 -- which is the evil greedy capitalist part as they should just be offering 30x wgs which offers exponentially greater connections/information for only $299 total as opposed to milking/monetizing every step/tier/channel for christmas bonuses), LivingDNA, and National Genographic (which closed down last July), and had uploaded to Gedmatch, WeGene, MyHeritage, DNAland, Genesis, Ysearch and Mitosearch (before they closed down), Promethease, and WikiTree.

I also used Ancestry for my new half-brother to confirm father-side (paid that test for him because at first he thought my showing up out of nowhere after 50 years might have been some sort of grift (!) and because he had already built a family tree there) as well as my new Aunt to confirm mother-side (as she was also shocked I had survived birth secretly on the opposite coast 4200km away) and then confirmed my new birth father there as well (then added them to promethease, ftdna, myheritage, gedmatch, etc.).

7. Thanks, I had found the latest reference genome, and am always wanting to use that, and contacted the government agency hosting the files (they had a mistake in one of the files) and they were kind enough to also tell me they will be updating the latest reference genome (in that link I shared) as per: "Datasets resource will be adding support for the alt/patch sequences in an upcoming release" so we can use that updated reference with WGSExtract hopefully soon!

8. Thanks for info on why nebula --> gedmatch upload didn't make a difference! Still bizarre to me that the "super kit" (combining all my consumer dna tests) at tier 2 with gedmatch provided more matches and connections than the WGSExtract combined export based on a wgs bam.

9. Yes, one last question, that great code snippet you made for cram --> bam conversion using the one reference genome, do you have/use a similar snippet for converting the raw .fna file (latest reference genome) to a big zip export/output .gz file using bgzip.exe (so I can use with the WGSExtract program) -- I do not have the python knowledge to do this. Had already unzipped the zipped .fna file because WGSExtract doesn't like non big zip .gz files, but the bgzip.exe does not just open in windows environment and requires a similar command prompt snippet to work its magic.

Truly appreciate your generous feedback.

4. For health-oriented websites, you're best off uploading the VCF file. As far as health goes, this is where the increase in 30x to 100x coverage is the most justified in my opinion.

5. I'm not sure how I feel about imputed data in this particular usage scenario. From a genealogical standpoint, it seems to make more sense to use it on microarray data which already lacks data on the vast majority of the human genome from the get-go, whereas most of that information is present in your BAM and VCF files.

6. -

7. The problem here is that the microarray tests that WGSExtract emulate don't have any information on genomic positions which occur within unplaced/unlocalised/alt/patch sequences, so you would derive no benefit from generating a BAM file with an alt/patch hg38 reference solely for use with WGSExtract. Where you will yield improvements in the WGSExtract output data is by using a GRCh37 BAM file (as some genomic positions used by these microarray tests are only present in GRCh37 and not hg38), but generating one of these files isn't easy at all and may not be worth the results depending on the individual. There are instructions on how to generate a basic BAM file in section 3.3.6 of this document, but these instructions doesn't follow industry-wide best practices for post-processing the BAM file after it's been generated (i.e. marking duplicates). This most likely won't result in any difference when using WGSExtract, but may do for other purposes outside of WGSExtract.

8. -

9. Please refer to answer #7 above. There's no reason to use a reference with alt/patch contigs (I'm assuming this is what you're referring to when you mentioned "latest reference genome" in closed brackets) in WGSExtract. If I'm wrong then please clarify what you mean exactly.

Thanks for waiting.

[aaronbee2010]

And in case anyone is interested in nebula 100x bam stats according to WGSextract (which put my mind at easy regarding actually receiving the 100x upgrade):

wgsextract bam stats.png

I'm guessing the M avg read depth is actually 112x or 102x and a glitch?

There might be technical reasons, but disappointing the X and Y avg read depths are the lowest...

The M average read depth here is correct. Mitochondria are abundant in cells, which explains why they get sequenced to much higher coverage than chromosomes.

X and Y coverage is lower for biological males as we only have one copy of X-DNA and Y-DNA per cell, whereas we have two copies of each autosome per cell.

[MacUalraig]

Apologies if this isn't news but I just noticed that the YFull order form now says 'BAM/CRAM'

NextGen Seq raw data file type
.BAM/.CRAM .VCF

https://yfull.com/order/

I don't know when they changed that, on the wayback machine it said BAM on 6/19 but BAM/CRAM in 12/20. (I just let them log on to my account although I'm not suggesting others do that, I have built up trust with them over a very long period of time). Funny I didn't notice as I've done several uploads since it changed.

[rock]

And in case anyone is interested in nebula 100x bam stats according to WGSextract (which put my mind at easy regarding actually receiving the 100x upgrade):

wgsextract bam stats.png

Your Nebula 100x WGS Y coverage is only 33x lower than yseq 50x WGS, Y coverage is 40x.
y40.PNG