Saturday, January 26, 2019

Testing Follies 2019

I’ve finally lost all confidence in the commercially available “state of the art.”

A recent article I just discovered.  This is just too much.

Excerpts, emphasis added:
To check out the accuracy of these test kits, Charlsie Agro – who fronts Canadian telly watchdog show Marketplace – and her identical twin sister Carly used them to submit their DNA to five separate consumer-grade genetic-testing outfits, and compared their results.

And their results were surprisingly varied. For one thing, the tests couldn't agree on where exactly their ancestors actually physically came from. Test kit supplier 23andMe reckoned the twins are about 40 per cent Italian, and 25 per cent Eastern European; AncestryDNA said they are about 40 per cent Russia or Eastern European, and 30 per cent Italian; and MyHeritageDNA concluded are about 60 per cent Balkan, and 20 per cent Greek.
As bad as that is, it gets worse.
Two of the tests reported that the twins had no Middle Eastern ancestry, while the three others did, with FamilyTreeDNA saying 13 per cent of their sample matched with the region.
13% vs 0%!  I need to rethink my previous comments in which I stated that anything less than 10% was questionable; now I’ll put that up to, say, 20%. Maybe 25%.  Or maybe anything less than majority ancestry.  Or maybe not even that – in the case described above, the tests could not even agree on what the predominant ethnic ancestry was.  It would seem therefore that the only dependable metric for these tests is what the majority racial-continental ancestry is – and everything after that is a crapshoot (possibly depending on parental populations, see below).  Has anything improved since DNAPrint 15+ years ago?  
On top of this, each test couldn't quite agree on the percentages between the sisters, which is odd because the twins share a single genetic profile.
Identical twins for godssakes!  We are not talking about siblings, who may have inherited different ancestral components from the parents, but identical twins!   A useful control would have been to have the same twin submit the sample twice.  Would the results have been anything consistently close even then?  

23andMe I’ve completely lost confidence in at this point and the other tests named in this article do not seem to be much better. I may still look into some of these other ones in more detail, to see if more parental populations improve results, but given what’s reported here it doesn’t seem promising.  But we’ll see. It may be a more accurate (but not more precise?) version of nonsense.  Improved nonsense, if you will.

Another basic problem, which I’ve written about before, and that may be contributing to differences between tests, is how different ancestral components are labelled. This may also be the reason why folks online are complaining about how their ancestral percentages change markedly when a test gets “updated.”  What one test, or version of a test, labels as “ancestry X” may be labeled by another test or test version as “ancestry Y” – even though of course the actual gene sequences do not change.  Different nomenclature, different algorithms, different parental populations end up giving inconsistent and sometimes absurd results.  DNAPrint was criticized - with justification – when different versions of the test gave radically different results to the same customers; the same principle applies here.  

I won’t even repeat discussing in detail the retardation of “50% confidence levels.”  That is essentially an educated guess. That’s also related to the parental population problem I think. One test of that hypothesis is whether a person’s percentage of “unknown ancestry” markedly increases as the chosen confidence interval increases.  If so, there is obviously a lack of parental population coverage for that customer, and the company is making, at best, an educated guess as to what certain haplotypes represent, based on whatever parental samples they happen to have in their database, how their algorithm works – and how they decide what to name a particular ancestry based on some criterion or another.

It would follow that those tests that use the greatest number and variety of parental populations would provide the most accurate results (one still wonders about precision). Most people do not understand the importance of parental population choice.  We can take this to an absurd extreme for the purposes of illustration.  Assume Joe Schmoe is used as the parental sample for population X.  If we then take Joe’s DNA and test it against the metrics of X – established with Joe’s own DNA – then in theory Joe will test out as “100% X.”  He’s being compared to himself.  Although given what is described above for the Agro sisters, one wonders if every time Joe is tested his values would markedly fluctuate 10-15 % (or more).  Even with a lack of precision, one would assume accuracy of ancestral determinations would be optimized with an increasing number of parental populations (assuming the samples used are actually indigenous representatives).

Ultimately however, the value of any of these tests is the raw genetic data, which can be plugged into genetic kinship assays, to get a more objective measurement of genetic relatedness and genetic interests.  The rest of this is borderline useless. I note for the record that none of these companies (that I know of) offers genetic kinship analysis as part of their metrics.  A telling omission.  Why not give that data?  As far as biopolitics goes, with respect to EGI, the only metric of importance is genetic kinship.  And again, that can be calculated from raw data, bypassing the nonsense and political biases of companies.
Ancestry got back to us with a long statement that did not address any specific points raised by the Marketplace twins report. "Genomics is advancing rapidly, and as an industry leader Ancestry remains committed to investing in and creating ‘what’s next’," was the most coherent sentence a spokesperson could give us.
“An industry leader” – a left-handed self-compliment.
A spokesperson for 23andMe declined to comment on the record.
Surprise!  Did they say anything off the record?