A (non)-Bias in Primes
There is a bias in primes, typically known as Chebyshev's Bias or the "Prime Race". Basically, this bias says that if you start going along the number line and sort the primes into two bins as you go along, one bin for prime that are 1 mod 4 and the other for those that are 3 mod 4 (we can ignore2), then for 99% of this "race" there will be more primes in the 3 mod 4 bin. That is, most of the time there are more primes that are 3 mod 4 than there are primes 1 mod 4.
This works in greater generality. If we sort primes mod N then there will (usually) be more primes in buckets x mod N where x is not a square mod N than in the buckets y mod N where y is a square mod N. It seems as though the primes try to avoid being quadratic residues as much as they can.
This is a bit odd in light of Dirichlet's Theorem on Primes in Arithmetic Progressions, which says that the primes are all equidistributed in appropriate residue classes as a whole. Though, it isn't inconsistent with this result, as it merely says that at the end - even if non-residues are in the lead for most of the race - it will end in a tie. But this bias still exists and there should be a meaningful reason for it.
To dig into these results a bit we, naturally, end up working with the Zeta function and its friends. These do, after all, hold information about how primes are distributed within its analytic properties. But with these functions (and some strong Riemann Hypothesis assumptions) we can prove this bias and quantify it pretty well (see here for more). But with this analysis we can actually see how we are, in a way, creating this bias.
The zeta functions do not tell us about primes. They tell us about prime powers. A lot of results about primes which use the zeta function actually prove a related result for prime powers, and then get rid of the contributions to higher prime powers through some kind of approximation or correction. You can see this happen in the Divergence of the Sum of Reciprocal Primes and in the Prime Number Theorem when we pass to the Chebyshev Function. Overall, the zeta functions tell us that prime powers are the natural objects of study when it comes to the distribution of primes.
The proof of Chebyshev's bias is along the same lines: Get rid of the contributions of higher powers through some asymptotic bound. But through this process is when we end up creating the bias itself. There is no Chebyshev's Bias for the prime powers and it is through the exclusion of higher powers that we create the bias. To see how this works, we can consider the case of sorting the prime powers mod 4. When we do this, every prime-square will fall into the 1 mod 4 bucket. That's a lot of numbers that automatically go to one of the buckets and to balance it out we will need the primes to act as counterweights which means favoring the 3 mod 4 bucket. And so Chebyshev's bias exists to prevent a bias in the distribution of prime powers.
I'm not the best at making graphics, but you can track how often one bucket is favored over the other as a percent and when you compare primes to prime powers, you can see the bias disappear in this graph). There are some scaling factors in here, but these arise from how Chebyshev's function tells us we should weight prime powers when counting them. Overall we can see that, in the end, Chebyshev's bias is an anti-biasing mechanism.
But, in the end, Terry Tao can explain things much better than I, though he didn't include a poorly made Python graph to illustrate things.
Posted Jan 22, 2021, 4:49 AM UTC
Your observation about the contribution from squared primes being the source of the bias is also described by Granville and Martin. See the second and third paragraphs of p. 21 of their paper: https://arxiv.org/pdf/math/0408319.pdf.
When comparing the Chebyshev bias with Dirichlet's theorem, you write
but this is apples and oranges: the quantity being measured in Chebshev's bias is a difference of two prime number counts up to x while Dirichlet's theorem is about a ratio of prime number counts up to x. The tie at the end in Dirichlet theorem is about the limit of certain ratios of prime number counts, and such ratios completely wash out anything involving differences. So it's misleading to say Dirichlet's theorem is consistent or inconsistent with Chebyshev's bias, as they're not about the same kind of prime counting.
Here's an analogy. The functions x^2 + x and x^2 + sqrt(x) both have a ratio with x^2 that tends to 1 and a difference with x^2 that goes to infinity, so if you consider the asymptotic relations x^2 + x ~ x^2 and x^2 + sqrt(x) ~ x^2 as saying there is a "tie" in the growth rates, that is neither consistent nor inconsistent with the behavior of the differences. Knowing two functions are asymptotic is operating on a different scale than knowing something about their difference.