You're trying to assign agency to a natural process. It doesn't work like that no matter how many times you ask why.

The biggest element you are not accounting for is time. It takes an unfathomably long amount of time for the benefits of random mutation to shape a population.

how does nature know

It doesn't. Step by tiny step random mutations that grant a fraction of a percentage of survivability, multiplied across millions of generations.

random mutation /survival etc just doesn’t explain why it happens.

Yes it does, assuming there's a benefit to be had along the way. It doesn't have to be as effective as it is today to confer some tiny amount of benefit. It just has to be better than those without the mutation.

Evolution can only build off what came before, and in this case, all the parts were already there, they just needed to be fine tuned. Beetles across the world manipulate wavelengths of light to iridesce using variable reflective layers of chitin.

Nature as a whole is way beyond our present understanding of math and science. In fact, math and science only exist as concepts for humans to try to explain elements of nature

There doesn't need to be any knowledge involved. It happens, because it works. Neither the beetle nor evolution itself "know" anything about quantum physics. The beetle is just a beetle and evolution is not even an entity that has any agency, it's just a process that's happening and that leads to remarkable results over time.

This is just one more example for the old discussion how complexity can develop through evolution. The classic example is the eye of vertebrates. Read up on that, if you're interested in that discussion.

There's nothing "special" in the way you imagine about quantum phenomena. They are complicated to describe mathematically because we are limited to a fundamentally imperfect set of symbols, but they are not complicated to obtain the requirements for.

All chemical and light-interaction processes use quantum phenomena if you dig enough into how they work, and it's especially clear on a smaller scale. If you just make something thin enough, it will start displaying quantum effects, but there is nothing that complicated about "thin".

They're not manipulating wavelengths with great complexity, they're just growing a really thin layer on their shell.

It didn't. It just does random shit in different offspring until they manage to survive and pass on the genes to the next generation and the next round of random shit, ad infinitum.

It's always funny seeing arguments like this as someone with a computer science education. A lot of people act like you can't have anything complex unless some intelligent being deterministically writes a lot of if-else statements to implement it, which requires them to know and understand in detail what they are implementing at every step.

But what people don't realize is that this is not how it works at all, there are many problems that are just impractical to actually "know" how to solve yet we solve them all the time, such as voice recognition. Nobody in human history has ever written a bunch of if-else statements to be able to accurately translate someone's voice to text, because it's too complicated of a problem, no one on earth knows how it works.

Yet, of course, your phone can do voice recognition just fine. That is because you can put together a generic class of algorithms which find solutions to problems on their own, without you even understanding how to solve problem. These algorithms are known as metaheuristics. Metaheuristics fundamentally cannot be deterministic, they require random noise to work properly, because something that is deterministic will always greedily go in the direction of a more correct solution, and will never explore more incorrect solutions, whereby an even better solution may be beyond the horizon of many incorrect ones. They also do have to be somewhat deterministic as well, because you need some greed or else the random exploration would be aimless.

A simple example of a metaheuristic is that of annealing. If you want to strengthen a sword, you can heat up the metal really hot and let it slowly cool. While it's really hot, the atoms in the sword will randomly explore different configurations, and as it cools, they will explore less and less, and the overall process leads them to finding rather optimal configurations that strengthen the crystaline structure of the metal.

This simple process can actually be applied generally to solve pretty much any problem. For example, if you are trying to figure out the optimal route to deliver packages, you can simulate this annealing process but rather than atoms searching for an optimal crystaline structure, you have different orders of stops on a graph searching for the shortest path. The "temperature" would be a variable that represents how much random exploration you are willing to accept, i.e. if you alter the configuration and it's worse, how much worse does it have to be for you to not accept it. A higher temperature would accept worse solutions, at very low temperatures you would only accept solutions that improve upon the route.

I once implemented this algorithm to solve sudoku puzzles and it was very quick at doing so, and the funny thing is, I've never even played sudoku before! I do not know how to efficiently solve a sudoku puzzle, I've honestly never even solved one by hand, but with sudoku it is very easy to verify whether or not a solution is correct even if you have no idea how to find the solution and even if finding it is very difficult, verifying it is trivially easy. So all I had to do is right the annealing algorithm so that the greedy aspect is based on verifying how many rows/columns are correct, and the exploration part is just randomly moving numbers around.

There are tons of metaheuristic algorithms, and much of them we learn from nature, like annealing, however, there's also genetic algorithms. The random exploration is done through random mutations through each generation, but the deterministic and greedy aspect of it is the fact that only the most optimal generations are chosen to produce the next generation. This is also a generic algorithm that can be applied to solve any problem. You can see a person here who uses a genetic algorithm to teach a computer how to fly a plane in a simulation.

Modern AI is based on neural networks, which the greedy aspect of them is something called backpropagation, although this on its own is not a metaheuristic, but modern AI tech arguably qualifies because it does not actually work until you introduce random exploration like a method known as drop out whereby you randomly remove neurons during training to encourage the neural network to not overfit. Backpropagation+dropout forms a kind of metaheuristic with both a greedy and exploratory aspect to it, and can be used to solve just about any generic problem. (Technically, ANNs are just function-approximators, so if you want to think of it as a metaheuristic, the full metaheuristic would have to include all the steps of creating, training, and then applying the ANN in practice, as a metaheuristic is a list of steps to solve any generic problem, whereas an ANN on its own is just a function-approximator.)

Indeed, that's how we get phones to recognize speech and convert it to text. Nobody sat down and wrote a bunch of if-else statements to translate speech into text. Rather, we took a generic nature-inspired algorithm that can produce solutions for any problem, and just applied it to speech recognition, and kept increasing the amount of compute until it could solve the problem on its own. Once it solves it, the solution it spits out is kind of a black box. You can put in speech as an input, and it gives you text as an output, but nobody really even knows fully what is going on in between.

People often act like somehow computers could not solve problems unless humans could also solve them, but computers already have solved millions of problems which not only has no human ever solved but no human can even possibly understand the solution the computer spits out. All we know from studying nature is that there are clever ways to combine random exploration and deterministic greed to form processes which can solve any arbitrary problem given enough time and resources, so we just implement those processes into computers and then keep throwing more time and resources at it until it spits out an answer.

We already understand how nature can produce complex things without anyone "knowing" how it works, because we do that all the time already! You do not need a sentient being to tell the beetle how to evolve to fit into its environment. There is random exploration caused by genetic mutations, but also a deterministic greedy aspect caused by "survival of the fittest." This causes living organisms to gradually develop over many generations to something fit for its environment. And life has had plenty of time and resources to become more suited to its environment, life has been evolving for billions of years, with the whole resources of the planet earth and the sun.

So what factors affect this then

Easy, mature killed all the other Beatles of the same species which were not shiny. Then probably the shiny females only liked shiny males for mating. Finally the male penis got some weird curly twist and that eventually locked the mutation to just one species. I don't known, just making stuff up.

A bit like razzle dazzle!