These explanations may all be right. But there is a common factor which is neglected in the media: The speculation mania for commodities has ebbed up. There has been a time when betting on climbing commodity prices was very popular and successful - the epoch from around 2003 till 2014 (with a short break in the recession years 2008/09). In these years commodities were the darlings of the speculators. Bank analysts and other pundits recommended to invest generally into commodities. They claimed that oil, industrial metals and agriculture commodities will get scarce because the global demand for them should rise faster than the supply, especially in China, India and other emerging markets.
These claims weren`t new. In the early 19th century the English cleric Robert Malthus predicted the world would drift into a catastrophe because the rising global population will need more food than the world can produce. He claimed that "the power of population is indefinitely greater than the power in the earth to produce subsistence for man" (wikipedia).
Even though that Malthus has been continuously proven wrong by a constantly rising commodity production and a climbing global wealth - his theories are still popular. In the 1970s the "Club of Rome", a huge organization of pundits, predicted that the world will eat her resources soon ("The Limits to Growth"). Notwithstanding that the dire prediction got refuted in the economic boom years at the end of the 20th century - with abundant and cheap oil and other commodities - around the year 2003 started a Malthusian revival: Banks proclaimed the super cycle for commodities and claimed that oil producers and other commodity suppliers would not be able to meet the rising demand (driveby). Many believed in the "peak oil theory" which asserted that the peak of oil production has been reached and that a climbing global oil demand will meet falling supplies, which should lead to sharp rising prices.
The Malthusian revival lead to a speculative buying spree: Hedge funds, banks, brokers, specialized commodity funds and other investors subscribed into the "super-cycle" theory and pumped billions of dollars into the commodity markets. The commodity speculators bought futures, ETFs and other financial instruments which were backed by commodities. Thanks to the financialization of the commodity sector prices for different commodities like oil, copper and sugar all moved in the same direction.
The massive commodity purchases worked as a self-fulfilling prophesy: Prices for oil, sugar, gold, wheat, cacao and a lot other commodities went up because of the speculative demand. These price gains seemed to confirm the Malthusian claims and attracted more speculators. Many jumped onto the "Malthusian" bandwagon and bought commodities (or financial products based on them) just because their prices were climbing in the hope that prices will go further up (momentum players) - the herding behavior of the commodity speculators lead to a snow ball effect.
So the speculative purchases created price bubbles for oil and many other commodities. But the elevated prices over many years did what high prices always do: They animated the producers to produce more and prompted the consumers to consume less. In the US people invested into shale oil, worldwide miners drew more metal from the earth and farmer planted more crops, all driven by higher prices & profits. The technological progress and a learning process also raised the efficiency of the commodity production which lead to a climbing supply.
Long Term Trend
It seems that we are back on the long term (secular) trend of falling commodity prices: "There has been a downward trend in real commodity prices of about 1 percent per year from 1862-1999 (commodity prices fell 1 per cent p.a. relative to the general price level) (jstor) .
As described above this trend has been temporary interrupted thanks to the hype induced by banks and brokers, who earn a lot money with trading futures and other commodity related financial instruments. I agree with scientists who claim that "the dramatic price increases over the last decade were just a temporary blip up within the context of a longer-run trend of stable or declining relative prices"(econbrowser).
The secular trend of falling commodity prices is based on human nature and factors like ingenuity, frugality and profit orientation. In fact, people have been responding to scarcity and rising commoditiy prices all the time. Generally higher prices have been animating producers to produce more because they get higher profits. On the other side, consumers have been switching to cheaper alternatives and have been developing methods to deal with rare resources for thousands of years.
The shale oil revolution in the US is just one example for why commodities get cheaper over the time. Technological progress and human ingenuity are reducing the cost of oil exploration and are enabling investors to tap reserves which have been to costly before. Farmers use more and more machines for planting & harvesting and benefit from improvements in irrigation, fertilizer, pesticides, and hybridization. Technological progress leads to advanced seed and fertilizer products, along with more efficient agricultural equipment liker tractors, loaders, harvesters. All are enhancing the productivity of farming. And miners also use better machines and new technologies to prospect metals & coal.
On the other hand, technological innovations lead to a demand destruction: Cars are becoming more and more energy efficient; modern refrigerators, washing machines, air conditioners and many other household appliances use less electricity than older models. Consumers & companies also are using metals and other commodities more economically.
Demoralizing Commodity Speculation
Low and falling prices are already demoralizing the commodity speculators because they are losing money now. The speculative money, that has ballooned commodity prices in the recent years, is now oozing out of the commodity sector. Hedgefunds, which specialize in commodities, are already losing customers and are shrinking because their profits turned into losses (bloomberg). I suppose that this exodus will foster a de-financializing of commodities. The sales of hedge funds and other speculators (downsizing of speculative positions) will squeeze commodity price further down.
The situation reminds me of the epoch from mid-1980s through 2000 as low & falling commodity prices curbed costs, inflation rates and encouraged consumers to spend more (scottgranni) and permitted so the economic boom years at the end of the last century. Welcome to a new epoch of cheap commodities!