Steve Goldstein of MarketWatch reports that the tech giants who built empires on weightless bits and bytes are now grappling with atoms. Goldstein reports:
Investors have greeted the news that Scott Bessent will be President-elect Donald Trump’s choice to be Treasury secretary with relief, hoping the hedge-fund manager will smooth out the rough edges of the incoming president’s policies.
But this year’s stellar stock-market returns have less to do with Trump and more with hopes for artificial intelligence. Lux Capital counts AI-focused companies including Together AI and Hugging Face in its portfolio, and in its latest letter to shareholders, the New York venture-capital firm discussed how power-hungry AI is.
“Today’s largest AI clusters use around 100,000 Nvidia H100 chips and consume roughly 100 megawatts of power. Next year’s clusters will scale to 300,000 to 500,000 chips requiring nearly a gigawatt of power — about the consumption of a small city,” the firm said. That’s imposing real physical constraints, as “the tech giants who built empires on weightless bits and bytes are now grappling with atoms — steel, copper, water rights, and critically, natural gas,” according to the firm.
While there have been some AI-fueled nuclear power deals, Lux said “abundant natural gas from the Texas Permian seems a wiser bet” to power AI-related demand.[…]
The firm talked about the need for companies to do more with less. Some of that is talking the firm’s book — Together AI makes software to run generative AI workloads more efficiently — but Lux also discussed Apple Inc. “Apple has quietly published research showing how to run large language models directly on devices with limited memory. By leveraging novel storage techniques and the inherent sparsity in AI models, the company can reduce memory requirements by 50% while maintaining performance. Advances like these could enable future iPhones and Macs to run sophisticated AI models locally and privately — a development that could radically reshape the entire AI infrastructure landscape,” the firm said. […]
It previously forecast that up to 50% of venture-capital firms would exit the industry, and it said the “inferno” is spreading faster than it imagined, with 25% of active investors from 2023 having already vanished, as 40% of general partners having either canceled or delayed raising money for a new fund.
Read more here.
Read Apple’s research paper, LLM in a flash: Efficient Large Language Model Inference with Limited Memory – July 30th 2024
