What is a Transformer?

The Transformer model was proposed by Ashish Vaswani and seven other authors from Google Brain in 2017. They then published a paper titledAttention Is All You Need, which first introduced the Transformer model.

Undoubtedly, the Transformer model is one of the most fundamental baselines in the current field of artificial intelligence. It is like a crucial 'piece of the puzzle' in the development of AI, which has revolutionized AI in just a few years.

Figure 1: The Transformer - model architecture.

During the AI Day in 2021,Teslashowed its Transformer-based BEV (Bird's Eye View) technology. This advanced system enables Teslato transform 2D images from eight onboard cameras into a comprehensive 3D vector space, providing a more accurate understanding of the vehicle's surroundings. This breakthrough represents a significant milestone, as it marks Tesla'sadoption of the Transformer modelin the field of autonomous driving technology.

TheNVIDIA DRIVE Orinchip was officially launched by NVIDIAon December 18, 2019, marking a significant step forward in autonomous driving technology. The Orin chiphas seen substantial progress in the Chinese market, with its powerful 254 TOPScomputing capabilities enabling advanced autonomous driving features like urban navigation and highway driving. It has been widely adopted by leading Chinese automakers such as NIO, XPENG,and Li Auto, as well as by Baidu's Apollo Gofor its Robotaxi fleets, driving large-scale deveployment ofautonomous driving technologyin the electric vehicle (EV)sector in China.

Initially,NVIDIAplanned to introduce the Atlanchip, which was first revealed at the 2021 GTC conferencewith an intended 1000 TOPSof computing power, positioned as the successor to Orin. However, in the 2022 fall GTC conference,NVIDIAannounced the cancellation of the Atlanchip, shifting focus to the more advanced Thorchip.

The transition from Atlanto Thorwas driven by the superior performance of Thorin supporting Transformer models. The Drive Thor platformincorporates a new Transformer inference engineas part of its Tensor Cores, which dramatically enhances the speed and efficiency of deep neural network processing, particularly for Transformer-based models. This enables Thorto deliver up to a ninefold increase in inference performance, making it highly effective for managing the complex AI workloads required in modern autonomous driving systems.

In 2018, OpenAIintroduced the first version of Transformer-based GPT model, marking a pivotal moment in the Natural Language Processing (NLP)field. Since then, OpenAIhas continually refined this architecture, leading to breakthroughs that have revolutionized NLP. The Transformer's self-attention mechanism excels at processing long sequences of data, allowing models to grasp context and generate coherent text, making it highly effective in natural language generation.

With pre-training on large-scale datasets, OpenAIdeveloped advanced models like GPT-3and GPT-4, featuring billions of parameters and enhanced reasoning abilities. These models can handle not only question answeringand conversational tasks, but also complex applications such as code generation,data analysis, and tool control. This evolution has transformed Large Language Models (LLMs)from traditional NLP tools into versatile intelligent assistants, driving real-world applications across various industries.

TheLLM seriesby OpenAIhas become a core force in advancing generative AIby scaling model size and driving architectural innovation. Through the power of Transformers,OpenAIhas expanded LLMsfrom simple language understanding systems into robust general AI platforms, providing a foundation for diverse intelligent applications. The GPT serieshas elevated Transformer technologyto new heights, with GPT-4as the latest milestone. It not only achieves human-like text generation but also supports multimodal inputsof images and text, showcasing the Transformer model'sfurther expansion into the multimodal AIspace.

Figure 2：The principle of electromagnetic induction.

Now let's learn something about power transformers.

Apower transformeris an electrical energy converter. Based on Faraday's law of electromagnetic induction, it transforms (either step up or step down) AC voltage while maintaining the same power frequency.

The core structure of a transformerconsists of two or more sets of coils—primary coiland secondary coil—along with a magnetic core. Its main functions include adjusting AC voltage,changing impedance, and providing circuit isolation.

Transformers that alter AC voltage levelsare known as step-upor step-down transformers, designed to increase or decrease voltage, respectively. Additionally, transformers can offer current isolationbetween circuits and serve as coupling stages in signal processing circuits. Since the development of the first constant potential transformerin 1885, transformers have become essential components of AC power transmissionand distribution systems. The design of transformers varies greatly, from RF transformerssmaller than a cubic centimeter to large-scale transformers used ininterconnected power grids, weighing hundreds of tons.

Figure 3：Three-phase 33kV Oil-immersed Transformer with OLTC from Aputon

The fundamental principle of an electric power transformeris based on Faraday's law of electromagnetic induction. A simple single-phase transformerconsists of two conductive bodies. When a variable current, such asalternating current (AC)or pulsed direct current (DC), flows through one of these conductors, it creates a changing magnetic field. According to mutual inductance, this varying magnetic field induces a potential differencein the second conductor. If this conductor is part of a closed circuit, it generates a current, thus allowing the transfer of electrical power.

In practical transformers, these conductors are typically coilsmade of conductive wire, most often copper, as a coil generates a much stronger magnetic field than a straight wire. The transformer's operation relies on applying a changingvoltageto the primary coil, which generates a fluctuating magnetic field within the magnetic core, thereby inducing a varying electromotive force (EMF)in the secondary coil.

The relationship between currentand voltageon either side of a transformer is determined by the turns ratioof the coils. The side with a higher number of turns produces a higher voltagebut lower current, while the side with fewer turns produces a lower voltage but higher current. Disregarding factors like leakage, the voltage ratio between the primary and secondary sides is directly proportional to the turns ratio, meaning that voltageis proportional to the number of turns.