Pike and shot
He started by re-arranging the formations to be thinner, typically only four to six ranks deep, spreading them out horizontally into rectangles instead of squares. This further maximized the number of musketeers near the front of the formation. Additionally he introduced the practice of volley fire, where all of the gunners in the ranks would fire at the same time. This was intended to bring down as many members of the opposing force's front line as possible, causing ranks moving up behind them to trip and fall as they were forced forward by the ranks further back. Finally, he embedded four small "infantry guns" into each battalion, allowing them to move about independently and not suffer from a lack of cannon fire if they became detached.
Gustav also placed detached musketeers in small units among the cavalry. In traditional deployments the infantry would be deployed in the middle with cavalry on both sides, protecting the flanks. Battles would often open with the cavalry attacking their counterparts in an effort to drive them off, thereby opening the infantry to a cavalry charge from the side. An attempt to do this against his new formations would be met with volley fire, perhaps not dangerous on its own, but giving the Swedish cavalry a real advantage before the two forces met. Under normal conditions detached musketeers without pikemen would be easy targets for the enemy cavalry, but if they did close to sabre range, the Swedish cavalry would be a more immediate concern.
The effect of these changes was profound. Gustav had been largely ignored by most of Europe after his mixed results in Poland, and when he arrived in Germany in 1630 he was not immediately challenged. He managed to build up a force of 24,000 regulars and was joined by a force of 18,000 Saxons of questionable quality under von Arnim. Battle was first joined in major form when Johann Tserclaes, Count of Tilly turned his undefeated 31,000-man veteran army to do battle, meeting Gustav at the Battle of Breitenfeld in 1631. Battle opened in traditional fashion, with Tilly's cavalry moving forward to attack the flanks. This drove off the Saxons on the one flank, but on the other Gustav's new combined cavalry/musket force drove off any attempt to charge. With one flank now open Tilly nevertheless had a major positional advantage, but Gustav's smaller and lighter units were able to easily re-align to face the formerly open flank, their light guns cutting into their ranks while the heavier guns on both sides continued to exchange fire elsewhere. Tilly was soon driven from the field, his forces in disarray. 
Follow-up battles had similar outcomes, and Tilly was eventually mortally wounded during one of these. By the end of 1632 Gustav controlled much of Germany. His successes were short-lived however, as the opposing Imperial forces quickly adopted similar tactics. From this point on pike and shot formations gradually spread out into ever-wider rectangles to maximize firepower of the muskets. Formations became more flexible, with more firepower and independence of action.

Outside of Europe: Meanwhile in East Asia, the utility of pike and shot style formations were still being tested. The Japanese army in the Imjin War supported their gunmen (25–30% of their initial force) with spear and bow levies, but the pike was not as emphasized as it was in contemporary Europe due to the lack of a large cavalry threat in either Japan or Korea. At the 1619 Battle of Sarhū, the Koreans (drawing on lessons from 1592 to 1598) deployed an all-shot formation (10,000 arquebusiers and 3,000 archers) using volley fire against the cavalry-heavy Manchus. The arquebusiers inflicted many losses on the Manchus, but were routed. This prompted a revision of military tactics in Korea. After the defeat at Sarhū, the Joseon forces revised their doctrine to have spearmen supporting the arquebusiers to better withstand shock cavalry. The new Korean force was tested against the Manchus again in 1627 and 1636–1637. Both times, they were defeated, but their performance left a strong impression on the Manchus. The first emperor of the newly declared Qing dynasty later wrote: "The Koreans are incapable on horseback but do not transgress the principles of the military arts. They excel at infantry fighting."

Decline: After the mid-seventeenth century, armies that standardized the adoption of the flintlock musket began to abandon the pike altogether (flintlocks and proto-flintlocks, such as the miquelet lock, had been in use since the mid-16th century, but remained less common than matchlocks until the late 17th), or to greatly decrease their numbers. A bayonet could be affixed to the musket, turning it into a spear, and the musket's firepower was now so deadly that combat was often decided by shooting alone. Moreover, the flintlock could be loaded and fired approximately twice as fast as the matchlock, and misfired far less. The abandonment of the pike, together with the faster firing rate made possible by the standardization of the flintlock musket and paper cartridge, resulted in the abandonment of the deeper formations of troops more ideal for the melee-oriented pikemen. Military thinking switched to shallower lines that maximized the firepower of an infantry formation. By one calculation, a formation equipped entirely with mid-18th century flintlocks could output ten times as many shots in an equivalent period of time as a typical early 17th century pike and shot formation equipped with matchlocks (pike:shot ratio of 3:2), enormously changing the tactical calculus of the infantry's armament. From 1688 to 1696, 3 out of every 5 Austrian and British soldiers had a flintlock musket, the rest mostly had matchlocks.
After the Battle of Rocroi fought on 19 May 1643, the Spanish abandoned the Tercio system and adopted the line infantry doctrine used by the French. A common end date for the use of the pike in infantry formations is 1700, although some armies such as the Prussians had already abandoned the pike decades prior, whereas others such as the Swedish and Russians continued to use it for several decades afterward—the Swedes of King Charles XII in particular were using it to great effect until 1721. The Army of the Holy Roman Empire officially stopped using pikemen in 1699. It had been progressively phasing out both pikes and lances since the Thirty Years War. Following Montecuccoli's reforms in the 1660s, the paper strength of an Imperial infantry company was 48 pikemen, 88 musketeers, and 8 shieldmen. The Imperial Army used this configuration during the Great Turkish War, most famously at the Battle of Vienna, where flintlock muskets were outnumbered by matchlocks (the flintlock would not enter regular service until 1699). During the same conflict, the Polish-Lithuanian army also made extensive use of pikes. In 1703, the French army also discontinued the use of the pike, followed in 1704 by the British and in 1708 by the Dutch. Between 1699 and 1721, Peter I converted almost all Russian foot-regiments to line infantry. 
Even later, the obsolete pike would still find a use in such countries as Ireland, Russia, and China, generally in the hands of desperate peasant rebels who did not have access to firearms. One attempt to resurrect the pike as a primary infantry weapon occurred during the American Civil War when the Confederate States of America planned to recruit twenty regiments of pikemen in 1862. In April 1862 it was authorized that every Confederate infantry regiment would include two companies of pikemen, a plan supported by Robert E. Lee. Many pikes were produced but were never used in battle and the plan to include pikemen in the army was abandoned.

External links: The Pike & Shot Society, a society devoted to study of the period

References: 
Arfaioli, Maurizio. The Black Bands of Giovanni: Infantry and Diplomacy During the Italian Wars (1526–1528).  Pisa: Pisa University Press, Edizioni Plus, 2005. ISBN 88-8492-231-3.
Baumgartner, Frederic J.  The French Reluctance to Adopt Firearms Technology in the Early Modern Period, in The Heirs of Archimedes: Science and the Art of War Through the Age of Enlightenment, eds Brett D. Steele and Tamera Dorland.  Cambridge, Massachusetts: MIT Press, 2005.
Baumgartner, Frederic J. France in the Sixteenth Century. New York, New York: St. Martin's Press, 1995.
Oman, Charles. A History of the Art of War in the Sixteenth Century.  London: Methuen & Co., 1937.
Jorgensen, Christer (et al.).  Fighting Techniques of the Early Modern World: Equipment, Combat Skills, and Tactics.  New York, New York: Thomas Dunne Books, 2006.
Schürger, André. The Battle of Lützen: an examination of 17th century military material culture. University of Glasgow, 2015 [1]
Taylor, Frederick Lewis. The Art of War in Italy, 1494–1529.