Faivish Zharkoy was born in 1921 in Pskov Oblast in a Jewish family. His father Mendel was a blacksmith.
In the late 1920s, after Mendel's death, the family moved to Leningrad, where Filipp (his Jewish name Faivish had been Russified by that time, although it remained unchanged in his ID documents) attended school.
Filipp's older brother Zalman was serving in Kazan as a second-rank military technician at advanced courses for Red Army personnel. In 1936, Filipp joined his brother in Kazan, where he finished school. Zalman Zharkoy was later transferred to Saratov, where he taught at an armor school. In 1939, Filipp moved to Saratov in his brother's footsteps and began to study at that armor school.
Zharkoy completed the school in late May 1941, a month before the outbreak of the Soviet-German War. He was sent to serve in the 32nd Armored Division in Lvov. He had reached his destination by the time the war broke out, and his frontline service began then and there. In the early period of the war, the Red Army sustained heavy losses. As a result, the 32nd Armored Division, where Zharkoy was serving, was disbanded.
Afterward, he served on the Southern Front as the commander of an armored platoon. In September 1942, Filipp, as a battalion commander in an armored brigade, took part in an ill-fated advance of the Stalingrad Front near the settlement of Kotluban (in present-day Volgograd Oblast). This operation led to many casualties and brought little gain to the Red Army.
In the summer of 1943, Zharkoy commanded an armored regiment that fought on the North Caucasus Front. Later, in February 1944, Filipp, who had by then joined the Leningrad Front, took part in lifting the Siege of Leningrad.
In the summer of 1944, his armored regiment fought in the territory of Estonia, and later Latvia, as part of the Baltic Front. He went on to take part in the Vistula-Oder offensive, the East Pomeranian offensive, and the Battle of Berlin – where he actually participated in the storming of the Reichstag. He was seriously wounded twice in the course of the war, but returned to frontline duty on both occasions, following a period of convalescence.
For his wartime service, Zharkoy was awarded three Orders of the Red Banner; 2 Orders of the Patriotic War, 2nd class; the Order of the Red Star, and several medals.
His brother Zalman Zharkoy was killed in action in January 1945, in a battle near Königsberg.
After the end of the war, Filipp Zharkoy continued his military career. In the second half of the 1930s, he commanded an armored battalion in the Primorsky Military District. In 1950-1955, he attended the Armored Forces Academy in Moscow, and then went back to commanding armored regiments in various military districts of the USSR.
From the late 1960s to the early 1970s, Zharkoy was the executive officer of one of the Soviet divisions stationed in Germany.
In 1972, after finishing his military career in the rank of colonel, Filipp Zharkoy returned to Leningrad, where he taught at the Central Military Courses at the Military Academy.
In 2010, Filipp Zharkoy published a book of memoirs titled Tank March, for which he received the literary award of the St. Petersburg Legislative Assembly. Together with his son, Filipp launched a website dedicated to the tank troops.
In June 2021, Filipp (Faivish) Zharkoy celebrated his 100th birthday.