There are probably several factors involved, but there are many fish that are larger than most marine mammals. For example the whale shark and the basking shark are larger than all of the pinnipeds and most of the toothed whales.
Fish are the most ancient group of vertebrates on the planet, first appearing over half a billion years ago. Whales are only about 50 million years old and pinnipeds are only around 30 million years old. Fish have had a lot more time to diversify and fill almost every niche available in the sea. These adaptations include miniaturization and gigantism. Pinnipeds are so evolutionarily young that they still split their time between the land and sea. Their upper size is limited to what their bodies can support while on land.
The minimum size of marine mammals probably has to do with both the need to conserve heat and the distance that they are able to move. Most scuba divers learn that water conducts heat 20 -25 times faster than air. That is why most divers hear some kind of wet suit, even in the tropics. Hypothermia in humans can start when your core body temperature drops below 95 F. You can get hypothermia in 80 F water.
The square-cube law states “When an object undergoes a proportional increase in size, its new surface area is proportional to the square of the multiplier and its new volume is proportional to the cube of the multiplier.” So volume increases much faster than surface area. Volume in important in generating heat while surface area is important in retaining heat. Larger animals have an easier time staying warm in water and a harder time staying cool on land. This helps explain the limit in size of terrestrial animals as well.
Movement distance is also a factor I believe to be important, although I don’t know if anyone else has thought of this. All marine mammals breath air and have to come to the surface to breathe. Both lung capacity and movement rate in terms of body length increase with increasing body size. There is a semi-aquatic shrew that is able to stay underwater for about 15 seconds, It can probably get several feet under water in this time, and when it stops swimming, air trapped in its fur makes it shoot to the surface. The sperm whale, on the other hand, can dive to depths of over 3000 feet and it can hold its breath for an hour and a half. The blue whale is not a deep diver but can still hold its breath for over half an hour. Dolphins can only hold their breath for 15 to 20 minutes, which is comparable to the human record of 19 minutes. A cold-blooded animal like a sea turtle can hold its breath for 5 hours. Animal speeds are often correlated with body length so a larger animal can travel farther in a given time. So a large whale can dive for a long time and travel a great distance. A miniaturized whale only a foot long would not be able to hold its breath for long and would need to remain near the surface. The largest whales have no natural predators, while a foot long whale would be prey for even many fish.
The evolution of Ichthyosaurs might provide an interesting model to compare to whales. Ichthyosaurs were a marine reptile that lived at the same time as the dinosaurs. Some species grew to giant size, up to 70 feet for the largest species while the smallest species were just over a foot long. They were reptiles, however, but we don’t know if they were strictly cold-blooded. Many pelagic fish, such as tuna, have body temperatures above that of the surrounding water, and are able to regulate body temperature. Tuna are also not dependent on surface air and are able to stay submerged indefinitely. Ichthyyosaurs may have been somewhere between whales and tuna, able to stay submerged for longer times and thus allowing for a smaller body size than whales are able to obtain.