Industrialization undercut the previous economic base share, leading to high differential growth, resulting in a rapid rise in novel Western-inspired formal systems, with the more traditional established economy that dominated before the Meiji Era undermined. So, on paper, there is economic growth, but the average guy lived in highly uncertain times due to the radical changes, politically, economically, and socially.
I imagine it sent massive economic shocks toward the average citizen despite the pragmatic high productivity. It's the same with GDP. A rise in GDP doesn't necessarily mean an objective rise in wages in all forms of distributions of an economy. TFR decline could reflect those disruptive shocks that overall are viewed as economic positives in hindsight. At the time, I imagine the distributive systems did not even out the productive goods enough.
It's like if AI replaces all industry workers, making much higher productivity with lesser costs, which translates into higher growth, it does not mean the people who were laid off now suddenly are enjoying those goods or feel comfortable knowing their way became industrially obsolete.
What you have, given that the system will re-arrange itself to siphon that "unproductive" base by potentially making them a fitting human capital to enter effective economic places, increasing their human capital orientation toward those managements, is an economic lag for those people that might only enjoy the goods down the line potentially (potentially less for the AI problems, if those systems don't result in much higher tax burden on the automated productivity that can feed into a much more robust safety-system for everyone, but I digress).
One major thing as well. There were massive regional and politico-infrastructure arrangement changes that likely had very big non-alignments with previous orders of balances. Offsetting that too likely fed into the aforementioned.
TFR reflects those pressures on the average citizen.