Why Retiring Without a Plan for Your Time Is More Dangerous Than Retiring Without Enough Money

David Brooks, host of the Retire SMART Podcast and president of Retire SMART LLC, spent much of Episode 446, “The Cost of Divorce in the Retirement Years,” on a warning that gets far less airtime than portfolio math. Couples obsess over the number they need to retire and skip the harder question of what they will actually do with the roughly 2,000 additional waking hours a year that stop being filled by a job.

Brooks argued that an empty calendar, more than an underfunded 401(k), is what pushes late-career marriages into crisis. You can hear the full episode on the show’s Apple Podcasts page.

The Quote That Sets the Frame

Speaking from his own household, Brooks said, “If I were not working full-time, it would create some stress in my household if we didn’t plan through what we’re gonna do.”

He then walked through a client couple with opposite instincts: she loves to travel, he does not. Their negotiated answer was concrete. She takes three vacations a year without him; he joins her on one, and he spends his time and budget on tools and restoring fancy cars. Brooks’ verdict: “They have agreed upon where their joy comes from, and as long as you’ve mapped it out, then it’s fine. But if you go into retirement and haven’t mapped that out, that could create some stress.”

Why the Timing of This Message Matters

Brooks is delivering this message into a genuinely anxious market. The University of Michigan Consumer Sentiment Index came in at 44.8 in May 2026, down from 49.8 in April and 61.7 in July 2025.

Behavior mirrors that mood.

The Bureau of Economic Analysis shows the personal saving rate falling from 6.2% in the first quarter of 2024 to 3.9% in the first quarter of 2026, even as per capita disposable personal income climbed to $68,391. Americans are earning more but saving less, leaving many households with less financial cushion despite higher incomes.

At the same time, the 2026 Social Security cost-of-living adjustment was just 2.8%, while average annual household expenditures reached $78,535 in 2024, according to the Consumer Expenditure Survey. In that environment, financial misalignment within a marriage can become a source of strain long before any broader economic shock arrives.

Design the Week Before You Design the Withdrawal

Brooks says “planning is paramount here” and urges couples to interview themselves about time the way they interview an advisor about money. Full-time volunteering, a part-time role, or a specific passion project all qualify. His late father-in-law had a fully formed version of this: “I just wanna be a starter at a golf course. I’ll work 20 hours a week as the starter, and I’ll get my greens fees for free.” That is 20 hours of structure, identity, and a subsidized hobby, negotiated in a single sentence.

A caller during the segment offered the sharpest framing. Retirement strips away the two routines that organize adult life, working and raising children, so the operative question becomes “what’s the new center of gravity?” Brooks’ instruction is to map where each partner finds joy, get explicit agreement, and build the retirement schedule around those answers before the last paycheck lands.

What Investors Should Take From This

The financial case for pre-retirement planning is already well documented on our site.

Brooks is adding a second axis. A spreadsheet that survives a bear market can still fail if two people wake up on a Tuesday with no reason to leave the house and no agreement on how to fill it. Readers approaching the transition should treat the “time budget” as a required document alongside the withdrawal plan: which hobbies get funded, which trips happen solo, which part-time role provides structure, and what each spouse considers non-negotiable joy. Do that work while both incomes are still coming in, and the eventual portfolio conversation gets much easier.

Contact [email protected] for any questions or corrections.