Going on vacation is a great way for workers to decompress from their busy schedules, but some don’t feel comfortable taking even a few days off, according to a new report from FlexJobs.

Though the majority have access to PTO, the report found that 23% of employees didn’t take a single vacation day in the past year, based on a survey of over 3,000 U.S. workers.

There are a few reasons why workers may be reluctant to take time off, according to FlexJobs career expert Keith Spencer.

Workload was a major concern for many employees: 43% felt that they had too much work to justify taking vacation, while 30% worried about falling behind, according to the survey.

“I think a lot of teams are being asked to do more with less, so there might just be kind of a lack of coverage if your team is understaffed,” Spencer says. “People might be feeling sort of guilty about leaving their team with a burden while they’re gone.”