Would it be okay to get some guinea pigs even though I already have ferrets?

Okay, hear me out. I have 2 neutered male ferrets. I've been wanting to get 2 maybe 3 guinea pigs for a while now. They would live in SEPARATE cages. I know ferrets are carnivores. I like to let my pets roam around out of their cages a few hours of the day, but they WOULD NOT be let out at the same time. So they would never be in direct contact. BUT their cages would be right next to each other, so they will be able to smell & see each other. So my question is if just having ferrets around would freak out the guinea pigs.