Correct: both of those apply "Defense Down", which is just one debuff.
Matching opposites is quite effective though.
For example, Offense Up is +25%, and Defense Down is -20%. If you have a OffUp target attacking a DefDown target, the end result is (1.0 x 1.25) / 0.8, or 1.56, in easier numbers, a 56% improvement over the result you'd get without either.
Also, passives that claim to give various bonuses to defense/offense/etc. DO stack with buffs/debuffs, since these apply directly rather than by sticking Defense Down or other actual status effects on the target.