Sub-element is super important to get max stats and I recommend using it on your UR orbs.
Yes, you need to luck out, but this is even better reason to use them, as you will aim to get good speed orbs with sub-element you want for multiple orb events (they return monthly).
Sub element is less of a problem for characters whose good dresses are in their main element, like Ao, Aka or Akisa. But if you want to consistently win speed, you need some light-sub speed orbs for your Eliza. Not to mention the effing Iroha, whose every good dress is in different color.
Consider this - if you match element but mismatch character, or match character but mismatch element, it is 5% stats less from that dress. This is around 8 raw speed less on your dress.
Maybe for PvE content, 8 speed is not that much of a difference. But for Sabbath 8 speed is like 2 leagues of difference.