Nesting Orioles

We've had a big pile of brush in our fire circle for weeks now, since we cut down all the old flower stalks earlier this spring. In the last couple of weeks it's been attracting all kinds of birds -- robins, starlings, catbirds...but we never dreamed it would attract a pair of Baltimore orioles! They've come back repeatedly over the last couple of days. We have yet to figure out where they're building the nest -- they're definitely flying out of our yard with the materials. One is the male and the other the female. Which one is which?