Using filler piece to bridge gap with wall unit using framed door

Don't know if this makes sense but: If using a filler piece between the side of a wall cabinet (with a framed door) and the wall, should the filler be lined up flush with the front of the cabinet - or flush with the frame of the cabinet door. I'm also fitting a lighting pelmet and wondered whether it looked odd to see the top side of the pelmet revealed where it runs under the filler piece if I line up the filler flush with the cabinet. Thank you for any responses.