How to wear a white button down shirt without it looking too smart like office wear?
Wear a coloured vest top underneath the white shirt (maybe bright orange would go with the brown trousers) and leave the shirt unbuttoned a bit lower than you usually would so that the vest top can just be seen underneath it. Get a waist belt (brown) and wear over the top of the untucked shirt. Wear some big, bold, colourful jewellery, a big handbag to accessorise and, depending on the leg length of the trousers, either wedge sandals in brown, or flats to complete the casual look.
