Combine the sugars, water, ginger slices, cinnamon, allspice berries, and peppercorns in a small saucepan set over medium heat. Bring to a gentle simmer, stirring until the sugar is completely dissolved. Reduce heat and let simmer for 15 minutes.

Remove from heat and let steep for 1 hour. Remove the cinnamon stick and ginger slices, then filter the syrup through a coffee filter to remove the remaining spices. (At this point, the gingerbread syrup can be transferred to a small bottle and kept in the refrigerator for several weeks.)

Add the milk to a saucepan and warm over low heat – do not allow the milk to boil. Meanwhile, whip the cream and sugar until it forms soft peaks.

Combine the coffee, warmed milk, 4 Tablespoons of gingerbread syrup, and bourbon (if using) and divide between four glasses. Add sugar to taste before topping each serving with whipped cream.

Serve immediately.