You have quite logically answered most of your won questions. Quite true, conglomerate can look like weathered concrete, which is really a man-made form of conglomerate. However, concrete is made from cheap, plentiful material that is quarried on Southern Ontario, such as limestone, sand, lime and I doubt that there will be much in the way of igneous, or metamorphic rock found within concrete.
Your sample has either a piece of diorite or syenite, hence it is conglomerate with a sand matrix. I much finer version of sedimentary rock with tiny sharp-edges pieces within a much finer-grained matrix, like silt is called breccia. Since your sample slightly fluoresces under UV light the light & dark speckled rock is probably syenite, instead of diorite, because the former contains fluorescent sodalite. Some of the red-pink colour comes from pseudo-fluorescence coming from potassium feldspar (k-spar), also called orthoclase. The rounded stone with all the poorly fluorescent red is caused by feldpsar. The dark minerals within it are probably hornblende, maybe some biotite mica. It is, essentially, red granite with a lot of k-spar.
