Based on comments on the 1881 census that I was previously unaware of, and a query to a local Yeovil historian, I have almost certainly confirmed that John Harris lived at the end of Sherborne Road near Yeovil Pen Mill station.
I was probably him. Nearly every significant place I remembered has matched real locations and this only drives it home for me.
I had been wrong about living on the north side. The historian I spoke to said that based on the way the districts were drawn and the comments on the census, it was probably the house next to the B&B (in fact, located right next door to it). However, that does not preclude my memories of having seen the train from there; in fact I either could have seen it from John’s window in the other house, or I could have been visiting a friend across the street.
This confirms that I knew the neighborhood and that it matches my memories, a confirmation I have been waiting almost two years for.
I was probably him… holy shit. I was probably him.