It is generally believed the first white man to live in Independence Township was Daniel Weir who came in 1842. Pioneers Dawson, Howell, and Perin came in 1845, one year before Iowa became a state. The first land entered was in 1849. Settlers began to establish homes. In 1870 a school house was built and became the educational and religious center for the surrounding residents. Erastus O. Seeley was the first postmaster. Another influential citizen was Sidney Baxter Higgens. The name Independence Center was established in January 1870. The government would not accept that name as there was already an Independence in Iowa. Next they submitted the name Sidney, this too being rejected for the same reason. The third choice was Baxter and thus the name was changed in February 1872. There were several stores and homes built and the town grew.

As is common with many early settlements the coming of the railroad was the demise of this community. In July of 1883 the contract was let for "grading the diagonal." However, where the railroad line was planned was one and one half miles to the east and north. So with much labor and Vandike's twelve-yoke ox team, the town was moved. In a few weeks all the buildings except the school house were relocated. (The school house still stands in 2000, being used as a dwelling.) A more detailed account of early Baxter history can be found in "Recollections and Sketches of Northwest Jasper" a work written in 1970 for the Baxter Centennial Celebration, copies available at the Baxter Library. In 1883, when the town was moved, "the name Baxter went along too, but the ghost, still claiming ownership, continues as OLD BAXTER."