Dear Mr. DeVoe,
I was asked by the site manager of the John Jay Homestead, Bruce Fullem, to respond to your letter. Since I'm there only once a week, I apologize for the delay in answering your letter. I am a volunteer researcher/docent at the Homestead and can answer some of your questions.
The John Jay mentioned in the 1870 Census form is John Jay II (1817-1894), the grandson of John Jay (1745-1829), the Peace Commissioner for the Peace Treaty of 1783, the 2nd secretary for foreign affairs, the 1st chief justice of the US Supreme Court, the author of the trade treaty of 1794 (known as the Jay Treaty) and the 2nd governor of NYS. John Jay II was a lawyer, a prominent abolitionist and served as US minister to Austria after the Civil War. His main residence was in NYCity but we know he spent much time at Bedford (as it was known then.) He was married (1837) to Eleanor Kingsland Field (1819-1909); they had 5 children who survived to adulthood. The youngest was Anna (1849-1925) who, in 1872, married Hans Lothar von Schweinitz. I believe John Jay II was in Europe in July 1870 but certainly his son William II (1841-1915) would have been there as well as members of the extended family. The property was a working farm and, although JJII may not have been physically present in 1870, the Jays really did live there a great deal of the time.
Their property holdings were around 900 acres and "Cantitoe Corners" is just up the street from the present, much diminished, property. So to say that the Walpoles were neighbors would not be inaccurate.