Built around 1910 and opened on January 30, 1911, the Tudorbethan architecture of the station's original has earned it a listing on the National Register of Historic Places as an intact example of an early commuter rail station.<ref name="NRHP nom">Template:Cite web</ref> It is the only station on the Hudson Line besides Poughkeepsie to be so recognized. The plot of land on which it was built had once been part of the vast Philipsburg Manor, which was confiscated at the end of the American Revolutionary War and auctioned off in parcels. That particular parcel went through several owners, from Rodrigo Curls, a Portuguese immigrant who used it for fishing, to Ambrose Kingsland, a wealthy merchant and politician who had a large country estate along this part of the Hudson River.
The construction of the Hudson River Railroad and its later acquisition by the New York Central in the late 19th century opened up the river towns in Westchester County for suburbanization. It became possible for those of sufficient means to live in large houses amid the pastoral and scenic riverside, and accordingly villages like Irvington, Tarrytown and North Tarrytown (today's Sleepy Hollow) began to grow and develop.<ref name="NRHP nom" />
Undeveloped areas along the railroad line were soon snapped up by developers who saw the possibilities. In 1899 one, John Brisben Walker, acquired the old Ambrose Kingsland estate in North Tarrytown and began subdividing its northern part while building an automobile plant, the Mobile Company of America, in the southern section. One of his selling points for the residential development, called Philipse Manor in a confused reference to nearby Philipsburg Manor House,<ref name="Philipse/Philipsburg Manor" group=note>The actual Philipse Manor is in Yonkers, some Template:Convert to the south (both manor houses served the original Philipsburg Manor).</ref> was the rail access, but this failed to materialize. The Mobile Company of America, which produced steam-powered carriages, also failed, and Walker had to sell the property to William Abraham Bell, who had invested in Walker’s automobile venture.<ref>Template:Cite web</ref> The new owner continued the residential construction at Philipse Manor, and the subdivided land was sold under the name Philipse Manor Company.<ref>Template:Cite news </ref> Bell, with his extensive experience in railroad development, made the rail service possible by building the station and presenting it to the railroad.<ref name="NRHP nom" /> Train service began on January 30, 1911,<ref name="1911open" /> and wealthy New Yorkers started eagerly buying homes in Philipse Manor.
The station remained in use throughout the private ownership of the railroad. When the Metropolitan Transportation Authority assumed passenger commuter operations of the then-bankruptPenn Central in the early 1970s and passed it along to Metro-North in 1983, it eventually closed the station house in favor of automated ticketing operations, and the main house fell into disrepair. The station has since been reused as the Hudson Valley Writers Center, which won an award from the Preservation League of New York State for its work on the station in 2005.<ref name="HVWC award">Template:Cite press release</ref>
Template:As of, the MTA has been working to extend the platforms to accommodate eight-car trains and improve service and capacity. It is part of a $56 million program focused on all the Rivertowns stations. The agency expects it will be complete by 2010.<ref name="platform lengthening">Template:Cite newsTemplate:Dead link</ref>Template:Update inline
The main building (no longer used for rail purposes) is a one-story hip-roofedoctagonal structure of rock-faced granite block with stone, stucco and wood trim. It is built into the bluff created when the tracks were cut, and thus access to them was provided through the basement, through doors which have since been bricked off.<ref name="NRHP nom" />
The station's east facade is augmented with two gabledporte-cocheres projecting at oblique angles, each supported by a heavy granite pier. Trapezoidal wings also jut from the narrow sides of the octagon. The loggia across the facade has central round arched opening with a parapet. This does not lead to an entrance, instead backing the fireplace and its corbeled stone chimney. The original roof used slate, but it has been replaced with asphaltshingles.<ref name="NRHP nom" />
Inside, the fireplace uses several colors of granite, flanked with original iron radiators. It is complemented by dark oak matchboards over the stucco, laid to simulate paneling and form a dado. Further ornamentation includes a double frieze at ceiling level.<ref name="NRHP nom" />
The more modern station subsequently built by the Metropolitan Transportation Authority (MTA) consists of two long concrete, elevated side platforms with dark-green painted steel shelters. Between them are the four tracks of this section of the Hudson Line, all with third rails. The inside tracks carry express trains, and diesel-powered Amtrak and Metro-North trains bound for the non-electrified sections between Croton–Harmon and the northern end of the line at Poughkeepsie, none of which stop at Philipse Manor. An overpass connects the two platforms.
The huge cast-iron eagle facing the southbound platform of the station is one of 11<ref>Template:Cite web</ref> that graced the monumental clock towers of New York City's Grand Central Station for 12 years between its 1898 renovation and 1910, when the eagles were removed in the process of the demolition of the station and construction of the current Grand Central Terminal building. The eagles were auctioned off to institutions and private estates throughout the metropolitan region. One of them was obtained by William Bell's Philipse Manor Company and installed at Philipse Manor station before its opening in 1911.
The Template:Convert eagle has a 14-foot wingspan. In 2019, the eagle was restored to the appearance matching the two other original eagles<ref>Template:Cite news</ref> that had been recovered and installed above Grand Central Terminal's entrances on Lexington Avenue.<ref>Template:Cite web</ref> <ref>Template:Cite book</ref>
