Open heart surgery on the data center: Switchgear replacement in a live facility

數(shù)據(jù)中心的心臟手術(shù)：開關(guān)柜在線更換

By Mark Johns

U.S. Bank emerged during the 1990s from mergers and acquisitions among several major regional banks in the West and Midwest. Since then, the company continued to grow through additional large acquisitions and mergers with more than 50 banks. Today, U.S. Bancorp is a diversified American financial services holding company headquartered in Minneapolis, MN. It is the parent company of U.S. Bank National Association, which is the fifth largest bank in the United States by assets, and fourth largest by total branches. U.S. Bank’s branch network serves 25 midwestern and western states with 3,081 banking offices and 4,906 ATMs. U.S. Bancorp offers regional consumer and business banking and wealth management services, national wholesale and trust services and global payments services to over 15.8 million customers.

20世紀(jì)90年代，中部與中西部幾家大型地方銀行通過兼并，成立了美國合眾銀行。隨后，又接連并購了50多家銀行?，F(xiàn)在，美國合眾銀行成為了一家多元化金融服務(wù)類控股公司，總部位于美國明尼蘇達(dá)州。美國合眾銀行是美國國家銀行協(xié)會(huì)的母公司，是全美資產(chǎn)規(guī)模排名第五、分支機(jī)構(gòu)數(shù)排名第四的銀行。美國合眾銀行擁有3081個(gè)辦事處和4906臺(tái)自動(dòng)取款機(jī)，服務(wù)遍及中西部與西部25個(gè)州。美國合眾銀行為超過1580萬消費(fèi)者、企業(yè)提供經(jīng)紀(jì)、理財(cái)、投資、抵押、信托和全球支付服務(wù)。

Rich in history, US Bancorp operates under the second oldest continuous national charter—originally Charter #24—granted during Abraham Lincoln’s administration in 1863. In addition, U.S. Bank helped finance Charles Lindbergh’s historic flight across the Atlantic. For sheer volume, U.S. Bank is the fifth-largest check processor in the nation, handling 4 billion paper checks annually at 12 processing sites. The bank’s air and ground courier fleet moves 15 million checks each day.

美國合眾銀行歷史悠久，擁有全美第二古老的營業(yè)執(zhí)照，是林肯政府于1863年頒發(fā)的。此外，還資助了查爾斯·林德伯格飛越大西洋這一歷史壯舉。美國合眾銀行是全美第五大支票商，12家網(wǎng)點(diǎn)每年要處理40億張支票業(yè)務(wù)，每天通過空運(yùn)與陸運(yùn)的支票數(shù)達(dá)到1500萬張。

Energy Park Site

Energy Park概況

U.S. Bank relies on its Energy Park site in St. Paul, MN, to support these operations. Energy Park comprises a 350,000-square-foot (ft²) multi-use building that houses the check production operations and 40,000-ft² data center, as well as support staff for both. Xcel Energy provides two 2,500-kilovolt-ampere (kVA) feeds to the data center and two 2,000-kVA feeds to the rest of the building.

美國合眾銀行依靠明尼蘇達(dá)州圣保羅的Energy Park數(shù)據(jù)中心維持業(yè)務(wù)運(yùn)營。園區(qū)有一座35萬平方英尺的多功能大樓，運(yùn)營其支票業(yè)務(wù)和4萬平方英尺的數(shù)據(jù)中心，并配備了后勤人員。Xcel Energy公司為數(shù)據(jù)中心提供雙路2500kVA電力供應(yīng)，為大樓其他業(yè)務(wù)提供雙路2000kVA供電。

The utility’s data center feeds supply power to two automatic throw over switches (ATO); each ATO feeds two transformers. Two transformers support the data center, and two other transformers support check production and power for the rest of the building, including offices and HVAC (see Figures 1-3).

通過將市電分配至2個(gè)自動(dòng)雙投開關(guān)ATO，ATO再分別供給2臺(tái)變壓器。其中2臺(tái)變壓器給數(shù)據(jù)中心進(jìn)行供電，另外2臺(tái)變壓器給支票業(yè)務(wù)及大樓其他部分提供電力，包括辦公與空調(diào)用電。見圖1～3。

Figure 1. Temporary stand-alone power plant

圖1 臨時(shí)發(fā)電站

Figures 2 and 3. Utility transfer and ATS

圖2和圖3 變壓器與ATS

A single UPS module feeds the check production area. However, two separate multi-module, parallel redundant UPS systems feed data center loads. Four N 1 1,500-kilowatt (kW) standby-rated engine generators backup the three UPS systems through existing switchgear distribution. The data center switchgear is a paralleling/closed-transition type, and the check production area switchgear is an open-transition type. The remaining office area space is not backed up by engine generators.

支票業(yè)務(wù)區(qū)由一臺(tái)單模塊UPS進(jìn)行供電，2臺(tái)獨(dú)立多模塊UPS并列運(yùn)行給數(shù)據(jù)中心進(jìn)行供電。4臺(tái)1500kW（N 1）冗余發(fā)電機(jī)組通過開關(guān)柜給3臺(tái)UPS提供后備保障。數(shù)據(jù)中心開關(guān)柜采用短接型（先合后斷）切換開關(guān)，支票業(yè)務(wù)區(qū)開關(guān)柜采用非短接型（先斷后合）切換開關(guān)。辦公區(qū)沒有配置發(fā)電機(jī)。

Project Summary

項(xiàng)目概述

To ensure data center reliability, U.S. Bank initiated an Electric Modernization Project (data center electrical distribution). The project included replacing outdated switchgear and UPS systems, which were no longer supported by the manufacturer. In the project’s first phase, Russelectric paralleling switchboards were selected to replace existing equipment and create two separate distribution systems, each backed up by existing engine generators. Mechanical and UPS loads are divided between the two systems, so that either one can support the data center. Switchgear tie breakers increase overall redundancy. The facility benefits from new generator controls and new switchgear SCADA functionality, which will monitor and control utility or generator power.

為了保證數(shù)據(jù)中心可靠性，美國合眾銀行對(duì)數(shù)據(jù)中心進(jìn)行了現(xiàn)代化配電改造，包括更換制造商不再維保的開關(guān)柜和UPS系統(tǒng)。首先，用羅斯電氣公司生產(chǎn)的重疊轉(zhuǎn)換開關(guān)柜替換現(xiàn)有設(shè)備，并建立兩套獨(dú)立配電系統(tǒng)，現(xiàn)有發(fā)電機(jī)作為備用電源提供保障。兩套配電系統(tǒng)為動(dòng)力負(fù)載和UPS負(fù)載供電，任一配電系統(tǒng)都可支撐數(shù)據(jù)中心運(yùn)行。通過開關(guān)柜內(nèi)聯(lián)絡(luò)開關(guān)，提高了供電的整體可靠性。新的發(fā)電機(jī)控制系統(tǒng)和新的開關(guān)柜SCADA系統(tǒng)將對(duì)市電和發(fā)電機(jī)供電進(jìn)行監(jiān)控。

Since this project was undertaken in a live facility, several special considerations had to be addressed. In order to safely replace the existing switchgear, a temporary stand-alone power plant, sized to support all of the data center loads, was assembled in a parking lot just outside the building’s existing electric/switchgear room (see Figures 4-6). The temporary power plant consisted of a new utility transformer, powered from one of the utility’s ATOs, which supplies power to an automatic transfer switch (ATS). The ATS supplies power from either the utility feeds or the standby-rated engine generators to a new distribution switchboard to support data center loads. The switchboard was installed inside a small building to protect it from the elements. Maintenance bypass switches enable staff to work on the ATS.

該項(xiàng)目由于是在線改造，需要考慮到特殊情況。為了順利更換開關(guān)柜，在大樓配電室外的停車場位置搭建一個(gè)臨時(shí)的發(fā)電站，足以支撐整個(gè)數(shù)據(jù)中心全部負(fù)載（見圖4～6）。臨時(shí)發(fā)電站有1臺(tái)新變壓器，由現(xiàn)有的1路市電ATO進(jìn)行供電，輸出引至ATS。ATS完成市電與發(fā)電機(jī)互投后，輸出至新配電柜給數(shù)據(jù)中心負(fù)載供電。配電柜安裝于一座小建筑物內(nèi)以免受到干擾。通過旁路檢修開關(guān)，運(yùn)維人員可以對(duì)ATS進(jìn)行維護(hù)。

Figure 4. Maintenance bypass switches were installed to allow for work on the ATS

圖4 安裝檢修旁路開關(guān)，以便檢修ATS

Figure 5 (Top) and 6 (Bottom). Switchboard was installed in a small building

圖5（上）和圖6（下） 小建筑物內(nèi)的開關(guān)柜

Each standby-rated engine generator has two sources of fuel oil. The primary source is from a bulk tank, with additional piping connected to the site’s two existing 10,000-gallon fuel oil storage tanks to allow for filling the bulk tank or direct feed to the engine generators (see Figure 7).

每臺(tái)發(fā)電機(jī)有兩路燃油源。現(xiàn)場有2個(gè)1萬加侖儲(chǔ)油罐通過油管連接大容量油箱與發(fā)電機(jī)，主用油路通過大容量油箱給發(fā)電機(jī)提供燃油，亦可通過儲(chǔ)油罐直接提供燃油（見圖7）。

Transferring Data Center Loads

數(shù)據(jù)中心負(fù)載遷移

U.S. Bank’s commissioning of the stand-alone power plant including testing the ATS, load testing the engine generators, infrared (IR) scanning all connections, and a simulated utility outage. Some additional cabling was added during commissioning to address cable heating due to excessive voltage drop. After commissioning was completed, data center loads were transferred to the stand-alone plant. This required providing temporary circuits for select mechanical equipment and moving loads away from four panelboards (two for mechanical equipment and two for the UPS), so that they could be shut down and re-fed from the temporary power plant. The panelboards were transferred one at a time to keep the data center on-line throughout all this work. The transfer work took place over two weekends.

發(fā)電站調(diào)試內(nèi)容包括ATS測試、發(fā)電機(jī)加載測試、紅外掃描全部電氣連接點(diǎn)，以及市電停電模擬。調(diào)試期間，通過外加電纜解決過度壓降引起的電纜發(fā)熱現(xiàn)象。調(diào)試結(jié)束后，搭建臨時(shí)電路，將2臺(tái)動(dòng)力配電箱和2臺(tái)UPS配電箱轉(zhuǎn)移到臨時(shí)發(fā)電站，將數(shù)據(jù)中心負(fù)載遷移到發(fā)電站。一次轉(zhuǎn)移一個(gè)回路負(fù)載，保證數(shù)據(jù)中心在線運(yùn)行。整個(gè)負(fù)載遷移工作用時(shí)兩個(gè)周末。

The mechanical loads were sequenced first in order to put load on the stand-alone plant to provide a stable power source when the UPS systems were cut over and brought on-line. Data center loads were transferred to engine-generator power at the beginning of each day to isolate the data center from the work.

按照順序先將動(dòng)力負(fù)載轉(zhuǎn)移到臨時(shí)發(fā)電廠，以便UPS上線時(shí)能提供穩(wěn)定的電源供應(yīng)。每天一早，把數(shù)據(jù)中心負(fù)載轉(zhuǎn)移到發(fā)電機(jī)，動(dòng)力負(fù)載進(jìn)行隔離。

On the first Saturday devoted to the transfer process, the mechanical loads were rotated away from the first panelboard to be re-fed. Equipment requiring temporary power was cut over (see Figure 8). The isolated panelboard was then shut down and re-fed from the stand-alone plant. Once the panelboard was re-fed and power restored to it, equipment receiving temporary power was returned to its normal source. Mechanical loads were rotated back to this panelboard, so that the second panelboard could be shut down and re-fed. Data center loads were transferred back to utility power at the end of each day.

第一周周六，對(duì)第一臺(tái)配電盤動(dòng)力負(fù)載進(jìn)行遷移工作。對(duì)臨電負(fù)載進(jìn)行轉(zhuǎn)移（圖8）。隨后，切斷隔離配電盤，再連接到發(fā)電站。當(dāng)配電箱恢復(fù)供電，設(shè)備也就恢復(fù)正常供電。再將動(dòng)力負(fù)載遷回該配電箱，以便對(duì)第二臺(tái)配電箱進(jìn)行遷移作業(yè)。每天結(jié)束時(shí)，恢復(fù)數(shù)據(jù)中心市電供電。

The Sunday mechanical cut over followed the same sequence as Saturday, except the stand-alone power plant, with live load, was tested at the end of the day. This testing included having Xcel Energy simulate a utility outage to the data center, which the utility did with data center loads still on engine-generator power so as not to impact the data center.

第二天（周日），用同樣的方法對(duì)第二臺(tái)配電盤動(dòng)力負(fù)載進(jìn)行遷移操作。當(dāng)天工作結(jié)束前，還對(duì)發(fā)電站進(jìn)行了帶載測試。Xcel Energy公司模擬了數(shù)據(jù)中心市電停電，通過發(fā)電機(jī)以保證數(shù)據(jù)中心正常運(yùn)行。

UPS were transferred the following weekend. On Saturday, the two UPS systems were transferred to engine-generator power and put into maintenance bypass so their primary power sources could be re-fed from the stand-alone power plant. At the end of the day, the two UPS systems went back on-line and transferred back to utility power. On Sunday, workers cut over the UPS maintenance bypass source. That day’s work concluded with additional testing of the stand-alone power plant, including another simulated utility outage to see how the plant would respond while supporting entire data center.

第二個(gè)周末對(duì)UPS負(fù)載進(jìn)行了遷移。周六，將2臺(tái)UPS轉(zhuǎn)換到發(fā)電機(jī)供電回路，并將開關(guān)柜切換至維護(hù)旁路，以便將UPS主用電源轉(zhuǎn)換到發(fā)電站上。當(dāng)天工作結(jié)束前，恢復(fù)2臺(tái)UPS市電供電。周日，將UPS切換至外部維護(hù)旁路。當(dāng)天，最后還對(duì)發(fā)電站進(jìn)行了測試，再次模擬了市電停電以觀察發(fā)電站的運(yùn)行情況，數(shù)據(jù)中心完全由發(fā)電站供電。

Figure 7. Standby-rated engine generators have two sources of fuel oil

圖7 后備發(fā)電機(jī)組

Figure 8. Data center loads were transferred to the  temporary stand-alone power plant over the course of two weekends

圖8 數(shù)據(jù)中心負(fù)載遷移到臨時(shí)發(fā)電站

Cable Bus Installation

電纜橋架安裝

At the same time the stand-alone power plant was assembled and loads cut over to it, four sets of cable trays and cables were installed to facilitate dividing the UPS loads. These four sets of cable trays had to be run through office and production areas to get to the existing UPS room, which is a run of approximately 625 feet (see Figure 9). Each tray served one of the four primary and maintenance bypass UPS systems.

在組裝發(fā)電站、將負(fù)載轉(zhuǎn)移至發(fā)電站的同時(shí)，完成了4組橋架安裝與電纜敷設(shè)，以實(shí)現(xiàn)UPS負(fù)載均衡分配。4組UPS主路和旁路電纜橋架穿越辦公區(qū)與生產(chǎn)區(qū)到達(dá)UPS配電室，全長接近625英尺（見圖9）。

Figure 9. New cable buss ran about 625 feet through the facility

圖9 電纜敷設(shè)約625英尺

Switchgear and Generators

開關(guān)柜與發(fā)電機(jī)組

After the data center loads were transferred over to the stand-alone power plant, the old switchgear was disconnected from utility power so it could be disassembled and removed from the facility (see Figures 10 and 11). Then, the new switchgear was installed (see Figures 12 and 13).

數(shù)據(jù)中心負(fù)載全部遷移到發(fā)電站后，移除舊的開關(guān)柜（圖10和圖11），安裝新開關(guān)柜（圖12和圖13）。

The switchgear was designed for even distribution of loads, with an A (yellow) side and a B (blue) side (see Figure 14). Each side supports one of the two UPS systems, one of the two chillers with its pumps and towers, and half of the computer room cooling units.

開關(guān)柜采用A（黃色標(biāo)牌）B（藍(lán)色標(biāo)牌）兩路負(fù)載均衡設(shè)計(jì)（見圖14）。A/B路各給1套UPS和1套制冷單元（冷機(jī)及相關(guān)的泵、冷卻塔），以及一半機(jī)房空調(diào)進(jìn)行供電。

Figure 10 and 11. Old switchgear was disassembled and removed from the facility

圖10圖11 舊開關(guān)柜拆除

After installation, portable load banks were brought in for commissioning the new switchgear. The engine generators also received a full re-commission due to the changes in the controls and the additional alarms.

開關(guān)柜安裝完成后，用便攜式假負(fù)載對(duì)新開關(guān)柜進(jìn)行測試。由于更換了控制系統(tǒng)與新增報(bào)警功能，重新對(duì)發(fā)電機(jī)組進(jìn)行全面調(diào)試。

Figure 12 and 13. New switchgear was installed

圖12圖13 新開關(guān)柜安裝

Figure 14. Switchgear supporting Yellow and Blue sides, equally dividing the critical load

圖14 雙路開關(guān)柜（黃藍(lán)牌）

After the new switchgear was fully commissioned, data center loads were cut over to the new switchgear following a similar transfer sequence as to the stand-alone power plant. The panelboards supporting mechanical and UPS equipment were again each cut over one panel at a time to keep the data center on-line, again requiring transferring data center loads to engine-generator power to isolate the data center throughout this work.

新開關(guān)柜全面調(diào)試結(jié)束后，采用負(fù)載遷移到發(fā)電站同樣的順序，將數(shù)據(jù)中心負(fù)載遷移到新開關(guān)柜。一次遷移1個(gè)配電回路，將動(dòng)力配電箱和UPS配電箱切換到發(fā)電機(jī)組，隔離數(shù)據(jù)中心負(fù)載以保證其運(yùn)行。

Figure 15 and 16. Upgraded engine-generator controls and alarming were installed, with panels installed in the Engineers Office

圖15圖16 更換后的控制系統(tǒng)與報(bào)警面板

As previously mentioned, upgraded engine-generator controls and alarming were installed as part of the project (see Figures 15 and 16). The older controls had to be upgraded to allow communication with the new switchgear. Upgraded alarm panels were installed in the Engineering Office. In addition, each switchboard has a SCADA screen with a workstation installed in the Engineering Office (see Figure 17). The project also included updating MOPs for all aspects of the switchgear operation (see Figure 18).

正如前面提到，本次改造包括發(fā)電機(jī)控制系統(tǒng)與報(bào)警的升級(jí)（圖15和圖16）。升級(jí)控制系統(tǒng)以實(shí)現(xiàn)與新開關(guān)柜的通信，新的報(bào)警面板安裝在工程部辦公室。每臺(tái)配電柜配一塊SCADA顯示屏，系統(tǒng)工作站安裝于工程部辦公室（圖17）。還對(duì)開關(guān)柜維修操作手冊進(jìn)行了修改（圖18）。

Figure 17. New switchgear includes a new SCADA system

圖17 新開關(guān)柜SCADA系統(tǒng)

Figure 18. Updated MOPs for the switchgear

圖18 新版開關(guān)柜維修操作手冊

The overall project went well and was completed on time with no impact to the data center. Since this phase of the project was completed, we have performed a number of live load engine-generator tests, including a few brief utility power tests, in which the engine generators were started and supported transferred load. In each test, the new equipment performed great. Phase 2 of the modernization project is the replacement of UPS System 1, which is currently underway and anticipated to be completed later in 2014. Phase 3 is replacement of UPS System 2, scheduled for 2015

該項(xiàng)目整體進(jìn)行很順利，按時(shí)完成了改造工作，沒有對(duì)數(shù)據(jù)中心運(yùn)行造成影響。在本階段結(jié)束后，我們對(duì)發(fā)電機(jī)進(jìn)行了帶載試驗(yàn)，包括發(fā)電機(jī)待機(jī)狀態(tài)下的幾次市電測試。每次測試中，新設(shè)備運(yùn)行非常好。現(xiàn)代化配電改造的第二階段是更換UPS-1系統(tǒng)，目前系統(tǒng)在使用中，預(yù)計(jì)2014年晚些時(shí)候完成更換。第三階段是更換UPS-2系統(tǒng)，計(jì)劃于2015年完成更換改造。

Mark Johns is chief engineer, U.S. Bank IT Critical Facilities Services. He has more than 26 years data center engineering experience, completing numerous Infrastructure upgrade projects, including all commissioning, without interruption to data center operations. Mr. John’s long career prior to U.S. Bank includes working in a 7-story multi-use facility, which includes data center operations, check processing operations, and support staff.