FINAL REPORT TO THE CPUC
S. David Freeman, Trustee
Re: Current Status of the ISO and PX Contracts for Hardware and Software
Last winter we came to the table with many strong industry players, and those
players came with varied and conflicting agendas. We challenged ourselves to
rise to a higher and unified goal, and went to work. I'm proud to report that
we've progressed from plans and dreams to two functioning companies committed to
launching a new California electricity market on 1/1/98.
We were fortunate to have a wide range of Technical Advisory Committee (TAC)
members who were intimately familiar with the technical functions of the ISO and
PX and the intentions of the commission, the Governor and the legislature. We
must give full credit the TAC members who helped us sort through the tradeoffs
and reach the decisions that were necessary to meet our schedule. We would not
be where we are today had it not been for their expert advice and their time and
effort.
In the first few weeks, we refined the thoughtful planning and prepared the
requests for proposals to build the necessary hardware and software. In early
1997, the TAC and I put in place the contracts that have become the backbone of
the ISO and PX. We didn't have the luxury of fully developed concepts that had
gone through thorough revisions or staged design reviews, nor did we have the
luxury of several rounds of competitive bidding for the contracts. Time was of
the essence. To meet 1/1/98, we needed contracts to get the work underway
without delay.
We negotiated and secured contracts with major players in the industry who have
become our partners. Despite the necessary rush, the contracts we have executed
have been effective. They have served us well in this large and complex effort.
We have made some changes along the way, and have faced some change orders, but
no more than are typically experienced. Where the need for change has become
apparent, we have acted without hesitation.
In addition to the contracts, we undertook the massive FERC filings, the
reliability studies and the recruitment of key personnel during the months
before the corporations began to function. Without our initiatives in these
areas, making the 1/1/98 deadline would be impossible.
Certainly there is considerable work remaining to fulfill the hardware, software
and facilities contracts. This work involves dealing with the usual design bugs
and implementation challenges that any large complex system faces.
As we stand ready to hand off our work to the Corporations, we take great pride
in what we have accomplished and what we have set in motion.
I would like to summarize the status of the key contracts to point out that much
of the hardware has been built and the remaining development and testing are on
a schedule to meet the 1/1/98 deadline, but with very little time to spare.
CALIFORNIA INDEPENDENT SYSTEMS OPERATOR (ISO)
Power Management System (PMS) - ABB System Control
The Power Management System will perform the real-time monitoring, control and
analyses of the ISO-coordinated power system. The integrated system will consist
of new primary and backup Power Management Systems at the ISO primary and backup
facilities, which will be linked with the existing energy management systems at
PG&E, SCE, SDG&E and participating municipal utility control centers.
The PMS contract was awarded to ABB System Control and provides for both the
primary development work on the Power Management System, costing $8.4 million
and a software maintenance agreement of $1.7 million. Change orders
approximating $1.2 million are expected, but these change orders will have only
a slight impact on the work schedule and will not endanger the 1/1/98 completion
date. The total anticipated cost of $11.3 million is well below the original
estimate submitted to the Commission of $30.0 million.
Major accomplishments related to the ISO PMS to date include:
- - equipment installed at ABB for testing;
- - test systems installed at both the ISO primary and backup facilities in
April;
- - Inter Control Center Protocol (ICCP) implemented at all three IOU sites and
initial testing with the PMS system implemented at the ABB site;
- - basic functionality test of the new Dispatch Coordination scheme for
generation control completed 8/15/97.
The Phase 1 control scheme is a 2-level hierarchical scheme. The ISO's PMS will
monitor the ISO's control area and coordinate the multiple subordinate Area
Control Centers (ACCs). The ACC's will continue to utilize the existing
generation control schemes (i.e. utility energy management systems) to move the
generating units to satisfy the ISO ancillary services generation requirements.
This control scheme has been devised with the following guiding principles in
mind:
- - minimize the changes to existing generation control infrastructure
components, such as control software, hardware, data links and operating
procedures;
- - allow the ISO to operate as a single control area:
- - enable the ISO to allocate normal or emergency generation requirements to
page 2
subordinate ACCs to meet overall ISO ACE;
- - establish a fall-back mode which maintains basic generation control integrity
if certain ISO-ACC data links are out of service.
During the 4th quarter of 1997, the operational dry run will be started and
completed, as will real time scheduling and integration with other systems.
There is good reason to expect that on 1/1/98, the ISO will commence function as
a single control area.
Scheduling Infrastructure, Congestion/Transmission Management Software, Business
Systems (SI/SA/BBS) - ISO Alliance
The Scheduling Infrastructure (SI) accepts and validates the generation
schedule. Scheduling Applications (SA) evaluates feasibility of implementing the
submitted schedules. Congestion/Transmission Management Software refers to the
software necessary to assist in congestion management. Finally, the Balance of
Business Systems (BBS) calculates settlements and production billing.
The ISO Systems contract was awarded to a consortium composed of ABB System
Control, Perot Systems and Ernst & Young - known collectively as the ISO
Alliance. The main contract provides for two stages of work for the primary ISO
systems, with Stage I originally priced at $45.6 million and Stage II at $3.4
million. As of mid-August, various change orders and development center costs
have increased this total from $49 million to $56.5 million. These approximately
$7 million in change orders were only recently approved and are primarily
focused on improving original design specifications. This cost is still below
the original estimate for all these contracts approved by the CPUC of $63.0
million.
Major design components for SA and SI were completed during the 3rd quarter of
1997, including the initial SI prototype, which was ready and available for
demo, and the test objectives, conditions and scripts for SA. Major components
of the BBS design were completed on schedule and the remaining construction and
testing phases are scheduled to meet the 1/1/98 deadline.
The ISO Alliance has further achieved the following:
- - support infrastructure is in place and the IBM help desk is working and
available;
- - all hardware has been procured and is being installed;
- - all design changes have been frozen, thus finalizing implementation
requirements for 1/1/98.
The problem that have been encountered recently are in the interface with the PX
and other scheduling coordinators. We have moved decisively to solve them due to
the excellent work of our Coopers & Lybrand project management team and continue
to anticipate a 1/1/98 start date.
page 3
ISO Communications Infrastructure Development - MCI
The Communications Infrastructure refers to the voice and data communications
that allow the ISO and PX to communicate with each other and with market
participants, to control and monitor the power grid and to provide a
communications backbone for the transport of metering data. The infrastructure
has been designed for as many as 2,000 connected entities.
The ISO Communication Infrastructure contract was awarded to MCI for $25.8
million for the initial installation compared to the original budget for this
item of $30 million. A voice and data service lease is expected to result in a
five year cost for service of $27.4 million annually commencing in 1998,
increasing to $27.8 million per annum thereafter.
Recent ISO Communications accomplishments include:
- - direct communication completed servicing Alhambra and the primary site at
Folsom;
- - completion of most wide area network facilities;
- - development of an integrated testing plan to compress overall time for other
vendor testing;
- - hand over of the network by MCI to the ISO for use and testing on 7/31/97.
This network will rank with the largest in the world and was developed and
available for the ISO to test in less than five months, which is unheard of for
a project of this size and complexity.
ISO Metering Data Acquisition System (MDAS) - UTS
The ISO Metering Data Acquisition Infrastructure refers to the metering
standards, data servers, interface equipment, databases and software that allow
the ISO to collect this data. The ISO MDAS, which was procured from Utility
Translation Systems, Inc. (UTS), is a system that is generally in operation in
the UK. The only new requirements for the ISO were interfaces to other
subsystems, interfaces to Scheduling Coordinator's Meter Data Acquisition
Systems and communicating over a wide area network using TCP/IP.
MDAS is on schedule. The system remains close to its original budget of $4.46
million with only $232,725 in change orders for interfaces requested and
approved. The original budget for this step was $5.0 million.
MDAS achievements thus far:
- - On August 14th, the MDAS Meter Prototype COMM Test was successfully completed
one day ahead of the planned test schedule. The system exceeded contract
expectations with the test demonstrating that MDAS can communicate over a
wide area network using TCP/IP.
- - Interface designs have been completed for ISO Sub-Systems with testing
scheduled for late September.
page 4
Interface testing with scheduling coordinators (SCs) will occur during the
upcoming operational dry run (ODR). UTS has communicated broadly and freely with
the SCs to ensure that meter data, which is crucial for a successful ISO, can be
transferred from their systems to Settlements by 1/1/98. Outlook for this system
continues to be excellent.
ISO Primary and Backup Facilities
The ISO/PX facility in Folsom is 79,000 square feet in area, 3,232 square feet
of which is allocated to the PX backup facility, leaving some 75,800 square feet
for the primary ISO operations. A backup ISO facility is located in Alhambra, in
close proximity to the PX site.
The tenant improvement work for the primary ISO site is being performed by
Turner Construction. The cap on the Turner Construction contract currently
stands at $6.75 million with $350,000 of this allocated to the PX backup
facility. The ISO backup facility in Alhambra is being improved by Coastal
Pacific Construction at a cost currently estimated at $1.04 million. Tenant
improvements at this site are substantially complete. The total cost of the
Folsom facility and the Alhambra backup is $13.9 million as compared to the
original estimate of $17.0 minion.
Approximately 50,000 square feet of the ISO primary facility baa been completed
and 25,000 square feet was available to ISO staff 6/30/97 - almost 3 months
ahead of the originally contracted 10/1/97 turnover to the ISO.
Major accomplishments to date at the ISO facilities are as follows:
- - Completed construction of the Phase I areas of the Folsom site, including the
telecommunication, main computer and development room early in the 2nd
quarter of 1997.
- - Construction of the Phase II areas, including the dispatch and network
systems room, was completed on 6/30/97.
- - Temporary occupancy permits for Phase I and Phase II were obtained from the
City of Folsom on 7/7/97.
- - Computer and communication room facilities at both the ISO primary and backup
sites were completed on schedule with no negative Impact on vendor computer
equipment schedules.
Construction of the Phase III area - the remaining office area - is scheduled
for completion by 8/30/97, at which time all service level agreements will be in
place. The remaining minor architectural features will be completed by 9/1/97.
Mapboards and consoles are on order and expected to arrive on schedule.
page 5
California Power Exchange (PX) Corporation
The PX Corporation began its development in the first quarter of this year.
During the first quarter, vendors were selected and all primary development
tasks commenced.
PX Business Systems - OM Technology AB
The PX business systems refer to the computers and other systems to support the
following functions: Bidding, Electricity Scheduling, Settlements, Billing, and
Administration. The successful bidder, OM Technology AB, was selected after a
competitive bid process that included a written evaluation of the bids, oral
review of vendors, review evaluations, competitive negotiations, and
recommendation to the Trustee by the TAC for selection. The contract was signed
on March 27, 1997. Contract attachments, including technical schedules and a
schedule of milestones and related payments were completed on April 23, 1997.
Phase I expenditures to provide for the basic functional requirements are
estimated at $23 million. As of mid-August, various change orders have
increased the total to $25.3 million. For many reasons, as development
progressed, the work was broken into stages. Stage 1 (providing day ahead
market without iterations) and Stage 1a (real time settlements) will be
delivered 1/1/98 while Stage 2 (hour ahead market) and Stage 3 (iterative
bidding) will be delivered after 1/1/98 (on 2/15/98 and 4/30/98 respectively).
Phase II expenditures to provide enhancements to the system are estimated at
$2.0 million. The total of $25 million is well below the original estimate of
$31 million.
There is a major issue involving the ISO/PX interface; one of the most
significant unresolved items is the design divergence regarding Net/Individual
Schedules. On August 11th during the first design walk-through with ISO and the
scheduling coordinators, including the PX, it was discovered that the ISO
Alliance made design assumptions that diverged from requirements followed by
the PX and other scheduling coordinators. The ISO built its systems around net
schedules while the PX built its around individual schedules as required by the
FERC Tariff and the ISO DSOW. This issue, if undiscovered would have made it
impossible for the ISO and PX systems to communicate. It was uncovered prior to
the operational dry run through the diligent oversight of our project manager.
However, this issue poses serious risks to the project. The modification of the
ISO system to handle individual schedules will not be completed until early
December. This allows little more than a week to test the interfaces with an
operational dry run and perform a production turnover.
Computing Services Infrastructure - IBM
The computing services infrastructure contract with IBM includes the network
and system management tools required by the PX to monitor the reliability and
performance of its computing environment, as well as office automation
hardware/software such as personal computers and associated desktop software,
file servers, printers, copiers, FAX machines,
page 6
and a Help Desk System for problem reporting, tracking and resolution
coordination. The price for this contract is estimated at $2.58B million for
year one of the proposed lease.
PX Facilities
The PX Primary Facility location was approved on 1/9/97 to be established in
Alhambra (Los Angeles area), in close proximity to the ISO backup facility. Both
facilities are managed by Lincoln Properties. The backup facility for the PX has
been co-located with primary ISO facility, in Folsom (Sacramento area). An
allocation of approximately 4% of the costs of this facility has been made to
the PX, based on square footage that the PX will occupy.
The primary facility will house the PX equipment and personnel. The construction
of the facility is completed, the offices and computer rooms are finished, and
the utility and telephones lines are operation. We have a temporary occupancy
permit that allows the PX to begin working at the facility as of 8/25/97. The
current cost for the facility including lease, construction, and furnishings is
at $3.6 million as compared to an initial estimate of $6.0 million.
Communicates Infrastructure - MCI
This refers to the voice and data communications that allow the ISO and PX to
communicate with each other and with the market participants, control and
monitor the power grid, as well as provide a communications backbone for the
transport of metering data. Duke Engineering & Services is responsible for both
PX and ISO Communication Infrastructure project management.
The Communications Infrastructure contract was awarded to MCI, as MCI's proposal
was evaluated as the most comprehensive and technically competent. The PX was
allocated a share of this cost equal to $5.65 million for the development and
implementation of the voice and data communications infrastructure is
equivalent to the original budget estimate. This infrastructure included some
on-premises equipment that will be transferred to the PX at the end of the
agreement.
Conclusion
In summary, while there is no time to spare, all of the hardware and software
necessary to commence operation of the ISO and PX are either completed or on
schedule to be completed and tested before 1/1/98. The original estimate for the
hardware, software and rites (excluding the start-up costs) was $223.0 million.
Our current contracts should complete the work needed for a 1/1/98 startup (as
well as the hardware and systems to be completed during 1998) for $212.2
million.
page 7
[GRAPHIC OMITTED]
CALIFORNIA INDEPENDENT SYSTEM
OPERATOR
November 24,1997
Mr. H. Ronald Nash
Vice President, Perot Systems Corporation
12377 Merit Drive
Suite 1100
Dallas, Texas 75251
Dear Ron:
Many thanks for your and Chuck Bell's trip to Folsom last week to help
work out the conflict of interest matter involving Perot's contemplated
consulting services ("the Services") in conjunction with Policy Analysis
Corporation ("PAC"). I believe that we are in agreement on the need to move
forward quickly to remedy the situation.
Your recognition that the appearance of conflict weighs as heavily as
actual conflict was most welcome. Public confidence in the fairness of the ISO
and PX Systems must be one of its salient characteristics; it is a major and
substantial interest of ours - and the FERC's - in the design and procurement of
the system and the market. That the ISO is a quasi-public if not a public entity
and the procurement funds for the system are quasi-public if not public funds,
are important elements of the environment in which we are operating.
Achievement of the December 5 deadline for having our conflict solution
in place will be important, as I outlined. By way of summary, the elements that
need to be available or in place are as follows:
- The form of a disclaimer which will be issued to past and future
prospects for the Services. We are to have the total number of
prospects approached and "pitched"; you will approach all past and
future prospects and secure permission to disclose their identity,
if possible.
- Provide a set of Ethics Assurance Procedures to assure appropriate
isolation of the ISO/Alliance Project from the offering and
providing of the Services. We need to identify the Project personnel
in their varying capacities and their exposure or access to
protected or non-public information relating to the Project, and
identify as well the appropriate applicatory time periods.
- Provide a control process to assure the continuing implementation of
the above.
- Provide certification in appropriate form that no protected or
non-public information has been released, either to PAC or to
prospective clients in the course of soliciting clients for the
services or actually providing the services, as well as that the
above described activities are being implemented-
151 BLUE RAVINE ROAD - FOLSOM, CALIFORNIA - 95630
TELEPHONE: 916.151.2222 'FACSIMILE: 916.351.2350
-2- November 24,1997
I appreciate your looking into the form of the "teaming" arrangements
between Perot and PAC for provision of the services so that we can be aware of
their circumstances. Also of importance is your continuing assessment regarding
how the ISO can receive assurance that the system is free of any gratuitous
insertions or "hooks" designed to enable some feature of the Services, both past
and future.
Again, Ron, thank you for your positive response to our concerns. We
look forward to responding rapidly to the items set forth above, after we
receive your draft proposal.
Sincerely,
/s/
Jeffrey Tranen
President and Chief Executive Officer
CALIFORNIA INDEPENDENT SYSTEM
OPERATOR
October 22, 1997
Mr. H. Ronald Nash
Vice President
Perot Systems Corporation
Suite 1100
12377 Merit Drive
Dallas, Texas 75251
Re: ISO Alliance and Perot Systems Corporation Conflicts of Interest
Dear Mr. Nash:
This is to follow up on our telephone conversation of October 21, 1997
regarding reported solicitations by Perot Systems Corporation ("Perot Systems")
of parties expected to market energy in California.
As we discussed, such solicitations are inconsistent with the ISO's
Alliance's and Perot Systems' conflict of interest obligations under the
Scheduling Applications, Scheduling Infrastructure and Business Systems Contract
between the ISO Alliance and the ISO Restructuring Trust dated as of March 14,
1997 (the "Contract") which you executed on behalf of the ISO Alliance and Perot
Systems.
Specifically, it was reported to me that representatives of Perot Systems
contacted Gary Cotton of San Diego Gas Sc Electric (SDG&E), and perhaps other
potential market participants, and offered the services of Perot Systems in
helping SDG&E to exploit the new California energy market by exploiting
potential weaknesses and shortcomings in the ISO's system to their commercial
advantage.
It was further reported that Perot Systems' representatives at these
solicitations included Ed Smith and Paul Gribik, both of whom have been
significantly involved in ISO Alliance's and Perot Systems' activities in
negotiating and performing the Contract, and that Perot Systems' involvement in
developing the ISO's system was cited as evidence of Perot Systems' knowledge of
the potential weaknesses and shortcomings in that system.
Perot Systems' marketing of its inside knowledge of the ISO's system to
third parties so that they may economically exploit the new California energy
market, in addition to being a flagrant violation of basic norms of business
ethics and indicative of bad faith dealing, would seriously erode the integrity
of the new California energy market and materially compromise the work being
performed and the system being produced by the ISO Alliance and Perot Systems
for the ISO. Article 31 of the Contract expressly prohibits the ISO Alliance,
including Perot
151 BLUE RAVINE ROAD - SACRAMENTO, CALIFORNIA - 95630
PHONE: 916.351.2222 - FAX: 916.351.2181
- 2 - October 22, 1997
Systems, ABB and Ernst & Young, from performing services for others which may
create a material conflict of interest with the ISO or in any way otherwise
materially compromise the work being performed by the ISO Alliance and Perot
Systems on behalf of the ISO.
Perot Systems' behavior as described above is contrary to Perot Systems'
contractual obligations and is expressly prohibited under the Contract. While
reserving all rights and remedies available to the ISO, including but not
limited to those concerning breach and termination, under the Contract and
applicable law, the ISO will require that Perot Systems immediately:
(i) cease the solicitation and marketing as described above;
(ii) contact those parties previously solicited in this regard in writing,
retracting it's offer of such consulting services and stating that
such consulting services will not be offered by Perot Systems or the
ISO Alliance;
(iii) provide the ISO with a list of names of the market participants
actually contacted by Perot Systems;
(iv) describe what specifically was being marketed by Perot Systems and
the basis for such solicitations with such itemization to be renewed
at the end of Stage I and Stage II of the Contract;
(v) certify that neither Perot Systems nor the ISO Alliance has
introduced any changes or modifications other than those specified by
the ISO Contract, the Detailed Statement of Work (DSOW), or
documented change orders, with such certificates to be renewed at the
end of Stage I and Stage n of the contract; and
(vi) pursuant to Article 31.7 of the Contract provide all accounts and
records relating to any program of solicitation activity in this
regard.
Finally, as I emphasized in our telephone conversation, prompt cessation of
the offending solicitation activities and prompt retraction of all express
offers of such consulting services is essential to any ultimate resolution
of this matter. I look forward to hearing your response and the status of
any additional activities you may propose for Perot Systems and the ISO
Alliance to remedy this situation.
Sincerely,
/s/
Jeffrey D. Tranen
President and Chief Executive Officer
cc: Ake Almgren, President, ABB T&D Inc.
William Hunter, Operations Partner, Ernst & Young