EXHIBIT 99.4 PROMISSORY NOTE $3,000,000.00 December 31, 2004 Daniel R. O'Neal ("Borrower") promises to pay to the order of American Physicians Assurance Corporation ("APA"), in lawful money of the United States of America, at its principal office in East Lansing, Michigan or wherever else APA may specify, the sum of Three Million and No/100 Dollars ($3,000,000.00), with interest on the unpaid principal balance at the rate and on the terms provided in this Promissory Note (including all renewals, extensions or modifications hereof, this "Note"). USE OF PROCEEDS. Borrower shall use the proceeds of the loan evidenced by this Note to acquire shares of Physicians Insurance Company, a Florida stock insurance corporation ("PIC"), owned by APA. INTEREST RATE. Interest shall accrue on the unpaid principal balance of this Note from the date hereof at 8% ("Interest Rate"). DEFAULT RATE. In addition to all other rights contained in this Note, if a Default (as defined below) occurs and as long as a Default continues, all outstanding Obligations shall bear interest at the Interest Rate plus 3% ("Default Rate"). The Default Rate shall also apply from acceleration until the Obligations (as defined below) or any judgment thereon are paid in full. INTEREST AND FEE(S) COMPUTATION (ACTUAL/360). Interest and fees, if any, shall be computed on the basis of a 360-day year for the actual number of days in the applicable period ("Actual/360 Computation"). The Actual/360 Computation determines the annual effective interest yield by taking the stated (nominal) rate for a year's period and then dividing said rate by 360 to determine the daily periodic rate to be applied for each day in the applicable period. Application of the Actual/360 Computation produces an annualized effective rate exceeding the nominal rate. REPAYMENT TERMS. This Note shall be due and payable in 84 consecutive monthly payments of principal plus accrued interest, commencing on January 31, 2005, and continuing on the same day of each month thereafter until fully paid in accordance with the attached repayment schedule. In any event, subject to "Remedies Upon Default," all unpaid principal and accrued interest shall be due and payable on the due date of the 84th payment. APPLICATION OF PAYMENTS. Monies received by APA from any source for application toward payment of the Obligations shall be applied to accrued interest and then to principal. If a Default occurs, monies may be applied to the Obligations in any manner or order deemed appropriate by APA. If any payment received by APA under this Note or the other Loan Documents is rescinded, avoided or for any reason returned by APA because of any adverse claim or threatened action, the returned payment shall remain payable as an obligation of all persons liable under this Note or other Loan Documents as though such payment had not been made. DEFINITIONS. LOAN DOCUMENTS. The term "Loan Documents", as used in this Note, refers to all documents executed in connection with or related to the loan evidenced by this Note and may include, without limitation, this Note, the Stock Pledge Agreement between Borrower and APA of even date herewith (the "Stock Pledge Agreement"), and any renewals or modifications, whenever any of the foregoing are executed. OBLIGATIONS. The term "Obligations", as used in this Note and the other Loan Documents, refers to any and all indebtedness and other obligations under this Note and all other obligations under any other Loan Documents. CERTAIN OTHER TERMS. All terms that are used but not otherwise defined in any of the Loan Documents shall have the definitions provided in the Uniform Commercial Code. SECURITY. The Obligations are secured by the security interest granted to APA by Borrower pursuant to the Stock Pledge Agreement. LATE CHARGE. If any payments are not timely made, Borrower shall also pay to APA a late charge equal to 5% of each payment past due for 5 or more days. Acceptance by APA of any late payment without an accompanying late charge shall not be deemed a waiver of APA's right to collect such late charge or to collect a late charge for any subsequent late payment received. Any late charge paid shall be treated as liquidated damages and not as a penalty, to compensate APA for the expenses of administering the Default (as defined below). Notwithstanding any other provision of this Note, APA's acceptance of the full amount of any late payment with the accompanying late charge shall be deemed to cure the Default ascribed to such delinquent payment. ATTORNEYS' FEES AND OTHER COLLECTION COSTS. Borrower shall pay all of APA's reasonable expenses incurred to enforce or collect any of the Obligations including, without limitation, reasonable arbitration, paralegals', attorneys' and experts' fees and expenses, whether incurred without the commencement of a suit, in any trial, arbitration, or administrative proceeding, or in any appellate or bankruptcy proceeding. USURY. If at any time the effective interest rate under this Note would, but for this paragraph, exceed the maximum lawful rate, the effective interest rate under this Note shall be the maximum lawful rate, and any amount received by APA in excess of such rate shall be applied to principal and then to fees and expenses, or, if no such amounts are owing, returned to Borrower. DEFAULT. If any of the following occurs, a default ("Default") under this Note shall exist: NONPAYMENT; NONPERFORMANCE. The failure of timely payment or performance of the Obligations under this Note or any other Loan Documents. FALSE WARRANTY. A warranty or representation made or deemed made in the Loan Documents or furnished to APA in connection with the loan evidenced by this Note proves materially false, or if of a continuing nature, becomes materially false. CROSS DEFAULT. At APA's option, any failure to pay, perform or otherwise comply with any agreement or obligation under any other loans, contracts or agreements of Borrower or any Affiliate of Borrower with APA or its affiliates ("Affiliate" shall have the meaning as defined in 11 U.S.C. Section 101, except that the term "Borrower" shall be substituted for the term "Debtor" therein), including without limitation the Stock Pledge Agreement. CESSATION; BANKRUPTCY. The death of, appointment of a guardian for, appointment of a receiver for, assignment for the benefit of creditors of, or commencement of any bankruptcy or insolvency proceeding by or against Borrower or PIC. DEATH; DISABILITY; CESSATION OF DIRECTOR STATUS. The death or "total disability" of Borrower. For purposes of this Note, "total disability" refers to a condition resulting from injury or illness to Borrower which prevents Borrower from performing the duties he or she has previously performed, and could be reasonably expected to perform, on behalf of PIC, for a period of 365 consecutive days, and "total disability" shall be deemed to occur on the first day following the initial 365-day period. MATERIAL ADVERSE CHANGE. APA determines in good faith, in its sole discretion, that the prospects for payment or performance of the Obligations are impaired or there has occurred a material adverse change in the business or prospects of Borrower or PIC, financial or otherwise. REMEDIES UPON DEFAULT. If a Default occurs under this Note or any Loan Documents, APA may at any time thereafter, take any one or more of the following actions: LIEN. Foreclose its security interest or lien under the Stock Pledge Agreement without notice. ACCELERATION UPON DEFAULT. Accelerate the maturity of this Note and, at APA's option, any or all other Obligations, in each case by giving written notice of the same to Borrower, whereupon this Note and the accelerated Obligations shall be immediately due and payable; provided, however, if the Default is based upon a bankruptcy or insolvency proceeding commenced by or against Borrower, PIC or any guarantor or endorser of this Note, all Obligations shall automatically and immediately be due and payable. CUMULATIVE. Exercise any rights and remedies as provided under the Note and other Loan Documents, or as provided by law or equity. 2 SET OFF RIGHTS. APA shall have the right at any time to set off any other indebtedness or obligation that APA owes to Borrower against the Obligations that are then due and payable. FINANCIAL AND OTHER INFORMATION. Borrower shall deliver to APA such information as APA may reasonably request from time to time, including without limitation, financial statements and information pertaining to Borrower's financial condition. Such information shall be true, complete, and accurate. WAIVERS AND AMENDMENTS. No waivers, amendments or modifications of this Note and other Loan Documents shall be valid unless in writing and signed by an authorized officer of APA. No waiver by APA of any Default shall operate as a waiver of any other Default or the same Default on a future occasion. Neither the failure nor any delay on the part of APA in exercising any right, power, or remedy under this Note and other Loan Documents shall operate as a waiver thereof, nor shall a single or partial exercise thereof preclude any other or further exercise thereof or the exercise of any other right, power or remedy. Borrower waives presentment, protest, notice of dishonor, demand for payment, notice of intention to accelerate maturity, notice of acceleration of maturity, notice of sale and all other notices and formalities of any kind. Further, Borrower agrees that APA may extend, modify or renew this Note or make a novation of the loan evidenced by this Note for any period, and grant any releases, compromises or indulgences with respect to any collateral securing this Note without notice to or consent of Borrower and without affecting the liability of Borrower under this Note or any other Loan Document. MISCELLANEOUS PROVISIONS. ASSIGNMENT. This Note and the other Loan Documents shall inure to the benefit of and be binding upon the parties and their respective heirs, legal representatives, successors and assigns. APA's interests in and rights under this Note and the other Loan Documents are freely assignable, in whole or in part, by APA. In addition, nothing in this Note or any of the other Loan Documents shall prohibit APA from pledging or assigning this Note or any of the other Loan Documents or any interest therein to any Federal Reserve bank. Borrower shall not assign its rights and interest hereunder without the prior written consent of APA, and any attempt by Borrower to assign without APA's prior written consent is null and void. Any assignment shall not release Borrower from the Obligations. APPLICABLE LAW. This Note and, unless otherwise provided in any other Loan Document, the other Loan Documents shall be governed by and construed under the laws of the state of Michigan without regard to that state's conflict of laws principles. JURISDICTION. Borrower irrevocably agrees to non-exclusive personal jurisdiction in the state of Michigan. SEVERABILITY. If any provision of this Note or of the other Loan Documents shall be prohibited or invalid under applicable law, such provision shall be ineffective but only to the extent of such prohibition or invalidity, without invalidating the remainder of such provision or the remaining provisions of this Note or other such document. NOTICES. Any notices to Borrower shall be sufficiently given, if in writing and mailed or delivered to Borrower, 2760 NE 23rd Place, Pompano Beach, FL 33062 or such other address as provided hereunder, and to APA, if in writing and mailed or delivered to APA, 1301 North Hagadorn Road, East Lansing, MI 48825 or such other address as APA may specify in writing from time to time. In the event that Borrower changes Borrower's address at any time prior to the date the Obligations are paid in full, Borrower agrees to promptly give written notice of said change of address by registered or certified mail, return receipt requested, all charges prepaid. PLURAL; CAPTIONS. All references in the Loan Documents to Borrower, guarantor, person, document or other nouns of reference mean both the singular and plural form, as the case may be, and the term "person" shall mean any individual or entity. The captions contained in the Loan Documents are inserted for convenience only and shall not affect the meaning or interpretation of the Loan Documents. WAIVER OF JURY TRIAL. TO THE EXTENT PERMITTED BY APPLICABLE LAW, BORROWER BY EXECUTION HEREOF AND APA BY ACCEPTANCE HEREOF, KNOWINGLY, VOLUNTARILY AND INTENTIONALLY WAIVE ANY RIGHT THEY MAY HAVE TO A TRIAL BY JURY IN RESPECT OF ANY LITIGATION BASED ON, OR ARISING OUT OF, UNDER OR IN CONNECTION WITH THIS NOTE, THE LOAN DOCUMENTS OR ANY AGREEMENT CONTEMPLATED TO BE EXECUTED IN CONNECTION WITH THIS NOTE, OR ANY COURSE OF CONDUCT, COURSE OF DEALING, STATEMENTS (WHETHER VERBAL OR WRITTEN) OR ACTIONS OF ANY PARTY WITH RESPECT HERETO. THIS PROVISION IS A MATERIAL INDUCEMENT TO APA TO ACCEPT THIS NOTE. 3 BORROWER AND APA AGREE THAT THEY SHALL NOT HAVE A REMEDY OF PUNITIVE OR EXEMPLARY DAMAGES AGAINST THE OTHER IN ANY DISPUTE AND HEREBY WAIVE ANY RIGHT OR CLAIM TO PUNITIVE OR EXEMPLARY DAMAGES THEY HAVE NOW OR WHICH MAY ARISE IN THE FUTURE IN CONNECTION WITH ANY DISPUTE WHETHER THE DISPUTE IS RESOLVED BY ARBITRATION OR JUDICIALLY. IN WITNESS WHEREOF, Borrower, on the day and year first above written, has caused this Note to be executed. By: /s/ Daniel R. O'Neal ------------------------------ Daniel R. O'Neal 4