<Page>

                                                                     Exhibit 4.1

                                AMENDMENT NO. 1 TO
                           LOAN AND SECURITY AGREEMENT
                           ---------------------------

     AMENDMENT NO. 1, dated as of May 28, 2002 by and among Congress Financial
Corporation, a Delaware corporation ("Lender"), Anvil Knitwear, Inc., a Delaware
corporation ("Borrower"), Anvil Holdings, Inc., a Delaware corporation
("Holdings") and Cottontops, Inc., a Delaware corporation ("Cottontops";
together with Holdings, each individually a "Guarantor" and collectively
"Guarantors").

                               W I T N E S S E T H

WHEREAS, Borrower and Guarantors have entered into financing arrangements with
Lender pursuant to which Lender may make loans and provide other financial
accommodations to Borrower as set forth in the Loan and Security Agreement,
dated March 11, 1999, by and among Borrower, Guarantors and Lender, as amended
(as the same now exists and is amended hereby and may hereafter be further
amended, modified, supplemented, extended, renewed, restated or replaced, the
"Loan Agreement") and the other agreements, documents and instruments referred
to therein or at any time executed and/or delivered in connection therewith or
related thereto, including this Amendment (all of the foregoing, together with
the Loan Agreement, as the same now exist or may hereafter be amended, modified,
supplemented, extended, renewed, restated or replaced, being collectively
referred to herein as the "Financing Agreements");

     WHEREAS, Borrower and Guarantors have requested that Lender agree to
certain amendments to the Loan Agreement and Lender is willing to agree to such
amendments, subject to the terms and conditions contained herein; and

     WHEREAS, by this Amendment, Borrower, Guarantors and Lender intend to
evidence such amendments.

     NOW, THEREFORE, in consideration of the foregoing, and the agreements and
covenants contained herein, the parties hereto agree as follows:

     1. Definitions.

     1.1 Interpretation. For purposes of this Amendment, unless otherwise
     defined herein, all terms used herein, including, but not limited to, those
     terms used and/or defined in the recitals above, shall have the respective
     meanings assigned to such terms in the Loan Agreement.

     2. Maximum Credit. Section 1.57 of the Loan Agreement is hereby deleted in
     its entirety and the following substituted therefor:

     "1.57 "Maximum Credit" shall mean the amount of $50,000,000."

     3. Revolving Loan Limit. Section 1.78 of the Loan Agreement is hereby
     deleted in its entirety and the following substituted therefor:

     "1.78 "Revolving Loan Limit" shall mean $38,275,000."

     4. General.

     4.1 EFFECT OF THIS AMENDMENT. Except as modified pursuant hereto, no other
     changes or modifications to the Financing Agreements are intended or
     implied and in all other respects the Financing Agreements are hereby
     specifically ratified, restated and confirmed by all parties hereto as of
     the date hereof. To the extent of conflict between the terms of this
     Amendment and the Financing Agreements, the terms of this Amendment shall
     control.


<Page>

     4.2 FURTHER ASSURANCES. The parties hereto shall execute and deliver such
     additional documents and take such additional action as may be necessary to
     effectuate the provisions and purposes of this Amendment.

     4.3 GOVERNING LAW. The rights and obligations hereunder of each of the
     parties hereto shall be governed by and interpreted and determined in
     accordance with the internal laws of the State of New York (without giving
     effect to principles of conflict of laws).

     4.4 BINDING EFFECT. This Amendment is binding upon and shall inure to the
     benefit of Lender, Borrower and Guarantors and their respective successors
     and assigns.


     4.5 COUNTERPARTS. This Amendment may be executed in one or more
     counterparts, each of which when so executed shall be deemed to be an
     original but all of which when taken together shall constitute one and the
     same instrument. In making proof of this Amendment, it shall not be
     necessary to produce or account for more than one counterpart thereof
     signed by each of the parties hereto.

     IN WITNESS WHEREOF, Lender, Borrower and Guarantors have caused this
Amendment to be duly executed as of the day and year first above written.


                                CONGRESS FINANCIAL CORPORATION

                                By:  /s/ Terese Gatto
                                   ------------------
                                Title: Vice President



                                ANVIL KNITWEAR, INC.

                                By:   /s/ Jacob Hollander
                                   ----------------------
                                Title:  Executive Vice President



                                ANVIL HOLDINGS, INC.

                                By:   /s/ Jacob Hollander
                                   ----------------------
                                Title: Vice President



                                COTTONTOPS, INC.

                                By:  /s/ Jacob Hollander
                                   ---------------------
                                Title:  Vice President