As filed with the Securities and Exchange Commission on September 14, 2004

                                                     Registration No. __________
- --------------------------------------------------------------------------------

                       SECURITIES AND EXCHANGE COMMISSION
                              Washington, DC 20549
                                 _______________
                                    FORM S-3
                             REGISTRATION STATEMENT
                                      UNDER
                           THE SECURITIES ACT OF 1933
                                ________________
                         CASUAL MALE RETAIL GROUP, INC.
             (Exact name of registrant as specified in its charter)

             Delaware                                         04-2623104
  (State or other jurisdiction of                          (I.R.S. Employer
   incorporation or organization)                        Identification Number)


                555 Turnpike Street, Canton, Massachusetts 02021
                                 (781) 828-9300
    (Address, including zip code, and telephone number, including area code,
                  of registrant's principal executive offices)
                                 _______________

                                Dennis Hernreich
              Executive Vice President, Chief Operating Officer and
                             Chief Financial Officer
                         Casual Male Retail Group, Inc.
                               555 Turnpike Street
                           Canton, Massachusetts 02021
                                 (781) 828-9300
            (Name, address, including zip code, and telephone number,
       including area code, of registrant's principal executive offices)
                                _________________

                                    Copy to:
                              Peter G. Smith, Esq.
                       Kramer Levin Naftalis & Frankel LLP
                                919 Third Avenue
                            New York, New York 10022
                                 (212) 715-9100
                                ________________

Approximate Date of Commencement of Proposed Sale to the Public: From time to
time after the effective date of this Registration Statement.

                                ________________

   If the only securities being registered on this form are being offered
pursuant to dividend or interest reinvestment plans, check the following box.
|_|

   If any of the securities being registered on this form are to be offered on a
delayed or continuous basis pursuant to Rule 415 under the Securities Act of
1933, other than securities offered only in connection with dividend or interest
reinvestment plans, check the following box. |X|

   If this form is filed to register additional securities for an offering
pursuant to Rule 462(b) under the Securities Act, please check the following box
and list the Securities Act registration statement number of the earlier
effective registration statement for the same offering. |_|

   If this form is a post-effective amendment filed pursuant to Rule 462(c)
under the Securities Act, check the following box and list the Securities Act
registration statement number of the earlier effective registration statement
for the same offering. |_|

   If delivery of the prospectus is expected to be made pursuant to Rule 434,
please check the following box. |_|



                         CALCULATION OF REGISTRATION FEE

                                                      Proposed
Title of each class                                    maximum
         of                              Offering      aggregate     Amount of
  securities to be       Amount to        price        offering     registration
     registered        be registered     per unit      price(1)         fee
- --------------------------------------------------------------------------------
Common Stock, par
value $0.01 per share  1,818,504 (2)    $ 5.93 (3)   $ 10,783,729    $ 1,366.30
- --------------------------------------------------------------------------------

(1)   Estimated solely for the purpose of calculating the registration fee
      pursuant to Rule 457(o) of the Securities Act of 1933, as amended (the
      "Securities Act"), and exclusive of any accrued interest.

(2)   Consists of (i) 1,182,400 shares of common stock issued or issuable upon
      the exercise of warrants currently held by the selling securityholders,
      (ii) 430,000 shares of common stock issuable upon the exercise of options
      currently held by the selling securityholders and (iii) 206,104 additional
      shares of common stock currently held by the selling securityholders.

(3)   Estimated, solely for the purpose of calculating the registration fee,
      pursuant to Rule 457(c) promulgated under the Securities Act of 1933, as
      amended (the "Securities Act"), based on the average of the high and low
      sales prices for our common stock reported on the Nasdaq National Market
      on September 7, 2004, which is within five (5) business days prior to the
      date of filing of this Registration Statement.

                                   ___________


The registrant hereby amends this registration statement on such date or dates
as may be necessary to delay its effective date until the registrant shall file
a further amendment which specifically states that this registration statement
shall thereafter become effective in accordance with Section 8(a) of the
Securities Act of 1933 or until the registration statement shall become
effective on such date as the Commission, acting pursuant to said Section 8(a),
may determine.




                 Subject to Completion, dated September 14, 2004

                        1,818,504 Shares of Common Stock

      This prospectus relates to the offer and sale by the selling stockholders
listed on pages 20-21 of up to 1,818,504 shares of common stock, par value $0.01
per share, of Casual Male Retail Group, Inc., which shares consist of (i)
1,182,400 shares of common stock issued or issuable upon the exercise of
warrants currently held by the selling securityholders, (ii) 430,000 shares of
common stock issuable upon the exercise of options currently held by the selling
securityholders and (iii) 206,104 additional shares of common stock currently
held by the selling securityholders.

      We issued through private placements in 2003 $29.56 million principal
amount of 12% senior subordinated notes due 2010. Together with these notes,
which in most cases were issued net of any commission for an aggregate purchase
price equal to 98.4% of the aggregate principal amount, we also issued, through
such private placements, detachable warrants to purchase 1,182,400 million
shares of our common stock at exercise prices ranging from $4.76 to $7.32 per
share. The warrants are exercisable through July 2, 2010.

      Our common stock is quoted on the Nasdaq National Market under the symbol
"CMRG." The last reported sale price of our common stock on September 10, 2004
was $5.56 per share.

      Investing in our securities involves risks that are described in the "Risk
Factors" section beginning on page 4 of this prospectus.

      We will not receive any cash proceeds from the sale of the shares of
common stock offered under this prospectus. We are responsible for the payment
of certain expenses incident to the registration of the securities.

      Neither the Securities and Exchange Commission, any state securities
commission nor any other regulatory body has approved or disapproved of these
securities or determined if this prospectus is truthful or complete. Any
representation to the contrary is a criminal offense.



The information in this prospectus is not complete and may be changed.The
selling securityholders may not sell these securities until the registration
statement filed with the Securities and Exchange Commission is effective. This
prospectus is not an offer to sell these securities and it is not soliciting an
offer to buy these securities in any state where the offer or sale is not
permitted.



                The date of this prospectus is September 14, 2004.



       Important Notice about the Information Presented in this Prospectus

      You should rely only on the information contained or incorporated by
reference in this prospectus. We have not authorized any other person to provide
you with different information. If anyone provides you with different or
inconsistent information, you should not rely on it. For further information,
see the section of this prospectus entitled "Where You Can Find More
Information." We are not making an offer to sell these securities in any
jurisdiction where the offer or sale is not permitted.

      You should assume that the information appearing in this prospectus is
accurate as of the date on the front cover of this prospectus only. Our
business, financial condition, results of operations and prospects may have
changed since that date.




                                TABLE OF CONTENTS

PROSPECTUS SUMMARY...........................................................1
RISK FACTORS.................................................................4
FORWARD LOOKING INFORMATION.................................................11
USE OF PROCEEDS.............................................................12
DESCRIPTION OF CAPITAL STOCK................................................12
CERTAIN UNITED STATES FEDERAL TAX CONSIDERATIONS............................15
SELLING SECURITYHOLDERS.....................................................19
PLAN OF DISTRIBUTION........................................................25
LEGAL MATTERS...............................................................26
EXPERTS.....................................................................26
WHERE YOU CAN FIND MORE INFORMATION.........................................27






                                       i



                               PROSPECTUS SUMMARY

      The following summary may not contain all the information that may be
important to you and is qualified in its entirety by the more detailed
information appearing elsewhere or incorporated by reference in this prospectus.
You should read the entire prospectus, especially the risks set forth under the
heading "Risk Factors," as well as the information incorporated by reference,
before making an investment decision.

      When used in this prospectus, the terms "Casual Male," "the Company,"
"we," "our" and "us" refer to Casual Male Retail Group, Inc. (formerly known as
Designs, Inc.) and our consolidated subsidiaries, unless otherwise specified.
References in this prospectus to years are to our 52-week or 53-week fiscal
year, which ends on the Saturday nearest to January 31. For example, references
to "fiscal 2003" mean our fiscal year ended January 31, 2004.

Our Business

      We are the largest specialty retailer of big and tall men's apparel in the
United States. We operate 489 Casual Male Big & Tall stores, the Casual Male
catalog business, the Casual Male e-commerce website and 48 Levi's(R)/Dockers(R)
Outlet by Designs outlet stores, all of which are located throughout the United
States and Puerto Rico.

Background

      Prior to May 2002, our business primarily consisted of owning and
operating Levi's(R)/Dockers(R) and Candies(R) branded apparel mall and outlet
stores. With limited opportunity to expand our mature Levi's(R)/Dockers(R)
business, we acquired substantially all of the assets of Casual Male Corp. and
certain of its subsidiaries at a bankruptcy court-ordered auction in May 2002.
At the time of the acquisition, Casual Male Corp. was the largest retailer in
the United States of men's clothing in the "big and tall" market. In April 2002,
we entered into a joint venture with Ecko Complex, LLC, a leading design-driven
lifestyle brand targeting young men and women, to open and operate Ecko
Unltd.(R) branded outlet stores.

      Following our acquisition of Casual Male, we re-evaluated our strategic
initiatives. In light of the significant opportunity to grow the Casual Male
business and the continued significant deterioration in our Levi's(R)/Dockers(R)
business, we announced that we would downsize and eventually exit the
Levi's(R)/Dockers(R) business. We also announced that we would exit the
Candies(R) outlet business, which we did by the end of fiscal 2002. Then in the
second quarter of fiscal 2004, we sold our 50.5% interest in the Ecko joint
venture to Ecko Complex, LLC. These decisions enabled management to focus our
resources and energies primarily on growing our Casual Male business.

Pending Acquisition of Rochester Big & Tall

      On August 18, 2004, we signed an asset purchase agreement to acquire
substantially all of the assets of Rochester Big & Tall Clothing, Inc., a
privately held company headquartered in San Francisco, California ("Rochester").
The purchase price is $15 million in cash plus the assumption of bank and
subordinated debt of approximately $5 million, in addition to the assumption of
identified operating liabilities such as accounts payable and accrued
liabilities. There is a potential payment of an additional $4 million, to be
paid over a three-year period subject to an earn-out provision. The transaction
is subject to our due diligence and other customary closing conditions. Assuming
satisfaction of all closing conditions, we expect that the acquisition will
close on or about October 31, 2004. We expect to secure a $7.5 million term loan
which, together with borrowings from our $90 million credit facility, will be
used to finance the acquisition. We do not expect this acquisition to have any
significant impact on our liquidity position.



      Since the Casual Male acquisition, we have operated in two segments: our
"Casual Male business" and our "Other Branded Apparel businesses."

Casual Male Business

      Our Casual Male business is a multi-channel retailer that offers our
customers multiple ways to purchase men's big and tall apparel. The business
consists of:

      o     420 Casual Male Big & Tall full-price retail stores, located
            primarily in strip centers, power centers and stand-alone locations;

      o     69 Casual Male Big & tall outlet stores, located in outlet shopping
            centers;

      o     the "Casual Male Big & Tall" catalog, of which we issued 17 editions
            in fiscal 2003; and

      o     our e-commerce business, which includes the www.casualmale.com
            website and a Casual Male Big & Tall apparel shop on the Amazon.com
            website.

      Since our acquisition of the Casual Male business in May 2002, in order to
revitalize the Casual Male brand and increase our share of the $5.3 billion
men's big and tall apparel market, we have implemented several merchandising
strategies, including:

      o     changing our store format to merchandise our stores by lifestyle,
            such as traditional, active and contemporary;

      o     targeting the fast-growing "under 30" big and tall customer segment;

      o     announcing an exclusive marketing agreement with George Foreman, as
            well as launching an exclusive line of clothing with the George
            Foreman brand which commenced in Spring 2004;

      o     launching a custom fit program, by which customers can purchase
            certain styles of clothing that are custom made to specific fit
            requirements;

      o     broadening our merchandise offerings by introducing selected branded
            products, including professional sports apparel; and

      o     introducing new systems infrastructure to improve inventory
            management, maintain in-stock positions in critical sizes for all
            stores and tailor lifestyle merchandise assortments to the
            demographic characteristics of each store.

Other Branded Apparel Businesses

      Ecko Unltd.(R)

      From March 2002 through July 30, 2004, we operated a joint venture with
Ecko Complex, LLC ("Ecko"), under which we owned and managed retail outlet
stores bearing the name Ecko Unltd.(R) and featuring Ecko(R) branded
merchandise. On July 30, 2004, the Company sold to Ecko its 50.5% interest in
the joint venture for a purchase price of $800,000 in cash and a secured
promissory note in the principal amount of $6.2 million. In addition, the
Company will also continue to receive fees based on a percentage of sales for
providing transitional services to the joint venture related to its operating
and accounting systems, as needed until June 30, 2005. The above transaction
resulted in a gain of approximately $3.1


                                       2



million, which was included in our results of operations for the three and six
months ended July 31, 2004. Furthermore, pursuant to a mark-down allowance
agreement entered into on July 30, 2004, Ecko also executed and delivered an
additional secured promissory note for $1.0 million as a markdown allowance with
respect to purchases of certain goods made by the Company from Ecko.

      Levi's(R)/Dockers(R) Outlets

      We currently operate 48 Levi's(R)/Dockers(R) outlet stores, 17 of which we
expect to close by the end of fiscal 2004. We expect that the remaining
Levi's(R)/Dockers(R) stores will either be closed on or before the end of their
respective lease terms.

Corporate Information

      We were originally incorporated in Delaware as Designs, Inc. in 1976. We
changed our name to Casual Male Retail Group, Inc. shortly following our
acquisition of substantially all of the assets of Casual Male Corp. and certain
of its subsidiaries in May 2002. The address of our principal corporate and
executive office is 555 Turnpike Street, Canton, Massachusetts 02021. The
telephone number at our headquarters is (781) 828-9300. Our corporate website is
located at http://www.cmrginc.com. The information contained on our website is
not part of this prospectus.



                                       3



                                  Risk Factors

      You should carefully consider the following risks, as well as the other
information contained in this prospectus or incorporated by reference in this
prospectus, before investing in shares of our common stock. If any of the
following risks actually occurs, our business, financial condition, operating
results or prospects could be harmed. In that case, the trading price of our
common stock could decline, and you might lose all or part of your investment.

Risks Related to Our Company and Our Industry

Our ability to continue to expand our Casual Male stores may be limited.

      A large part of our growth has resulted from the addition of new Casual
Male stores and the increased sales volume and profitability provided by these
stores. We will continue to depend on adding new stores to increase our sales
volume and profitability. We believe that our ability to increase the number of
Casual Male stores in the United States substantially in excess of the number of
our current stores will be limited due to capital constraints, market conditions
and other factors. When we enter new markets, we must:

      o     obtain suitable store locations in light of the local real estate
            market conditions;

      o     hire and train personnel;

      o     establish distribution methods; and

      o     advertise our brand names and our distinguishing characteristics to
            consumers who may not be familiar with us.

      As a result of these and other factors, opening new stores is often costly
and entails significant risk. We cannot assure you that we will be able to open
and operate new stores on a timely and profitable basis. The costs associated
with opening new stores may negatively affect our results of operations.

We may be unable to successfully predict fashion trends and customer
preferences.

      Customer tastes and fashion trends are volatile and tend to change
rapidly. Our success depends in large part upon our ability to effectively
predict and respond to changing fashion tastes and consumer demands and to
translate market trends to appropriate saleable product offerings. If we are
unable to successfully predict or respond to changing styles or trends and
misjudge the market for products or any new product lines, our sales will be
lower and we may be faced with a substantial amount of unsold inventory or
missed opportunities. In response, we may be forced to rely on additional
markdowns or promotional sales to dispose of excess, slow-moving inventory,
which would decrease our revenues and margins. In addition, the failure to
satisfy consumer demand could have serious longer-term consequences, such as an
adverse impact on our brand value and the loss of market share to our
competitors.

Our business is highly competitive, and competitive factors may reduce our
revenues and profit margins.

      The United States men's big and tall apparel market is highly competitive
with many national and regional department stores, specialty apparel retailers
and discount stores offering a broad range of apparel products similar to the
products that we sell. Besides retail competitors, we consider any manufacturer
of big and tall merchandise operating in outlet malls throughout the United
States to be a competitor. It is also

                                       4


possible that another competitor, either a mass merchant or a men's specialty
store or specialty apparel catalog, could gain market share in men's big and
tall apparel due to more favorable pricing, locations, brand and fashion
assortment and size availability. The presence in the marketplace of various
fashion trends and the limited availability of shelf space also can affect
competition. We may not be able to compete successfully with our competitors in
the future and could lose brand recognition and market share. A significant loss
of market share would adversely affect our revenues and results of operations.

Our sales will decline if we do not successfully advertise and market our
products.

      Our business is directly affected by the success or failure of our
advertising and promotional efforts and those of our vendors. Future advertising
efforts by us, our vendors or our other licensors may be costly and may not
result in increased sales. If we were to undertake a major advertising campaign
without success, then our failure to realize any revenues from our advertising
and promotional expenditures, together with the possible adverse impact on our
brand value and loss of market share, would have a negative impact upon our
revenues. In either case, increased costs and decreased margins, accompanied by
static or decreased revenues, would cause a decline in our results of
operations.

Our success significantly depends on our key personnel and our ability to
attract and retain additional personnel.

      Our future success is dependent on the personal efforts, performance and
abilities of our key management. For example, the loss of the services of David
Levin, our President and Chief Executive Officer, or Dennis Hernreich, our Chief
Operating Officer and Chief Financial Officer, each of whom is an integral part
of our daily operations and is a primary decision maker in all our important
operating matters, could significantly impact our business until adequate
replacements could be identified and put in place. The loss of any of our senior
management may result in:

      o     a loss of organizational focus;

      o     poor operating execution;

      o     an inability to identify and execute potential strategic initiatives
            such as joint venture and licensing opportunities;

      o     an impairment in our ability to identify new store locations; and

      o     an inability to consummate possible acquisitions.

      These adverse results could, among other things, reduce potential
revenues, prevent us from diversifying our product lines and geographic
concentrations, and expose us to downturns in our markets. The loss of members
of our senior management as well as our chairman, Seymour Holtzman, who also has
many years of experience in the capital markets, could negatively impact our
ability to obtain additional debt or equity financing for our operations or to
refinance existing indebtedness, or the terms that might be negotiated for such
financing or refinancing. Those circumstances in turn could ultimately result in
a significant decline in profitability and decline in our financial condition.
The competition is intense for the type of highly skilled individuals with
relevant industry experience that we require and we may not be able to attract
and retain new employees of the caliber needed to achieve our objectives.


                                       5


We need to timely complete the implementation of our information systems and
control procedures.

      We depend heavily upon technology and information systems to control
inventory, sales, markdowns, merchandise on hand and other critical information.
Any significant deficiencies in our management information systems resulting in
less than optimal systems performance could have a negative impact upon our
business. For example, since the information systems provide vital information
with respect to specific merchandise sales at the SKU level, replenishment
requirements to maintain optimum inventory levels, and sell through data from
which markdown requirements are identified to most productively sell through
poor selling SKUs, if that information is not consistently provided on a timely
and accurate basis our sales could be severely impacted, or our gross margins
could be adversely affected.

      We periodically review, improve and, under certain circumstances, replace
our technology and management information systems to provide enhanced support to
all operating areas. If such upgrades and enhancements are not successfully
implemented, then the current systems may not be able to continue to support
adequately our management information requirements. Currently, we are undergoing
a significant effort to replace our existing antiquated legacy systems, as part
of the process of integrating the historical Designs, Inc. and Casual Male
operations. It is critically important to the successful operation of our
business that the implementation of our systems integration process, which
entails the replacement, enhancement, or upgrade of all Casual Male's vital
former information systems, be completed within budget and in a timely manner
without disruption to our daily operations. To implement these initiatives, we
spent approximately $4.1 million in fiscal 2002, $4.0 million in fiscal 2003 and
$1.5 million to date during fiscal 2004. We anticipate that the implementation
will require approximately 12 additional months to complete at a remaining cost
of approximately $10.5 million which includes upgrading our existing
point-of-sale and register systems and implementing a Customer Relationship
Management software system. If we are unable to complete these projects within
budget and on time, our operating results will suffer.

The loss of, or disruption in, our centralized distribution center could
negatively impact our business and operations.

      All merchandise for our Casual Male stores is received into our
centralized distribution center in Canton, Massachusetts, where the inventory is
then processed, sorted and shipped to our stores. We depend in large part on the
orderly operation of this receiving and distribution process, which depends, in
turn, on adherence to shipping schedules and effective management of the
distribution center. Although we believe that our receiving and distribution
process is efficient and well positioned to support our expansion plans, we
cannot assure you that events beyond our control, such as disruptions in
operations due to fire or other catastrophic events, employee matters or
shipping problems, would not result in delays in the delivery of merchandise to
our stores.

      Although we maintain business interruption and property insurance, we
cannot assure you that our insurance will be sufficient, or that insurance
proceeds will be timely paid to us, in the event our distribution center is shut
down for any reason or if we incur higher costs and longer lead times in
connection with a distribution at our distribution center.

We are dependent on third parties for the manufacture of the products we sell.

      We do not own or operate any manufacturing facilities and are therefore
entirely dependent on third parties for the manufacture of the products we sell.
Without adequate supplies of merchandise to sell to our customers in the styles
and fashions demanded by our particular customer base, sales would decrease
materially and our business would suffer. Furthermore, approximately 70% of our
merchandise is branded product made specifically for Casual Male and our
customers. In the event that manufacturers are unable or unwilling to ship
products to us in a timely manner or continue to manufacture products for us, we
would

                                       6


have to rely on other current manufacturing sources or identify and qualify new
manufacturers. We might not be able to identify or qualify such manufacturers
for existing or new products in a timely manner and such manufacturers might not
allocate sufficient capacity to us in order to meet our requirements. Our
inability to secure adequate and timely supplies of product would negatively
impact inventory levels, sales and gross margin rates, and ultimately our
results of operations.

      In addition, even if our current manufacturers continue to manufacture our
products, they may not maintain adequate controls with respect to product
specifications and quality and may not continue to produce products that are
consistent with our standards. If we are forced to rely on products of inferior
quality, then our brand recognition and customer satisfaction would be likely to
suffer. These manufacturers may also increase the cost to us of the products we
purchase from them. If our suppliers increase our costs, our margins may be
adversely affected.

      Should we experience significant unanticipated demand, we will be required
to significantly expand our access to manufacturing, both from current and new
manufacturing sources. If such additional manufacturing capacity is not
available on terms as favorable as those obtained from current sources, then our
revenues or margins, or both, will suffer.

      In addition, a significant portion of our merchandise is directly imported
from other countries, and U.S. domestic suppliers who source their goods from
other countries supply most of our remaining merchandise. If imported goods
become difficult or impossible to bring into the United States, due to tariffs,
embargoes or other reasons and if we cannot obtain such merchandise from other
sources at similar costs, then our sales, gross margins and profit margins would
significantly decline. Furthermore, in the event that commercial transportation
is curtailed or substantially delayed, we may not be able to maintain adequate
inventory levels of important merchandise on a consistent basis, which would
negatively impact our sales and potentially erode the confidence of our customer
base, leading to further loss of sales and an adverse impact on our results of
operations.

      In extreme circumstances, it may be necessary to close less productive
stores so as to consolidate important merchandise categories into our most
productive stores, which would severely impact our results of operations and
cash flow.

Exiting our Levi's(R)/Dockers(R) business may subject us to significant costs
and divert resources.

      In light of the continued significant deterioration in our
Levi's(R)/Dockers(R) operations, we announced that we would downsize and
eventually exit this business. In connection with this restructuring, we have
incurred and will need to continue to incur significant exit costs associated
with the termination of leases, liquidation of inventory and various employee
matters. In addition, the restructuring of this business may divert managerial
and other resources from our core businesses and may subject us to litigation.
We have recorded restructuring charges totaling $41.3 million to date in
connection with the restructuring of our Levi's(R)/Dockers(R) business, and
expect to record additional restructuring charges as we complete this
initiative. These charges have reduced and will continue to reduce our net
income, and if future charges exceed our expectations, our stock price may be
adversely affected.

Our results of operations will be adversely affected if our George Foreman line
of apparel is unsuccessful.

      We have entered into an exclusive endorsement and licensing arrangement
for a men's apparel line with George Foreman, the well-known boxing personality.
Under the terms of this arrangement, we are obligated to make significant
payments to Mr. Foreman regardless of the success of the product line, and we
intend to incur significant marketing costs in connection with the promotion of
this product line. As a result

                                       7


of these expenditures, if sales from this product line do not meet our
expectations our results of operations will be adversely affected. Furthermore,
we are subject to risks associated with having our brand identified with a
celebrity personality. If our customers do not care for Mr. Foreman, or if this
product line is not successful, our brand value will suffer.

The loss of any of our key trademarks or licenses could adversely affect demand
for our products.

      We own and use a number of trademarks and operate under several trademark
license agreements. We believe that these trademarks have significant value and
are instrumental in our ability to create and sustain demand for and to market
our products. We cannot assure you that these trademarks and licensing
agreements will remain in effect and enforceable or that any license agreements,
upon expiration, can be renewed on acceptable terms or at all. In addition, any
future disputes concerning these trademarks and licenses may cause us to incur
significant litigation costs or force us to suspend use of the disputed
trademarks.

Our business is seasonal and is affected by general economic conditions.

      Like most other retail businesses, our business is seasonal. Historically,
over 30% of our net sales have been made and approximately 70% of our operating
income has been generated during November, December and January. Like other
retail businesses, our operations may be negatively affected by local, regional
or national economic conditions, such as levels of disposable consumer income,
consumer debt, interest rates and consumer confidence. Any economic downturn
might cause consumers to reduce their spending, which could negatively affect
our sales. A sustained economic downturn would likely have an adverse affect on
our results of operations.

Acts of terrorism could negatively impact our operating results and financial
condition.

      The continued threat of terrorism and heightened security measures in
response to an act of terrorism may disrupt commerce and undermine consumer
confidence which could negatively impact our sales by causing consumer spending
to decline. Furthermore, an act of terrorism or war, or the threat thereof,
could negatively impact our business by interfering with our ability to obtain
merchandise from vendors or substitute suppliers at similar costs in a timely
manner.

Our cost savings and expense reductions resulting from the acquisition of Casual
Male may be less than anticipated.

      We anticipate significant, continued cost savings following our May 2002
acquisition of substantially all the assets of Casual Male, primarily through
headcount reductions, renegotiations of contractual arrangements for supplies
and services associated with the operation for more favorable pricing terms,
elimination of inefficient and costly business processes and costs by
streamlining our management information systems, and economies of scale in
purchasing. It is possible that some of the contemplated headcount reductions
could fail to take place on the scale proposed due to unforeseen or
underestimated needs for the employees in question. It is also possible that the
cost savings associated with achieving purchasing economies fail to materialize
due to unsuccessful negotiations with key vendors. There is also a cost to
realizing the potential savings and these costs could potentially be higher than
originally contemplated in management's projections. In such an instance, the
amount of the cost savings would be offset by the higher costs of realizing the
savings, thereby reducing the overall benefit of the acquisition of Casual Male
and reducing our expected profitability. If there are substantial failures to
achieve these cost savings, cash flow and the servicing of debt related to that
acquisition could also be reduced.


                                       8


We face greater challenges in managing several brands in multiple channels of
distribution.

      Several retailers have had problems executing a corporate strategy aimed
at operating multiple brands in multiple channels. We have expertise in the
outlet channel of distribution, but our acquisition of Casual Male caused us to
conduct operations in the specialty store and internet channels of distribution.
We are now also responsible for all aspects of brand management with respect to
the Casual Male brand, including advertising and promotion, and the servicing
and merchandising of private label and brand merchandise. If the managing of
multiple brands within multiple channels is poorly executed, we will not achieve
our expected level of profitability, and could ultimately be compelled to
eliminate the multiple brand strategy so that the organization may focus on a
single brand strategy.

A reduction in the size of our target market and shifts in customer purchasing
habits will adversely affect our sales.

      As more and more food retailers begin to compete on the basis of providing
more healthy menus, and American popular culture becomes more health conscious,
the size of our target demographic could decrease, resulting in lower sales. In
addition, recent statistics have shown that the overall levels of men's apparel
sales have been decreasing, in part due to a lesser percentage of men's apparel
being bought by women. If this trend continues and we are unable to adjust our
business model to reflect the trend, our results of operations and cash flow
will be impacted.

Risks Related to Our Corporate Structure and Stock

Our stock price has been and may continue to be extremely volatile due to many
factors.

      The market price of our common stock has fluctuated in the past and may
increase or decrease rapidly in the future depending on news announcements and
changes in general market conditions. Between January 31, 2003 and September 1,
2004, the closing price of our common stock ranged from a low of $2.28 per
share to a high of $10.57 per share. The following factors, among others, may
cause significant fluctuations in our stock price:

      o     news announcements regarding quarterly or annual results of
            operations;

      o     monthly comparable store sales;

      o     acquisitions;

      o     competitive developments;

      o     litigation affecting us; or

      o     market views as to the prospects of the retail industry generally.

Rights of our stockholders may be negatively affected if we issue any of the
shares of preferred stock which our Board of Directors has authorized for
issuance.

      We have available for issuance 1,000,000 shares of preferred stock, par
value $.01 per share. Our board of directors is authorized to issue any or all
of this preferred stock, in one or more series, without any further action on
the part of shareholders. The rights of our shareholders may be negatively
affected if we issue a series of preferred stock in the future that has
preference over our common stock with respect to the payment of dividends or
distribution upon our liquidation, dissolution or winding up.


                                       9


State laws and our certificate of incorporation may inhibit potential
acquisition bids that could be beneficial to our stockholders.

      We are subject to certain provisions of Delaware law which could delay or
make more difficult a merger, tender offer or proxy contest involving us. In
particular, Section 203 of the Delaware General Corporation Law prohibits a
Delaware corporation from engaging in certain business combinations with any
interested stockholder for a period of three years unless specific conditions
are met. In addition, certain provisions of Delaware law could have the effect
of delaying, deferring or preventing a change in control of us, including,
without limitation, discouraging a proxy contest or making more difficult the
acquisition of a substantial block of our common stock. The provisions could
also limit the price that investors might be willing to pay in the future for
shares of our common stock. For additional information, see the section of this
prospectus entitled "Description of Capital Stock -- Antitakeover Effects of
Provisions of Our Certificate of Incorporation, Our By-Laws and Delaware Law."

Conversion of our 5% Senior Subordinated Notes could result in dilution to
holders of our common stock.

      In the fourth quarter of fiscal 2003, we sold in a private transaction
$100 million principal amount of convertible senior subordinated notes due 2024
(the "Convertible Notes"). If the holders of the Convertible Notes convert such
notes, we would be required to issue to such holders approximately 9.39 million
additional shares of common stock, which would result in dilution to holders of
our common stock. Additionally, the 9.39 million additional shares of common
stock would not have an impact on historical earnings per share of the Company
because the conversion of such shares would have been antidilutive.




                                       10


                           FORWARD LOOKING INFORMATION

      Certain statements contained in this prospectus constitute
"forward-looking statements" within the meaning of the United States Private
Securities Litigation Reform Act of 1995. In some cases, forward-looking
statements can be identified by the use of forward-looking terminology such as
"may," "will," "estimate," "intend," "continue," "believe," "expect" or
"anticipate" or the negatives thereof, variations thereon or similar
terminology. The forward-looking statements contained in this prospectus are
generally located in the material set forth under the headings "Prospectus
Summary" and "Risk Factors," but may be found in other locations as well. These
forward-looking statements generally relate to plans and objectives for future
operations and are based upon management's reasonable estimates of future
results or trends. Although we believe that the plans and objectives reflected
in or suggested by such forward-looking statements are reasonable, such plans or
objectives may not be achieved. Actual results may differ from projected results
due, but not limited, to unforeseen developments, including developments
relating to the following:

      o     overall economic and business conditions;

      o     competitive factors in the industries in which we conduct our
            business;

      o     changes in governmental regulation;

      o     the demand for our goods and services;

      o     the fact that our customers may cancel orders they have placed with
            us, in whole or in part, without advance notice;

      o     changes in tax requirements, including tax rate changes, new tax
            laws and revised tax law interpretations;

      o     changes in generally accepted accounting principles or
            interpretations of those principles by governmental agencies and
            self-regulatory groups;

      o     developments in and results of litigation;

      o     interest rate fluctuations, foreign currency rate fluctuations and
            other capital market conditions;

      o     economic and political conditions in international markets,
            including governmental changes and restrictions on the ability to
            transfer capital across borders;

      o     changes in the cost of raw materials used in our business;

      o     the timing, impact and other uncertainties of acquisitions that we
            may consider or consummate;

      o     our ability to achieve anticipated synergies and other cost savings
            in connection with such acquisitions; and

      o     the other factors discussed under "Risk Factors" or elsewhere in
            this prospectus.

      You should read this prospectus completely and with the understanding that
actual future results may be materially different from what we expect. All
forward-looking statements contained in this prospectus and all subsequent
written and oral forward-looking statements attributable to us or persons acting
on our behalf are expressly qualified in their entirety by the foregoing
factors. These forward-looking statements speak only as of the date on which
they are made. We disclaim any obligation or undertaking to provide any updates
or revisions to any forward-looking statement to reflect any change in our
expectations or any change in events, conditions or circumstances on which the
forward-looking statement is based.

                                       11


                                 USE OF PROCEEDS

      The selling securityholders will receive all of the net proceeds from the
sales of shares of common stock pursuant to this prospectus, and the Company
will receive none of the proceeds.

                          DESCRIPTION OF CAPITAL STOCK

      We are authorized to issue a total of 76,000,000 shares, consisting of
75,000,000 shares of common stock and 1,000,000 shares of preferred stock. The
following is a summary of some of the rights and privileges pertaining to our
common stock. For a full description of our common stock and our preferred
stock, you should refer to our certificate of incorporation and by-laws.

Our Common Stock

      As of July 31, 2004, there were 34,207,108 shares of our common stock
outstanding. The holders of our common stock are entitled to one vote per share
on all matters to be voted upon by the stockholders. None of our common
stockholders will be entitled to cumulate votes at any election of directors.
Subject to preferences that are applicable to any series of our preferred stock
that may come into existence in the future, the holders of our common stock are
entitled to receive such dividends, if any, as may be declared from time to time
by our board of directors out of legally available funds. In the event of our
liquidation, dissolution or winding up, the holders of our common stock are
entitled to share ratably in all assets remaining after payment of liabilities,
subject to prior rights of any series of our preferred stock that may come into
existence in the future. Holders of our common stock have no preemptive or
conversion rights or other subscription rights. There are no redemption or
sinking fund provisions available to the holders of our common stock.

Our Preferred Stock

      Our board of directors has the authority, without action by the
stockholders, to designate and issue preferred stock in one or more series and
to designate the rights, preferences and privileges of each series, which may be
greater than the rights of the common stock. It is not possible to state the
actual effect of the issuance of any shares of preferred stock upon the rights
of holders of the common stock until the board of directors determines the
specific rights of the holders of such preferred stock. However the effects
might include, among other things:

      o     restricted dividends on the common stock;

      o     diluting the voting power of the common stock;

      o     impairing the liquidation rights of the common stock; or

      o     delaying or preventing a fundamental change in control of our
            company without further action by our stockholders.

Warrants

      As of July 31, 2004, there were outstanding warrants to purchase an
aggregate of up to 3,377,871 shares of our common stock at a weighted average
exercise price of $5.74 per share, including the warrants exercisable to
purchase the common stock offered by this prospectus. These warrants are
currently exercisable in full and expire between April 2007 and July 2010.


                                       12


Options

      As of July 31, 2004, there were an aggregate of 3,067,433 shares of our
common stock subject to outstanding options at a weighted average exercise price
of $5.56 per share under our 1992 Stock Incentive Plan, as amended. In addition,
as of July 31, 2004, there were an aggregate of 1,140,000 shares of our common
stock issued outside of our 1992 Stock Incentive Plan subject to outstanding
options at a weighted average exercise price of $3.40. As of July 31, 2004,
2,700,075 shares were reserved for future issuance upon exercise of options that
may be granted under the 1992 plan.

Registration Rights of Certain Holders

      From time to time, we have issued and may continue to issue shares of
capital stock, warrants, convertible notes or other securities entitled to
registration rights. All previously issued securities entitled to such
registration rights, other than the shares of stock offered under this
prospectus, have been registered pursuant to registration statements on Form S-3
filed in September 2002 and February 2004. Accordingly, all such previously
granted registration rights have been satisfied.

Antitakeover Effects of Provisions of Our Certificate of Incorporation, Our
By-Laws and Delaware Law

      Provisions of our certificate of incorporation and by-laws, as well as
provisions of Delaware law, could make it more difficult for a third party to
acquire us and to remove incumbent officers and directors. These provisions,
summarized below, are expected to discourage certain types of coercive takeover
practices and inadequate takeover bids and to encourage persons seeking to
acquire control of us to first negotiate with us. These provisions could
discourage potential acquisition proposals and could delay or prevent a change
in control. These provisions are also intended to enhance the likelihood of
continuity and stability in the composition of our board of directors and in the
policies formulated by our board of directors and to discourage certain types of
transactions that may involve an actual or threatened change of control. These
provisions are designed to reduce our vulnerability to an unsolicited
acquisition proposal and to discourage certain tactics that may be used in proxy
fights. These provisions could, however, have the effect of discouraging others
from making tender offers for our shares and, as a consequence, they also may
inhibit fluctuations in the market price of our shares that could result from
actual or rumored takeover attempts. These provisions also may have the effect
of preventing changes in our management. We believe that the benefits of
increased protection of our ability to negotiate with the proponent of an
unfriendly or unsolicited proposal to acquire or restructure us outweigh the
disadvantages of discouraging those proposals because, among other things,
negotiation of those proposals could result in an improvement of their terms.

      Preferred Stock. Our board of directors, without stockholder approval, has
the authority under our certificate of incorporation to issue up to 1,000,000
shares of convertible preferred stock with rights superior to the rights of the
holders of our common stock. As a result, preferred stock could be issued
quickly and easily, could hurt the rights of holders of common stock and could
be issued with terms calculated to delay or prevent a change of control or make
removal of management more difficult.

      Stockholder Meetings. Certain provisions of our by-laws may have the
effect of delaying, deferring or preventing a change of control through
limitations on the right of stockholders to call, or determine the agenda for,
special stockholder meetings, including a requirement for advance notification
of stockholder proposals.

      Delaware Antitakeover Law. We are subject to Section 203 of the Delaware
General Corporation Law, which prohibits a publicly held Delaware corporation
from engaging in a business combination with an interested stockholder for a
period of three years after the date of the transaction in which the person

                                       13


becomes an interested stockholder, unless, before that date: (i) the board of
directors of the corporation approves either the business combination or the
transaction that resulted in the stockholder becoming an interested stockholder;
(ii) upon consummation of the transaction that resulted in the stockholder
becoming an interested stockholder, the interested stockholder owns at least 85%
of the outstanding voting stock, excluding shares held by directors, officers
and employee stock plans; or (iii) on or after the consummation date, the
business combination is approved by the board of directors and by the
affirmative vote at an annual or special meeting of stockholders of at least
66 2/3% of the outstanding voting stock that is not owned by the interested
stockholder. For purposes of Section 203, a business combination includes a
merger, asset sale or other transaction resulting in a financial benefit to the
interested stockholder. An interested stockholder is generally a person who,
together with affiliates and associates of that person, (a) owns 15% or more of
the corporation's voting stock or (b) is an affiliate or associate of the
corporation and was the owner of 15% or more of the outstanding voting stock of
the corporation at any time within the prior three years.

Transfer Agent and Registrar

      The transfer agent and registrar for the common stock is American Stock
Transfer and Trust Company. Its address is 59 Maiden Lane, Plaza Level, New
York, NY 10038.

Listing

      Our common stock is quoted on the Nasdaq National Market under the symbol
"CMRG."


                                       14


                CERTAIN UNITED STATES FEDERAL TAX CONSIDERATIONS

      The following discusses the material U.S. federal income tax consequences
to holders, and U.S. estate tax consequences to non-U.S. holders (defined
below), relating to the ownership and disposition of common stock. This
discussion is for general information only and does not address all aspects of
U.S. federal income taxation that may be relevant to you in light of your
personal circumstances. This summary is based on the provisions of the Internal
Revenue Code of 1986, as amended (the "Code"), applicable existing and proposed
U.S. Treasury regulations, and judicial authority and current administrative
rulings and practice, all of which are subject to change, possibly on a
retroactive basis, or to differing interpretation. Except as otherwise noted,
this summary applies only to holders that hold our common stock as a capital
asset within the meaning of Section 1221 of the Code (generally, for
investment). It does not address tax consequences applicable to those U.S.
holders that may be subject to special tax rules, including financial
institutions, regulated investment companies, tax-exempt organizations,
expatriates, persons subject to the alternative minimum tax provisions of the
Code, pension funds, insurance companies, dealers in securities or foreign
currencies, persons that will hold common stock as a position in a hedging
transaction, straddle, conversion transaction or other risk reduction
transaction for tax purposes, persons deemed to sell common stock under the
constructive sale provisions of the Code, persons who hold common stock through
a partnership or other pass through entity, or persons whose functional currency
is not the U.S. dollar (except as disclosed below under "Non-U.S. Holders"). We
have not sought any ruling from the Internal Revenue Service (the "IRS") with
respect to the statements made and the conclusions reached in the following
summary, and there can be no assurance that the IRS or a court will agree with
our statements and conclusions. Moreover, this discussion does not address the
effect of the federal estate and gift tax laws on U.S. holders or the effect of
any applicable state, local or foreign tax laws.

      THE FOLLOWING DISCUSSION OF U.S. FEDERAL TAX CONSIDERATIONS IS FOR GENERAL
INFORMATION ONLY. IT IS NOT TAX ADVICE. HOLDERS SHOULD CONSULT THEIR OWN TAX
ADVISORS WITH RESPECT TO THE APPLICATION OF THE U.S. FEDERAL INCOME TAX AND
ESTATE TAX LAWS TO THEIR PARTICULAR SITUATIONS AS WELL AS ANY TAX CONSEQUENCES
ARISING UNDER THE LAWS OF ANY STATE, LOCAL OR FOREIGN TAXING JURISDICTION OR
UNDER ANY APPLICABLE TAX TREATY.

      For purposes of this discussion, the term U.S. holder means a beneficial
owner of common stock that is for U.S. federal income tax purposes:

      o     a citizen or resident of the U.S.;

      o     a corporation created or organized under the laws of the U.S. or any
            state;

      o     an estate, the income of which is subject to U.S. federal income
            taxation regardless of its source; or

      o     a trust if (a) its administration is subject to the primary
            supervision of a court within the U.S. and one or more U.S. persons
            have authority to control all of its substantial decisions, or (b)
            it has a valid election in effect under applicable Treasury
            regulations to be treated as a U.S. person.

      A non-U.S. holder means a holder of common stock (other than a partnership
or entity treated as such for U.S. federal income tax purposes) that is not a
U.S. holder for U.S. federal income tax purposes. Income earned through a
foreign or domestic partnership or entity treated as such is generally
attributed to its owners. A beneficial owner of common stock that is a
partnership for U.S. federal income tax purposes, and the partners in such a
partnership, should consult

                                       15


their tax advisors about the U.S. federal income tax consequences of holding and
disposing of the common stock.

U.S. Holders

      Dividends

      Distributions received on our common stock will be treated as a dividend,
subject to tax as ordinary income, to the extent of our current and accumulated
earnings and profits as of the end of the year of distribution. For taxable
years beginning after December 31, 2002 and before January 1, 2009, subject to
certain exceptions, dividends received by individual shareholders generally
would be taxed at the same preferential rates that apply to long-term capital
gains. Any excess will be treated as a tax-free return of capital to the extent
of the U.S. holder's adjusted tax basis in the common stock and thereafter as
gain from the sale or exchange of that stock. Subject to applicable rules, U.S.
holders that are corporations may be eligible to claim a deduction equal to a
portion of any distributions received that are treated as dividends. Special
rules may apply to corporate U.S. holders upon the receipt of any "extraordinary
dividends" with respect to the common stock.

      Sale

      Upon the sale, exchange or other taxable disposition of our common stock,
a U.S. holder will recognize capital gain or loss equal to the difference
between (i) the amount of cash and the fair market value of any property
received upon the disposition and (ii) the U.S. holder's adjusted tax basis in
the common stock. Such capital gain or loss will be long-term if the U.S.
holder's holding period is more than one year.

Non-U.S. Holders

      Dividends

      If we make distributions on our common stock, those distributions
generally will be treated as a dividend to the extent of our current and
accumulated earnings and profits as of the end of the year of distribution.
Subject to the discussion below of backup withholding, any such distribution
treated as dividends to a non-U.S. holder generally will be subject to a 30%
U.S. federal withholding tax, unless (i) the dividend is effectively connected
with the conduct of a U.S. trade or business of the non-U.S. holder or a lower
treaty rate applies and (ii) the non-U.S. holder provides us with proper
certification as to the non-U.S. holder's exemption from, or as to the reduced
rate of, withholding on Form W-8ECI or W-8BEN (or appropriate substitute form),
respectively. If the dividend is effectively connected with the conduct of a
U.S. trade or business, it will be subject to the U.S. federal income tax on net
income that applies to U.S. persons generally and, under certain circumstances
with respect to corporate holders, to the branch profits tax, which is generally
imposed at a 30% rate, subject in each case to income tax treaty exceptions.


                                       16


      Sale

      A non-U.S. holder will not be subject to U.S. federal income tax or
withholding tax on gain realized on the sale, exchange, redemption or other
disposition of common stock, unless:

      o     in the case of an individual non-U.S. holder, that holder is present
            in the U.S. for 183 days or more in the year of the disposition and
            certain other requirements are met; or

      o     the gain is effectively connected with the conduct of a U.S. trade
            or business of the non-U.S. holder.

      If the gain is effectively connected to the conduct of a U.S. trade or
business, it will be subject to the U.S. federal income tax on net income that
applies to U.S. persons generally and, under certain circumstances with respect
to corporate holders, to the branch profits tax, which is generally imposed at a
30% rate, subject in each case to income tax treaty exceptions.

      Notwithstanding the above, if we are or become a U.S. real property
holding corporation (a "USRPHC"), a non-U.S. holder could be subject to federal
income tax with respect to gain realized on the disposition of shares of common
stock. Amounts withheld, if any, with respect to such gain pursuant to the rules
applicable to dispositions of U.S. real property interests would be creditable
against that non-U.S. holder's U.S. federal income tax liability and could
entitle that non-U.S. holder to a refund upon furnishing required information to
the IRS. In general, we would be a USRPHC if interests in U.S. real estate
comprised most of our assets. We do not believe that we are a USRPHC or will
become a USRPHC in the future.

      United States Federal Estate Tax

      Common stock actually or beneficially held by an individual who is not a
citizen or resident of the U.S., as specifically defined for U.S. federal estate
tax purposes, at the time of death (or who previously transferred such stock
subject to certain retained rights or powers) will be subject to U.S. federal
estate tax unless otherwise provided by an applicable estate tax treaty.

Backup Withholding and Information Reporting

      Non-exempt U.S. holders will be subject to information reporting with
respect to payments of dividends on, and possibly proceeds from the disposition
of, common stock. Non-exempt U.S. holders who are subject to information
reporting and who do not provide appropriate information when requested may be
subject to backup withholding. U.S. holders should consult their tax advisors.

      Payments of dividends on common stock to non-U.S. holders will be subject
to information reporting on Form 1042-S. If the common stock is held by a
non-U.S. holder through a non-U.S., and non-U.S. related, broker or financial
institution, backup withholding generally would not be required. Backup


                                       17


withholding may apply if the common stock is held by a non-U.S. holder through a
U.S., or U.S. related, broker or financial institution and the non-U.S. holder
fails to provide appropriate information. Non-U.S. holders should consult their
tax advisors.

      Any amounts withheld from a payment under the backup withholding rules
will be allowed as a refund or credit against a holder's federal income tax
liability, provided that the required information is furnished to the IRS. Some
holders (including, among others, U.S. corporations) are generally not subject
to information reporting and backup withholding.





                                       18



                             SELLING SECURITYHOLDERS

      The selling securityholders, including their transferees, pledgees or
donees or their successors, may from time to time offer and sell a total of
1,818,504 shares of common stock under this prospectus, which amount consists of
(i) 1,182,400 shares of common stock issued or issuable upon exercise of
detachable warrants issued by us in private placements in 2003, (ii) 430,000
shares of common stock issuable upon exercise of options and (iii) 206,104
additional shares of common stock currently held.

      The following table sets forth certain information, as of June 22, 2004,
about the selling securityholders for which we are registering common stock for
resale to the public. To the best of our knowledge, none of the selling
securityholders has any plan, arrangement, understanding, agreement or
commitment to sell its securities. Within the past three years, the following
persons have held the following positions or offices within Casual Male, or have
had the following material relationship with the Company during such time: (a)
Stephen Duff, who was a director of the Company from May 14, 2002 to February
26, 2004, is the Treasurer of Ninth Floor Corporation, the general partner of
Clark Partners I, L.P., and is also the Chief Investment Officer of The Clark
Estates, Inc., the beneficial holder of approximately 7.1% of the outstanding
common stock of the Company; (b) Seymour Holtzman, the Chairman of the Company's
Board of Directors and the beneficial holder of approximately 14.3% of the
Company's outstanding common stock (principally held by Jewelcor Management,
Inc.), is also the President and Chief Executive Officer, and indirectly, with
his wife, the primary shareholder of Jewelcor Management, Inc., which is also
party to a consulting agreement with the Company; (c) Baron Small Cap Fund is a
series of Baron Asset Fund, which is an affiliate of Baron Capital Group, Inc.,
the beneficial holder of approximately 6.3% of the outstanding common stock of
the Company; (d) Marc Holtzman is the son of Seymour Holtzman, the Chairman of
the Company's Board of Directors; (e) Robert L. Patron, who was a director of
the Company from October 1999 to May 2002, is the president of Business Ventures
International Inc.; (f) Rose Gerszberg, Efrem Gerszberg, Seth Gerszberg, Marc
Ecko, Marci Tapper, Donniel Zinkin, Ephraim and Devora Zinkin and Alberto Verdi
are members of the family that owns the equity interests in, and Seth Gerszberg
is the Chief Executive Officer of, Ecko Complex, LLC, with which the Company
operated a joint venture from March 2002 through July 31, 2004; (g) Frank J.
Husic, a director of the Company since June 30, 2003, is the controlling person
with respect to the Frank J. Husic Rollover IRA and is also the sole shareholder
and general partner of Husic Capital Management, the beneficial holder of
approximately 6.5% of the outstanding common stock of the Company; (h) Alan S.
Bernikow has been a director of the Company since June 30, 2003; (i) each of
Raymond Perlman, Wayne T. Green and Efrem Gerszberg is an employee of an
affiliate of Jewelcor Management, Inc., the beneficial holder of approximately
12.1% of the outstanding common stock of the Company and a party to a consulting
agreement with the Company; (j) George Foreman, who was a director of the
Company from March to April 2004, is the Company's spokesperson; and (k) Diane
and Hugh Unger, the trustees of the Diane E. Unger Inter Vivos Trust are the
parents of Jeff Unger, the Company's Vice President of Investor Relations .

      The information regarding the selling securityholders' beneficial
ownership after the sales made pursuant to this prospectus assumes that all of
the shares of common stock subject to sale pursuant to this prospectus shall
have been sold. Each of the selling securityholders has provided the information
set forth below relating to the number of shares such securityholder currently
beneficially owns. The shares subject to sale pursuant to this prospectus may be
offered from time to time, in whole or in part, by the selling securityholders
or their transferees. Information about the selling securityholders may change
over time. Any changed information given to us by the selling securityholders
will be set forth in prospectus supplements if and when necessary.


                                       19





- -----------------------------------------------------------------------------------------
                                   Number of        Number of   Common Stock Beneficially
                                   Shares of        Shares of     Owned After the Sales
                                  Common Stock       Common     -------------------------
                                   Beneficially       Stock
                                  Owned Before      Subject to    Number**     Percent***
   Selling Securityholders         Any Sale (1)       Sale        --------     ----------
- -----------------------------------------------------------------------------------------
                                                                
Clark Partners I, L.P. (2)       2,446,324       100,000    2,346,324         6.8%
- -----------------------------------------------------------------------------------------
Jewelcor Management, Inc. (3)    4,123,973       286,104    3,837,869         11.1%
- -----------------------------------------------------------------------------------------
Baron Small Cap Fund (4)         2,257,353       100,000    2,157,353         6.0%
- -----------------------------------------------------------------------------------------
Paul R. Mancia DDS P.C. Profit
  Sharing Plan (5)                  84,066        10,000       74,066           *
- -----------------------------------------------------------------------------------------
Marc L. Holtzman (6)               306,300        20,000      286,300           *
- -----------------------------------------------------------------------------------------
Benchmark Partners, L.P. (7)       176,000        26,000      150,000           *
- -----------------------------------------------------------------------------------------
Family Trust created under the
  duly probated Last Will and
  Testament of Samuel L. Lane,
  Deceased (8)                       4,000         4,000         0              0
- -----------------------------------------------------------------------------------------
Prism Partners I, L.P. (9)          20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Prism Partners II Offshore Fund
  (10)                              20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Allen Brill (11)                    44,000         4,000       40,000           0
- -----------------------------------------------------------------------------------------
Diane E. Unger Inter Vivos
  Trust (12)                         2,000         2,000         0              0
- -----------------------------------------------------------------------------------------
Business Ventures International
  Inc. (13)                        110,000        10,000      100,000           *
- -----------------------------------------------------------------------------------------
Rose Gerszberg (14)                 20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Efrem Gerszberg (15)                10,000        10,000         0              0
- -----------------------------------------------------------------------------------------
Seth Gerszberg (16)                 20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Marc Ecko (17)                      20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Marci Tapper (18)                   20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Lawrence Seidman Retirement
  Plan and Trust (19)                4,000         4,000         0              0
- -----------------------------------------------------------------------------------------
Frank J. Husic Rollover IRA (20)    20,000        20,000         0              0
- -----------------------------------------------------------------------------------------
Alan S. Bernikow (21)               26,802         4,000       22,802           *
- -----------------------------------------------------------------------------------------
WEC Partners, L.P. (22)             46,600        10,000       36,600           *
- -----------------------------------------------------------------------------------------
Far West Capital Partners, L.P.
  (23)                             527,639        80,000      447,639
- -----------------------------------------------------------------------------------------
Robert G. Schiro 2001 Trust (24)    97,000        16,000       81,000           *
- -----------------------------------------------------------------------------------------
Alberto Verdi (25)                   6,000         6,000         0              0
- -----------------------------------------------------------------------------------------
Harbour Holdings Ltd. (26)         164,000       164,000         0              0
- -----------------------------------------------------------------------------------------
Strong Special Investment, L.P.
  (27)                             116,000       116,000         0              0
- -----------------------------------------------------------------------------------------
Ephraim Zinkin and Devora
  Zinkin (28)                        2,000         2,000         0              0
- -----------------------------------------------------------------------------------------
Donniel Zinkin (29)                  2,000         2,000         0              0
- -----------------------------------------------------------------------------------------
Raymond Perlman (30)                11,000         4,000        7,000           *
- -----------------------------------------------------------------------------------------
Wayne T. Green (31)                 12,000         2,000       10,000           *
- -----------------------------------------------------------------------------------------
FPA Crescent Fund (32)              60,000        60,000         0              0
- -----------------------------------------------------------------------------------------
Zeke LP (33)                       230,000        80,000      150,000           *
- -----------------------------------------------------------------------------------------
Hourglass Master Fund, Ltd. (34)   120,000       120,000         0              0
- -----------------------------------------------------------------------------------------
George Foreman (35)                215,000       100,000      115,000           *
- -----------------------------------------------------------------------------------------
Seymour Holtzman (36)            4,933,405       330,000    4,603,405         13.3%
- -----------------------------------------------------------------------------------------


- ---------------------------

*Less than 1%.


                                       20


**Assumes that the selling securityholders will sell all of their shares of
common stock subject to sale pursuant to this prospectus. We cannot assure you
that the selling securityholders will sell all or any of their shares of common
stock.

***Percentage ownership is based on 34,199,109 shares of common stock
outstanding as of June 22, 2004, plus securities deemed to be outstanding with
respect to individual stockholders pursuant to Rule 13d-3(d)(1) under the
Exchange Act.

(1)   Beneficial ownership is determined in accordance with the rules of the
      Securities and Exchange Commission and includes generally voting and/or
      investment power with respect to securities. Shares of common stock
      subject to warrants, options or convertible stock currently exercisable or
      convertible, or exercisable or convertible within 60 days of June 22, 2004
      are deemed outstanding for the purpose of computing the percentage
      beneficially owned by the person holding such warrants, options or
      convertible stock but are not deemed outstanding for the purpose of
      computing the percentage beneficially owned by any other person.

(2)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 100,000 shares of common
      stock issuable upon exercise of a warrant. The Board of Directors of Ninth
      Floor Corporation, the general partner of Clark Partners I., L.P.,
      exercises sole voting and dispositive power with respect to the shares
      subject to sale underlying the warrant owned by Clark Partners I., L.P.,
      which warrant was acquired in the ordinary course of business.

(3)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 80,000 shares of common stock
      issuable upon exercise of a warrant and 206,104 shares of common stock
      currently held by Jewelcor Management, Inc. as compensation under a
      consulting agreement with the Company. Seymour Holtzman is the Chairman,
      President and Chief Executive Officer of, and indirectly with his wife the
      primary shareholder of, Jewelcor Management, Inc., and, in such
      capacities, exercises sole voting and dispositive power with respect to
      the shares subject to sale currently held by Jewelcor Management, Inc. and
      underlying the warrant owned by Jewelcor Management, Inc., which warrant
      was acquired in the ordinary course of business.

(4)   The amount shown as beneficially owned before any sale includes 1,507,353
      shares of common stock issuable upon exercise of warrants, and the amount
      shown as subject to sale consists of 100,000 shares of common stock
      issuable upon exercise of a warrant. Clifford Greenberg is the Portfolio
      Manager of Bamco Inc., the investment adviser to Baron Small Cap Fund,
      and, in such capacity, exercises sole voting and dispositive power with
      respect to the shares subject to sale underlying the warrant owned by
      Baron Small Cap Fund, which warrant was acquired in the ordinary course of
      business.

(5)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 10,000 shares of common stock
      issuable upon exercise of a warrant. Paul R. Mancia, the President,
      Secretary and Treasurer of the Paul R. Mancia DDS P.C. Profit Sharing
      Plan, exercises sole voting and dispositive power with respect to the
      shares subject to sale underlying the warrant owned by the Paul R. Mancia
      DDS P.C. Profit Sharing Plan, which warrant was acquired in the ordinary
      course of business.

(6)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 20,000 shares of common stock
      issuable upon exercise of a warrant.

(7)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 26,000 shares of common stock
      issuable upon exercise of a warrant. Richard Whitman and Lorraine DiPaolo,
      the general partners of Benchmark Partners, L.P., exercise shared voting
      and dispositive power with respect to the shares subject to sale
      underlying the warrant owned by Benchmark Partners, L.P., which warrant
      was acquired in the ordinary course of business.

(8)   The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 4,000 shares of common stock
      issuable upon exercise of a warrant.


                                       21


(9)   The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant. Jerald Weintraub is the manager of Weintraub
      Capital Management LLC, the general partner of Prism Partners I, L.P.,
      and, in such capacity, exercises sole voting and dispositive power with
      respect to the shares subject to sale underlying the warrant owned by
      Prism Partners I, L.P., which warrant was acquired in the ordinary course
      of business.

(10)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant. Jerald Weintraub is the manager of Weintraub
      Capital Management LLC, the investment advisor of Prism Partners II
      Offshore Fund, and, in such capacity, exercises sole voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant owned by Prism Partners II Offshore Fund, which warrant was
      acquired in the ordinary course of business.

(11)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 4,000 shares of common stock
      issuable upon exercise of a warrant.

(12)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 2,000 shares of common stock
      issuable upon exercise of a warrant. Diane E. Unger and Hugh Unger are the
      trustees of the Diane E. Unger Inter Vivos Trust and, in such capacity,
      exercise shared voting and dispositive power with respect to the shares
      subject to sale underlying the warrant owned by the Diane E. Unger Inter
      Vivos Trust, which warrant was acquired in the ordinary course of
      business.

(13)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 10,000 shares of common stock
      issuable upon exercise of a warrant. Robert L. Patron, the President of
      Business Ventures International Inc., exercises sole voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant owned by Business Ventures International Inc., which warrant
      was acquired in the ordinary course of business.

(14)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant.

(15)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 10,000 shares of common stock issuable upon
      exercise of a warrant.

(16)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant.

(17)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant.

(18)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 20,000 shares of common stock issuable upon
      exercise of a warrant.

(19)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 4,000 shares of common stock
      issuable upon exercise of a warrant. Lawrence Seidman, the Trustee for
      Lawrence Seidman Retirement Plan and Trust, exercises sole voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant owned by Lawrence Seidman Retirement Plan and Trust, which
      warrant was acquired in the ordinary course of business.

(20)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 20,000 shares of common stock
      issuable upon exercise of a warrant. Frank Husic, the managing partner of
      the Frank J. Husic Rollover IRA, exercises sole voting and dispositive
      power with respect to the shares subject to sale underlying the warrant
      owned by the Frank J. Husic Rollover IRA, which warrant was acquired in
      the ordinary course of business.

(21)  The amount shown as beneficially owned before any sale includes 4,000
      shares of common stock issuable upon exercise of a warrant and 20,000
      shares of common stock issuable upon exercise of options. The amount shown
      as subject to sale consists of, 4,000 shares of common stock issuable upon
      exercise of a warrant.


                                       22


(22)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 10,000 shares of common stock
      issuable upon exercise of a warrant. Warren E. Clifford, the general
      partner of WEC Partners, LP, exercises sole voting and dispositive power
      with respect to the shares subject to sale underlying the warrant owned by
      WEC Partners, LP, which warrant was acquired in the ordinary course of
      business.

(23)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 40,000 shares of common stock
      issuable upon exercise of a warrant. Robert G. Schiro is the General
      Partner of Far West Capital Management, LP, the General Partner of Far
      West Capital Partners, L.P., and, in such capacity, exercises sole voting
      and dispositive power with respect to the shares subject to sale
      underlying the warrant owned by Far West Capital Partners, L.P., which
      warrant was acquired in the ordinary course of business.

(24)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 16,000 shares of common stock
      issuable upon exercise of a warrant. Robert G. Schiro, the Trustee for
      Robert G. Schiro 2001 Trust, exercises sole voting and dispositive power
      with respect to the shares subject to sale underlying the warrant owned by
      Robert G. Schiro 2001 Trust, which warrant was acquired in the ordinary
      course of business.

(25)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 6,000 shares of common stock issuable upon
      exercise of a warrant.

(26)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 164,000 shares of common stock issuable upon
      exercise of a warrant. Charles A. Parquelet is the Portfolio Manager of
      Skylands Capital, LLC, the Investment Advisor to Harbour Holdings Ltd.,
      and, in such capacity, exercises sole voting and dispositive power with
      respect to the shares subject to sale underlying the warrant owned by
      Harbour Holdings Ltd., which warrant was acquired in the ordinary course
      of business.

(27)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 116,000 shares of common stock issuable upon
      exercise of a warrant. Charles A. Parquelet is the Portfolio Manager of
      Skylands Capital, LLC, the Investment Advisor to Strong Special
      Investment, L.P., and, in such capacity, exercises sole voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant owned by Strong Special Investment, L.P., which warrant was
      acquired in the ordinary course of business.

(28)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 2,000 shares of common stock issuable upon
      exercise of a warrant. Ephraim and Devora Zinkin exercise joint voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant jointly owned by them.

(29)  The amounts shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 2,000 shares of common stock
      issuable upon exercise of a warrant.

(30)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 4,000 shares of common stock
      issuable upon exercise of a warrant.

(31)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 2,000 shares of common stock
      issuable upon exercise of a warrant.

(32)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 60,000 shares of common stock issuable upon
      exercise of a warrant. Steve Romick, the President of FPA Crescent Fund
      and the Senior Vice President of First Pacific Advisors, Inc., the
      investment adviser to FPA Crescent Fund, exercises sole voting and
      dispositive power with respect to the shares subject to sale underlying
      the warrant owned by FPA Crescent Fund, which warrant was acquired in the
      ordinary course of business.


                                       23


(33)  The amount shown as beneficially owned before any sale includes, and the
      amount shown as subject to sale consists of, 80,000 shares of common stock
      issuable upon exercise of a warrant. Ed Antoian, the general partner of
      Zeke LP, exercises sole voting and dispositive power with respect to the
      shares subject to sale underlying the warrant owned by Zeke LP, which
      warrant was acquired in the ordinary course of business.

(34)  The amounts shown as beneficially owned before any sale and shown as
      subject to sale consist of 120,000 shares of common stock issuable upon
      exercise of a warrant. John Barton is the Managing Member of Tablerock
      Fund Management, LLC, the Advisor to Hourglass Master Fund, Ltd., and, in
      such capacity, exercises sole voting and dispositive power with respect to
      the shares subject to sale underlying the warrant owned by Hourglass
      Master Fund, Ltd., which warrant was acquired in the ordinary course of
      business.

(35)  The amount shown as beneficially owned before any sale consists of 215,000
      shares of common stock issuable upon exercise of stock options, and the
      amount shown as subject to sale consists of 100,000 shares of common stock
      issuable upon exercise of stock options.

(36)  The amount shown as beneficially owned before any sale includes 4,123,973
      shares owned of record by Jewelcor Management, Inc., of which Mr. Holtzman
      is the Chairman, President and Chief Executive Officer and, indirectly
      with his wife, the primary shareholder. The amount shown as beneficially
      owned before any sale includes, and the amount shown as subject to sale
      consists of, 330,000 shares of common stock issuable upon exercise of
      stock options. The amount shown as beneficially owned before any sale does
      not include 66,667 shares currently exercisable pursuant to stock options
      granted to Mr. Holtzman on July 15, 2004.



                                       24



                              PLAN OF DISTRIBUTION

      The selling securityholders and their successors, which include their
pledgees, donees, partnership distributees and other transferees receiving the
warrants or the common stock from the selling securityholders in non-sale
transfers, may sell the common stock directly to purchasers or through
underwriters, broker-dealers or agents. Underwriters, broker-dealers or agents
may receive compensation in the form of discounts, concessions or commissions
from the selling securityholders or the purchasers. These discounts, concessions
or commissions may be in excess of those customary in the types of transactions
involved.

      The common stock may be sold in one or more transactions at:

      o     fixed prices that may be changed;

      o     prevailing market prices at the time of sale;

      o     prices related to the prevailing market prices;

      o     varying prices determined at the time of sale; or

      o     negotiated prices.

      These sales may be effected in transactions, which may involve cross or
block transactions, in the following manner:

      o     on any national securities exchange or quotation service on which
            the common stock may be listed or quoted at the time of sale;

      o     in the over-the-counter-market;

      o     in transactions otherwise than on these exchanges or services or in
            the over-the-counter market (privately negotiated transactions);

      o     through the writing and exercise of options, whether these options
            are listed on an options exchange or otherwise; or

      o     through any combination of the foregoing.

      Selling securityholders may enter into hedging transactions with
broker-dealers or other financial institutions which may in turn engage in short
sales of the common stock and deliver these securities to close out short
positions. In addition, the selling securityholders may sell the common stock
short and deliver the common stock to close out short positions or loan or
pledge the common stock to broker-dealers that in turn may sell such securities.

      Selling securityholders may sell or transfer their shares of common stock
other than by means of this prospectus. In particular, any securities covered by
this prospectus that qualify for sale pursuant to Rule 144 under the Securities
Act may be sold thereunder, rather than pursuant to this prospectus.

      The aggregate proceeds to the selling securityholders from the sale of the
common stock will be the purchase price of the common stock less any discounts
and commissions. A selling securityholder reserves the right to accept and,
together with its agents, to reject any proposed purchase of common stock to be
made directly or through agents. We will not receive any of the proceeds from
this offering.

      In order to comply with the securities laws of some jurisdictions, if
applicable, the holders of common stock may sell in some jurisdictions through
registered or licensed broker-dealers. In addition, under certain circumstances
in some jurisdictions, the holders of the common stock may be required to
register or qualify the securities for sale or comply with an available
exemption from the registration and qualification requirements.


                                       25


      Our common stock is quoted on the Nasdaq National Market.

      The selling securityholders and any underwriters, broker-dealers or agents
that participate in the sale of the common stock may be deemed to be
"underwriters" within the meaning of the Securities Act. In this case, any
discounts, commissions, concessions or profit they earn on any resale of the
shares of the common stock may be underwriting discounts and commissions under
the Securities Act. In addition, selling securityholders who are deemed to be
"underwriters" within the meaning of the Securities Act will be subject to the
prospectus delivery requirements of the Securities Act and may be subject to
statutory liabilities, including, but not limited to, liabilities under Sections
11, 12 and 17 of the Securities Act.

      The selling securityholders and any other persons participating in the
distribution of the common stock will be subject to applicable provisions of the
Exchange Act and the rules and regulations thereunder. Regulation M of the
Exchange Act may limit the timing of purchases and sales of the common stock by
the selling securityholders and any such other person. In addition, Regulation M
may restrict the ability of any person participating in the distribution to
engage in market-making activities with respect to the particular securities
being distributed for a period of up to five business days prior to the
commencement of the distribution. This may affect the marketability of the
common stock.

      If required, the specific common stock to be sold, the names of the
selling securityholders, the respective purchase prices and public offering
prices, the names of any agent, dealer or underwriter and any applicable
commissions or discounts with respect to a particular offer will be set forth in
an accompanying prospectus supplement or, if appropriate, a post-effective
amendment to the registration statement of which this prospectus is a part.


                                  LEGAL MATTERS

      The validity of the common stock issuable upon exercise of the warrants
issued to the selling securityholders is being passed upon for us by Kramer
Levin Naftalis & Frankel LLP, New York, New York.

                                     EXPERTS

      The consolidated financial statements of Casual Male Retail Group, Inc.
appearing in Casual Male's Annual Report (Form 10-K) for the year ended January
31, 2004 (fiscal 2003), have been audited by Ernst & Young LLP, independent
registered public accounting firm, as set forth in their report thereon included
therein and incorporated herein by reference. Such consolidated financial
statements have been incorporated herein by reference in reliance upon such
report given on the authority of such firm as experts in accounting and
auditing.


                                       26


                       WHERE YOU CAN FIND MORE INFORMATION

      We file annual, quarterly and current reports, proxy statements and other
information with the SEC. You may read and copy materials that we have filed
with the SEC at the SEC's public reference room located at 450 Fifth Street,
N.W., Room 1024, Washington, D.C. 20549. Please call the SEC at 1-800-SEC-0330
for further information on the public reference room. Our SEC filings also are
available to the public on the SEC's web site at www.sec.gov, which contains
reports, proxies and information statements and other information regarding
issuers that file electronically. Access to this information as well as other
information on the Company is also available on the Company's corporate website
at http://www.cmrginc.com and clicking on "Investor Relations."

      This prospectus "incorporates by reference" information that we have filed
with or furnished to the SEC under the Exchange Act, which means that we are
disclosing important information to you by referring you to those documents. Any
statement contained in this prospectus or in any document incorporated or deemed
to be incorporated by reference into this prospectus will be deemed to be
modified or superseded for purposes of this prospectus to the extent that a
statement contained in this prospectus or any subsequently filed document which
also is, or is deemed to be, incorporated by reference into this prospectus
modifies or supersedes that statement. Any statement so modified or superseded
will not be deemed, except as so modified or superseded, to constitute a part of
this prospectus. We incorporate by reference into this prospectus the following
documents that we have previously filed with the SEC and any future filings that
we make with the SEC under Sections 13(a), 13(c), 14 or 15(d) of the Exchange
Act after the date of this prospectus until all of the securities covered by
this prospectus are sold by the selling securityholders:

      o     Our Annual Report on Form 10-K for the fiscal year ended January 31,
            2004;

      o     Our Quarterly Reports on Form 10-Q for the fiscal quarters ended May
            1, 2004 and July 31, 2004;

      o     Our Current Reports on Form 8-K filed on March 25, 2004, April 9,
            2004, April 14, 2004, May 11, 2004, May 20, 2004, June 4, 2004, July
            8, 2004, August 4, 2004, August 11, 2004, August 19, 2004 and
            September 2, 2004 (excluding information deemed "furnished" as
            opposed to "filed" under the rules of the SEC); and

      o     All other reports filed by us pursuant to Section 13(a) or 15(d) of
            the Exchange Act since the end of the fiscal year covered by the
            annual report referred to above.

      You may request a copy of these filings at no cost, by writing or
telephoning us at the following address:

      Casual Male Retail Group, Inc.
      555 Turnpike Street
      Canton, Massachusetts 02021,
      Attn:  Dennis R. Hernreich
      (781) 828-9300


                                       27




                                     PART II

                   INFORMATION NOT REQUIRED IN THE PROSPECTUS

Item 14. Other Expenses Of Issuance And Distribution

      The registrant is paying certain of the expenses related to this offering.
The following table sets forth the approximate amount of fees and expenses
payable by the registrant in connection with this Registration Statement and the
distribution of the shares of the securities being registered hereby. The
selling securityholders will bear all underwriting discounts, commissions or
fees attributable to the sale of the registrable securities.

      SEC registration fee                   $1,366.30
      Legal fees and expenses               $10,000.00
      Accounting fees and expenses          $10,000.00
      Printing and engraving expenses        $5,000.00
      Miscellaneous                            $537.99
                                        ---------------
      Total                                 $27,500.00


Item 15. Indemnification of Directors and Officers

      The General Corporation Law of the State of Delaware, as currently in
effect, permits charter provisions eliminating the liability of directors for
breach of fiduciary duty, except that directors remain liable for (i) any breach
of the directors' duty of loyalty to a company or its stockholders, (ii) acts or
omissions not in good faith or which involve intentional misconduct or a knowing
violation of law, (iii) any payment of a dividend or approval of a stock
repurchase that is illegal under Section 174 of the Delaware General Corporation
Law, or (iv) any transaction from which the directors derived an improper
personal benefit. The registrant's Restated Certificate of Incorporation, as
amended (the "Certificate"), provides that no director of the registrant shall
be personally liable to the registrant or to any of its stockholders for
monetary damages arising out of such director's breach of fiduciary duty, except
to the extent that the elimination or limitation of liability is not permitted
by the Delaware General Corporation Law. The effect of this provision of the
Certificate is that directors cannot be held liable for monetary damages arising
from breaches of their duty of care, unless the breach involves one of the four
exceptions described in the first sentence of this Item 15. The provision does
not prevent stockholders from obtaining injunctive or other equitable relief
against directors, nor does it shield directors from liability under federal or
state securities laws. The Certificate and the registrant's By-Laws further
provide for indemnification of the registrant's directors and officers to the
fullest extent permitted by Section 145 of the Delaware General Corporation Law,
including circumstances in which indemnification is otherwise discretionary.

      Insofar as indemnification for liabilities arising under the Securities
Act of 1933 may be permitted to directors, officers and controlling persons of
the registrant, pursuant to the foregoing provisions, or otherwise, the
registrant has been advised that in the opinion of the Securities and Exchange
Commission such indemnification is against public policy as expressed in the Act
and is, therefore, unenforceable. In the event that a claim for indemnification
against such liabilities (other than the payment by the registrant of expenses
incurred or paid by a director, officer or controlling person of the registrant
in the successful defense of any action, suit or proceeding) is asserted by such
director, officer or controlling person in connection with the securities being
registered, the registrant will, unless in the opinion of its counsel the matter
has been settled by controlling precedent, submit to a court of


                                      II-1



appropriate jurisdiction the question whether such indemnification by it is
against public policy as expressed in the Act and will be governed by the final
adjudication of such issue.

Item 16. Exhibits

 Exhibit No.                         Description
 -----------   -----------------------------------------------------------------

     3.1       Restated Certificate of Incorporation of the Company, as amended
               (included as Exhibit 3.1 to Amendment No. 3 of the Company's
               Registration Statement on Form S-1 (No. 33-13402), and
               incorporated herein by reference).

     3.2       Certificate of Amendment to Restated Certificate of
               Incorporation, as amended, dated June 22, 1993 (included as
               Exhibit 3.2 to the Company's Quarterly Report on Form 10-Q filed
               on June 18, 1996, and incorporated herein by reference).

     3.3       Certificate of Amendment to Restated Certificate of
               Incorporation, as amended, dated August 8, 2002 (included as
               Exhibit 3.3 to the Company's Quarterly Report on Form 10-Q filed
               on September 17, 2002, and incorporated herein by reference).

     4.1       Form of Warrant to Purchase Common Stock.

     4.2       Form of Non-Qualified Stock Option Grant Agreement.

     5.1       Opinion of Kramer Levin Naftalis & Frankel LLP.*

     23.1      Consent of Ernst & Young LLP.

     23.2      Consent of Kramer Levin Naftalis & Frankel LLP (included in
               Exhibit 5.1).

     24.1      Power of Attorney (included on the signature page of this
               Registration Statement).

               * To be filed.


Item 17. Undertakings

      (a) The undersigned registrant hereby undertakes:

         (1) To file, during any period in which offers or sales are being made,
      a post-effective amendment to this registration statement;

            (i) To include any prospectus required by Section 10(a)(3) of the
      Securities Act of 1933;

            (ii) To reflect in the prospectus any facts or events arising after
      the effective date of this registration statement (or the most recent
      post-effective amendment thereof) which, individually or in the aggregate,
      represent a fundamental change in the information set forth in this
      registration statement. Notwithstanding the foregoing, any increase or
      decrease in volume of securities offered (if the total dollar value of
      securities offered would not exceed that which was registered) and any
      deviation from the low or high end of the estimated maximum aggregate
      offering range may be reflected in the form of prospectus filed with the
      Commission pursuant to Rule 424(b) if, in the aggregate, the changes in
      volume and price represent no more than 20 percent change in the maximum
      aggregate offering price set forth in the "Calculation of Registration
      Fee" table in the effective registration statement;

            (iii) To include any material information with respect to the plan
      of distribution not previously disclosed in this registration statement or
      any material change to such information in this registration statement;


                                      II-2


provided, however, that paragraphs (a)(1)(i) and (a)(1)(ii) do not apply if the
information required to be included in a post-effective amendment by those
paragraphs is contained in periodic reports filed by the registrant pursuant to
Section 13 or 15(d) of the Securities Exchange Act of 1934 that are incorporated
by reference in this registration statement.

         (2) That, for the purpose of determining any liability under the
      Securities Act of 1933, each such post-effective amendment shall be deemed
      to be a new registration statement relating to the securities offered
      therein, and the offering of such securities at that time shall be deemed
      to be the initial bona fide offering thereof.

         (3) To remove from registration by means of a post-effective amendment
      any of the securities being registered which remain unsold at the
      termination of the offering.

      (b) The undersigned registrant hereby undertakes that, for purposes of
determining any liability under the Securities Act of 1933, each filing of the
registrant's annual report pursuant to Section 13(a) or 15(d) of the Securities
Exchange Act of 1934 (and, where applicable, each filing of an employee benefit
plan's annual report pursuant to Section 15(d) of the Securities Exchange Act of
1934) that is incorporated by reference in the registration statement shall be
deemed to be a new registration statement relating to the securities offered
therein, and the offering of such securities at that time shall be deemed to be
the initial bona fide offering thereof.





                                      II-3



                                   SIGNATURES

      Pursuant to the requirements of the Securities Act of 1933, the registrant
certifies that it has reasonable grounds to believe that it meets all of the
requirements for filing on Form S-3 and has duly caused this registration
statement to be signed on its behalf by the undersigned, thereunto duly
authorized, in the City of Canton, Commonwealth of Massachusetts, as of the 14th
day of September, 2004.

                                    CASUAL MALE RETAIL GROUP, INC.

                                    By: /s/ Dennis R. Hernreich
                                       ---------------------------------------
                                       Name:  Dennis R. Hernreich
                                       Title: Executive Vice President, Chief
                                              Operating Officer, Chief Financial
                                              Officer and Treasurer (Principal
                                              Financial Officer and Principal
                                              Accounting Officer)


      KNOW ALL PERSONS BY THESE PRESENTS that each person whose signature
appears below each severally constitutes and appoints Dennis R. Hernreich, as
true and lawful attorney-in-fact and agent, with full powers of substitution and
resubstitution, for them in his name, place and stead, in any and all
capacities, to sign any and all amendments (including post-effective amendments)
to this Registration Statement, and to file the same, with all exhibits thereto,
and other documents in connection therewith, with the Securities and Exchange
Commission, granting unto said attorney-in-fact and agent, full power and
authority to do and perform each and every act and thing requisite and necessary
to be done in and about the premises, as fully to all intents and purposes as
they might or could do in person, hereby ratifying and confirming all which said
attorney-in-fact and agent, or his substitute or substitutes, may lawfully do,
or cause to be done by virtue thereof.

      Pursuant to the requirements of the Securities Act of 1933, this
registration statement has been signed by the following persons as of September
14, 2004 in the capacities indicated below.

Signatures

/s/ David A. Levin                    Director, President and Chief Executive
- -------------------------             Officer (Principal Executive Officer)
David A. Levin


/s/ Dennis R. Hernreich               Executive Vice President, Chief Operating
- ------------------------              Officer, Chief Financial Officer and
Dennis R. Hernreich                   Treasurer (Principal Financial Officer and
                                      Principal Accounting Officer)


/s/ Seymour Holtzman                  Chairman of the Board of Directors
- -----------------------
Seymour Holtzman


/s/ James Frain                       Director
- -----------------------
James Frain


                                      II-4




/s/ George T. Porter, Jr.             Director
- -----------------------
George T. Porter, Jr.


/s/ Joseph Pennacchio                 Director
- -----------------------
Joseph Pennacchio


/s/ Alan S. Bernikow                  Director
- -----------------------
Alan S. Bernikow


/s/ Jesse H. Choper                   Director
- -----------------------
Jesse H. Choper


/s/ Frank J. Husic                    Director
- -----------------------
Frank J. Husic




                                      II-5



                                  EXHIBIT INDEX

 Exhibit No.                        Description
 -----------   -----------------------------------------------------------------

3.1            Restated Certificate of Incorporation of the Company, as amended
               (included as Exhibit 3.1 to Amendment No. 3 of the Company's
               Registration Statement on Form S-1 (No. 33-13402), and
               incorporated herein by reference).

3.2            Certificate of Amendment to Restated Certificate of
               Incorporation, as amended, dated June 22, 1993 (included as
               Exhibit 3.2 to the Company's Quarterly Report on Form 10-Q filed
               on June 18, 1996, and incorporated herein by reference).

3.3            Certificate of Amendment to Restated Certificate of
               Incorporation, as amended, dated August 8, 2002 (included as
               Exhibit 3.3 to the Company's Quarterly Report on Form 10-Q filed
               on September 17, 2002, and incorporated herein by reference).

4.1            Form of Warrant to Purchase Common Stock.

4.2            Form of Non-Qualified Stock Option Grant Agreement.

5.1            Opinion of Kramer Levin Naftalis & Frankel LLP.*

23.1           Consent of Ernst & Young LLP.

23.2           Consent of Kramer Levin Naftalis & Frankel LLP (included in
               Exhibit 5.1).

               * To be filed.


                                      II-6