NOTICE OF GUARANTEED DELIVERY
 
                                 FOR TENDER OF
                            ANY AND ALL OUTSTANDING
                   8 3/8% SENIOR SUBORDINATED NOTES DUE 2008
                      (LIQUIDATION AMOUNT $1,000 PER NOTE)
                                       OF
                          UNITED STATIONERS SUPPLY CO.
      PURSUANT TO THE EXCHANGE OFFER AND PROSPECTUS DATED          , 1998
- --------------------------------------------------------------------------------
THE EXCHANGE OFFER AND WITHDRAWAL RIGHTS WILL EXPIRE AT 5:00 P.M., NEW YORK CITY
   TIME, ON             ,           , 1998 (THE "EXPIRATION DATE"), UNLESS THE
   EXCHANGE OFFER IS EXTENDED BY THE COMPANY.
- --------------------------------------------------------------------------------
 
    This Notice of Guaranteed Delivery, or one substantially equivalent to this
form, must be used to accept the Exchange Offer (as defined below) if (i)
certificates for the 8 3/8% Senior Subordinated Notes due 2008 (the "Old
Notes"), of United Stationers Supply Co. (the "Company") are not immediately
available, (ii) Old Notes, the Letter of Transmittal and all other required
documents cannot be delivered to The Bank of New York (the "Exchange Agent") on
or prior to 5:00 P.M. New York City time, on the Expiration Date (as defined in
the Prospectus referred to below) or (iii) the procedures for delivery by
book-entry transfer cannot be completed on a timely basis. This Notice of
Guaranteed Delivery may be delivered by hand, overnight courier or mail, or
transmitted by facsimile transmission, to the Exchange Agent. See "The Exchange
Offer -- Guaranteed Delivery Procedures" in the Prospectus. In addition, in
order to utilize the guaranteed delivery procedure to tender Old Notes pursuant
to the Exchange Offer, a completed, signed and dated Letter of Transmittal
relating to the Old Notes (or facsimile thereof) must also be received by the
Exchange Agent prior to 5:00 P.M. New York City time, on the Expiration Date.
Capitalized terms not defined herein have the meanings assigned to them in the
Prospectus.
 
                 THE EXCHANGE AGENT FOR THE EXCHANGE OFFER IS:
 
                              THE BANK OF NEW YORK
 

                                        
    BY REGISTERED OR CERTIFIED MAIL:            BY HAND OR OVERNIGHT DELIVERY:
          The Bank of New York                       The Bank of New York
           101 Barclays Street                        101 Barclays Street
                Floor 7-E                       Corporate Trust Services Window
        New York, New York 10286                         Ground Level
      Attn: Reorganization Section,                New York, New York 10286
             (212) 815-3428                      Attn: Reorganization Section,
                                                        (212) 815-3428

 
                            FACSIMILE TRANSMISSIONS:
                          (ELIGIBLE INSTITUTIONS ONLY)
                                 (212) 815-6339
 
                               FOR INFORMATION OR
                           CONFIRMATION BY TELEPHONE:
                                 (212) 815-6333
 
    DELIVERY OF THIS NOTICE OF GUARANTEED DELIVERY TO AN ADDRESS OTHER THAN AS
SET FORTH ABOVE OR TRANSMISSION OF THIS NOTICE OF GUARANTEED DELIVERY VIA
FACSIMILE TO A NUMBER OTHER THAN AS SET FORTH ABOVE WILL NOT CONSTITUTE A VALID
DELIVERY.
 
    THIS NOTICE OF GUARANTEED DELIVERY IS NOT TO BE USED TO GUARANTEE
SIGNATURES. IF A SIGNATURE ON A LETTER OF TRANSMITTAL IS REQUIRED TO BE
GUARANTEED BY AN "ELIGIBLE INSTITUTION" UNDER THE INSTRUCTIONS THERETO, SUCH
SIGNATURE GUARANTEE MUST APPEAR IN THE APPLICABLE SPACE PROVIDED IN THE
SIGNATURE BOX ON THE LETTER OF TRANSMITTAL.

Ladies and Gentlemen:
 
    The undersigned hereby tenders to United Stationers Supply, Co. an Illinois
corporation, upon the terms and subject to the conditions set forth in the
Prospectus dated         , 1998 (as the same may be amended or supplemented from
time to time, the "Prospectus"), and the related Letter of Transmittal (which
together constitute the "Exchange Offer"), receipt of which is hereby
acknowledged, the aggregate principal amount of Old Notes set forth below
pursuant to the guaranteed delivery procedures set forth in the Prospectus under
the caption "The Exchange Offer-- Guaranteed Delivery Procedures."
 
- --------------------------------------------------------------------------------
 
  Name(s) of Registered Holder(s): ___________________________________________
 
  Aggregate Principal Amount
  Amount Tendered: $ _______________________________________________________ *
 
  Certificate No.(s)
  (if available): ____________________________________________________________
 
  (Total Principal Amount Represented by
  Old Notes Certificate(s)) __________________________________________________
 
  $ __________________________________________________________________________
 
  If Old Notes will be tendered by book-entry transfer, provide the following
  information:
 
  DTC Account Number: ________________________________________________________
 
  Date: ______________________________________________________________________
 
  ----------------
 
  * Must be in denominations of $1,000 principal amount and any integral
    multiple thereof.
 
- --------------------------------------------------------------------------------
 
All authority herein conferred or agreed to be conferred shall survive the death
or incapacity of the undersigned and every obligation of the undersigned
hereunder shall be binding upon the heirs personal representatives, successors
and assigns of the undersigned.

- --------------------------------------------------------------------------------
 
                                PLEASE SIGN HERE
 

                                             
X
X
           Signature(s) of Owner(s)                          Date
           or Authorized Signatory
 
Area Code and Telephone Number:
 
    Must be signed by the holder(s) of the Old Notes as their name(s) appear(s)
on certificates for Old Notes or on a security position listing, or by
person(s) authorized to become registered holder(s) by endorsement and
documents transmitted with this Notice of Guaranteed Delivery. If signature is
by a trustee, executor, administrator, guardian, attorney-in-fact, officer or
other person acting in a fiduciary or representative capacity, such person must
set forth his or her full title below.
 
                     Please print name(s) and address(es)
 
Name(s):
 
Capacity:
 
Address(es):
 

 
- --------------------------------------------------------------------------------

              THE GUARANTEE ON THE REVERSE SIDE MUST BE COMPLETED
 
                                   GUARANTEE
                    (NOT TO BE USED FOR SIGNATURE GUARANTEE)
 
    The undersigned, a member of or participant in the Securities Transfer
Agents Medallion Program, the New York Stock Exchange Signature Program, the
Stock Exchange Medallion Program or a firm or other entity identified in Rule
17Ad-15 under the Securities Exchange Act of 1934, as amended, as an "eligible
guarantor institution," including (as such terms are defined therein): (i) a
bank; (ii) a broker, dealer, municipal securities broker, municipal securities
dealer, government securities broker, government securities dealer; (iii) a
credit union; (iv) a national securities exchange, registered securities
association or learning agency; or (v) a savings association that is a
participant in a Securities Transfer Association recognized program (each of the
foregoing being referred to as an "Eligible Institution"), hereby guarantees to
deliver to the Exchange Agent, at one of its addresses set forth above, either
the Old Notes tendered hereby in proper form for transfer, or confirmation of
the book-entry transfer of such Old Notes to the Exchange Agent's account at The
Depositary Trust Company ("DTC"), pursuant to the procedures for book-entry
transfer set forth in the Prospectus, and any other documents required by the
Letter of Transmittal within three New York Stock Exchange, Inc. trading days
after the date of execution of this Notice of Guaranteed Delivery.
 
    The undersigned acknowledges that it must deliver the Letter(s) of
Transmittal and the Old Notes tendered hereby to the Exchange Agent within the
time period set forth above and that failure to do so could result in a
financial loss to the undersigned.
 

                                        
- ------------------------------------------------------------------------------------
 
              Name of Firm                           Authorized Signature
 
                 Address                                     Title
 
                Zip Code                            (Please Type or Print)
 
       Area Code and Telephone No.                          Dated:
- ------------------------------------------------------------------------------------

 
NOTE: DO NOT SEND CERTIFICATES FOR OLD NOTES WITH THIS FORM. CERTIFICATES FOR
OLD NOTES SHOULD ONLY BE SENT WITH YOUR LETTER OF TRANSMITTAL.