The extent of attentional involvement in Gestalt-like perceptual organization has been the source of much ongoing debate. In particular, the question is whether perceptual organization takes place preattentively and if so, whether this is true for all forms of perceptual organization. In the present experiment, we investigated whether visual grouping processes operate preattentively by examining whether grouping takes place under conditions of “inattention” (Kimchi & Razpurker-Apfeld, 2004; Russell & Driver, 2005). Patients with visuo-spatial neglect, resulting from parietal lobe lesions, performed a demanding change-detection task on a matrix-like stimulus presented to the right of a central fixation point. Unbeknownst to the participants a task-irrelevant matrix of elements, organized perceptually by color similarity into rows/columns or into a coherent connected shape, was presented to the left of the fixation point (i.e., the neglected side). This task-irrelevant left-side distractor either changed or retained its grouping on each trial independently of whether the relevant right-side matrix stayed the same or changed, and therefore was either compatible or incompatible with the target response. We observed that task-irrelevant distractors appearing in the neglected (i.e., left) side of space nonetheless influenced accuracy and response time of change detection of information presented to the right of the fixation point. The magnitude of interference was modulated by strength of perceptual grouping (i.e., rows/columns vs. connected shapes), such that stronger grouping yielded greater interference. These results suggest that visual grouping may arise from perceptual processes that operate preattentively but that this is qualified by strength of organization.