[BlueOnyx:13398] Re: attempt to access beyond end of device

Mon Jul 22 17:03:56 -05 2013

On Sat, 20 Jul 2013 18:02:28 -0500, BO Mail wrote
> Hi,
> Wondering if anyone has seen this error and/or knows if/how its 
> fixable? This just started a couple weeks ago and sometimes it will 
> go a day without even seeing this error message in /var/messages. 
> Other times we will see it off and on 4 or 5 times a day.
> 
> attempt to access beyond end of device
> md4: rw=0, want=8942984760, limit=287193728
> 
> saw this message once and only once in the past day:
> 
> kernel: md4: rw=1, want=3302887440, limit=287193728
> kernel: Buffer I/O error on device md4, logical block 412860929
> kernel: lost page write due to I/O error on md4
> 
> Other than these messages everything seems fine on the server, no 
> other error messages and everything running normally. It has a raid1 
> mirror with 2 160gb seagate HD's and has been running along smoothly 
> for a few years. We did smartctl -a checks and they say they passed. 
> Did have to replace the original sdb drive 2 or 3 yrs ago as it 
> failed.
> 
> cat /proc/mdstat
> Personalities : [raid1]
> md6 : active raid1 sdb1[0] sda1[1]
>       104320 blocks [2/2] [UU]
> 
> md3 : active raid1 sdb3[0] sda3[1]
>       4192896 blocks [2/2] [UU]
> 
> md5 : active raid1 sdb5[0] sda5[1]
>       1052160 blocks [2/2] [UU]
> 
> md2 : active raid1 sdb6[0] sda6[1]
>       1052160 blocks [2/2] [UU]
> 
> md4 : active raid1 sdb7[0] sda7[1]
>       143596864 blocks [2/2] [UU]
> 
> md1 : active raid1 sdb2[0] sda2[1]
>       6289344 blocks [2/2] [UU]
> 
> unused devices: <none>
> 
> kernel: md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
> kernel: ata1: SATA max UDMA/133 cmd 0xE200 ctl 0xE302 bmdma 0xE600 
> irq 177 kernel: ata2: SATA max UDMA/133 cmd 0xE400 ctl 0xE502 bmdma 
> 0xE608 irq 177 kernel: md: raid1 personality registered as nr 3 
> kernel: md: Autodetecting RAID arrays. kernel: md: autorun ... 
> kernel: md: considering sdb7 ... kernel: md:  adding sdb7 ... 
> kernel: md: sdb6 has different UUID to sdb7 kernel: md: sdb5 has 
> different UUID to sdb7 kernel: md: sdb3 has different UUID to sdb7 
> kernel: md: sdb2 has different UUID to sdb7 kernel: md: sdb1 has 
> different UUID to sdb7 kernel: md:  adding sda7 ... kernel: md: sda6 
> has different UUID to sdb7 kernel: md: sda5 has different UUID to 
> sdb7 kernel: md: sda3 has different UUID to sdb7 kernel: md: sda2 
> has different UUID to sdb7 kernel: md: sda1 has different UUID to 
> sdb7 kernel: md: created md4 kernel: md: bind<sda7> kernel: md: bind<sdb7>
> kernel: md: running: <sdb7><sda7>
> kernel: raid1: raid set md4 active with 2 out of 2 mirrors
> kernel: md: considering sdb6 ...
> kernel: md:  adding sdb6 ...
> kernel: md: sdb5 has different UUID to sdb6
> kernel: md: sdb3 has different UUID to sdb6
> kernel: md: sdb2 has different UUID to sdb6
> kernel: md: sdb1 has different UUID to sdb6
> kernel: md:  adding sda6 ...
> kernel: md: sda5 has different UUID to sdb6
> kernel: md: sda3 has different UUID to sdb6
> vwh kernel: md: sda2 has different UUID to sdb6
> kernel: md: sda1 has different UUID to sdb6
> kernel: md: created md2
> kernel: md: bind<sda6>
> kernel: md: bind<sdb6>
> kernel: md: running: <sdb6><sda6>
> kernel: raid1: raid set md2 active with 2 out of 2 mirrors
> kernel: md: considering sdb5 ...
> kernel: md:  adding sdb5 ...
> kernel: md: sdb3 has different UUID to sdb5
> kernel: md: sdb2 has different UUID to sdb5
> kernel: md: sdb1 has different UUID to sdb5
> kernel: md:  adding sda5 ...
> kernel: md: sda3 has different UUID to sdb5
> kernel: md: sda2 has different UUID to sdb5
> kernel: md: sda1 has different UUID to sdb5
> kernel: md: created md5
> kernel: md: bind<sda5>
> kernel: md: bind<sdb5>
> kernel: md: running: <sdb5><sda5>
> kernel: raid1: raid set md5 active with 2 out of 2 mirrors
> kernel: md: considering sdb3 ...
> kernel: md:  adding sdb3 ...
> kernel: md: sdb2 has different UUID to sdb3
> kernel: md: sdb1 has different UUID to sdb3
> kernel: md:  adding sda3 ...
> kernel: md: sda2 has different UUID to sdb3
> kernel: md: sda1 has different UUID to sdb3
> kernel: md: created md3
> kernel: md: bind<sda3>
> kernel: md: bind<sdb3>
> kernel: md: running: <sdb3><sda3>
> kernel: raid1: raid set md3 active with 2 out of 2 mirrors
> kernel: md: considering sdb2 ...
> kernel: md:  adding sdb2 ...
> kernel: md: sdb1 has different UUID to sdb2
> kernel: md:  adding sda2 ...
> kernel: md: sda1 has different UUID to sdb2
> kernel: md: created md1
> kernel: md: bind<sda2>
> kernel: md: bind<sdb2>
> kernel: md: running: <sdb2><sda2>
> kernel: raid1: raid set md1 active with 2 out of 2 mirrors
> kernel: md: considering sdb1 ...
> kernel: md:  adding sdb1 ...
> kernel: md:  adding sda1 ...
> kernel: md: created md6
> kernel: md: bind<sda1>
> kernel: md: bind<sdb1>
> kernel: md: running: <sdb1><sda1>
> kernel: raid1: raid set md6 active with 2 out of 2 mirrors
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: md: Autodetecting RAID arrays.
> kernel: md: autorun ...
> kernel: md: ... autorun DONE.
> kernel: EXT3 FS on md1, internal journal
> kernel: EXT3 FS on md6, internal journal
> kernel: EXT3 FS on md4, internal journal
> kernel: EXT3 FS on md2, internal journal
> kernel: EXT3 FS on md3, internal journal
> kernel: Adding 1052152k swap on /dev/md5.  Priority:-1 extents:1
> 
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md1               6190592    940700   4935428  17% /
> /dev/md6                101018     15385     80417  17% /boot
> none                    509160         0    509160   0% /dev/shm
> /dev/md4             141342888   2106040 132057008   2% /home
> /dev/md2               1035596     78316    904672   8% /tmp
> /dev/md3               4127040    490928   3426468  13% /var
> 
> Oh and this is a BQ server, our last server holdout from switching to
> BlueOnyx. They say they have some "custom" stuff that won't work yet 
> on newer OS, but they are working on it so we can switch over to BO 
> eventually. If anyone has any ideas we would greatly appreciate them.
> 
> Thanks,
> Tom
> 
> _______________________________________________
> Blueonyx mailing list
> Blueonyx at mail.blueonyx.it
> http://mail.blueonyx.it/mailman/listinfo/blueonyx

Well, we found many instances for this error message from searching google,
but most of them were because of earlier versions kernel bugs. We also found
one that sounded like it could be similar and was because of file system
errors and fixed by running e2fsck in bad blocks mode.. does that sound like a
possibility for this error? Go into single user mode in centos and run esfsck
on md4?

http://pith.org/notes/2005/07/23/how-i-fixed-my-raid-1-partition-size-error/

Thanks,
TomW