PCI device ordering

[adler-wRvlPVLobi1/31tCrMuHxg at public.gmane.org (Stephen Adler)]

Derek Atkins wrote:
> Dan Ritter <dsr-mzpnVDyJpH4k7aNtvndDlA at public.gmane.org> writes:
>
>   
>> On Mon, Nov 29, 2010 at 03:32:24PM -0500, Stephen Adler wrote:
>>     
>>> Hi blu,
>>>
>>> I'm trying to upgrade a server from rhel5 to rhel6 to find that the 
>>> order by which the kernel "discovers" the attached devices changes. 
>>> Specifically, the mother board has 6 sata ports and I added 4 more using 
>>> a PCI-e card. Under the 2.6.18 kernel of rhel5, the on board 6 channel 
>>> sata controller is listed first and the PCI-e 4 port sata controller 
>>> comes second. When I boot up under 2.6.32 kernel of rhel6, the PCI-e 
>>> controller is seen first and then the on board 6 port controller is 
>>> "seen" second. What this does is jumble up the driver order so that 
>>> /dev/sda under the 2.6.8 kernel is now /dev/sdd under the 2.6.32 kernel. 
>>> Is there anyway to force the kernel to scan the onboard sata controller 
>>> first and then the PCI-e controller second?
>>>       
>> Don't do that.
>>
>> Mount by label or by UUID instead.
>>
>> That way, it doesn't matter what order they're detected in, and
>> you can even swap around cables. (Boot drive detection
>> permitting, of course.)
>>     
>
> Hmm, but how would this affect MD raid devices?  For example,
> /proc/mdstat reports raid devices based on sdX.  Although I guess
> /etc/mdadm.conf does set up devices based on UUID.  So perhaps it's safe
> from device re-ordering as well?
>
> I'm considering adding another SATA card to my machine in order to
> increase the number of drives from 4+DVD to 6+DVD.
>   
The issue isn't so much the order of the /dev/sdX devices, but rather 
the fact that the md devices aren't being started.
The real problem I'm having is that I have 3 raid devices defined. 
/dev/md[0-2]. Only /dev/md1 is assembled when
I run rhel6. Well, what I did was start the installation process. At 
some point, there is a a menu/installation stage which
asks you to define the file systems and how you want to mount them etc. 
At this stage, it seemed to have found all 3
raid devices, but on the most critical one, it didn't recognize the file 
system type, (it should have reported it to be ext3).
The installation process did recognize the other 2. So what I did next 
was to boot up in rescue mode using the rhel6
installation DVD. This way, I boot up using the 2.6.32 kernel and all 
auto detection mechanisms should kick in so
that all 3 raid devices should be auto assembled and started. Only 1 of 
the 3 was assembled and started, the other 2
failed to assemble. I'm not sure why one did assemble and start and the 
other 2 not, under 2.6.32. At this point I'm
shelving the upgrade until this is resolved. (I'm thinking the problem 
is due to the reordering of the controllers)

Steve.