[Discuss] Dev Ops - architecture (local not cloud)

[greg at freephile.com (Greg Rundlett (freephile))]

We are replacing a monolithic software development IT infrastructure where
source code control, development and compiling all take place on a single
machine with something more manageable, scalable, redundant etc.  The goal
is to provide more enterprise features like manageability, scalability with
failover and disaster recovery.

Let's call these architectures System A and System B.  System A is
"monolithic" because everything is literally housed and managed on a single
hardware platform.  System B is modular and virtualized, but still running
in a traditional IT environment (aka not in the cloud).  The problem is
that the new system does not come close to the old system in performance.
 I think it's pretty obvious why it's not performing: user home directories
(where developers compile) should not be NFS mounted. [1]  The source
repositories themselves should also not be stored on a NAS.

What does your (software development) IT infrastructure look like?

One of the specific problems that prompted this re-architecture was disk
space.  Not the repository per se, but with 100+ developers each having one
or more checkouts of the repos (home directories), we have maxed out a
4.5TB volume.

More specifically, here is what we have:
system A (old system)
single host
standard Unix user accounts
svn server using file:/// RA protocol
4.5TB local disk storage (maxed out)
NFS mounted NAS for "tools" - e.g. Windriver Linux for compiling our OS

system B (new system)
series of hosts managed by VMWare ESX 5.1 (version control host + build
servers connected via 10GB link to EMC VNXe NAS for home directories and
tools and source repos
standard Unix user accounts controlled by NIS server (adds manageability
across domain)
svn server using http:/// RA protocol (adds repository access control and
management)
NFS mounted NAS for "tools", the repositories, the home directories

Notes:
The repos we're dealing with are multiple "large" repositories eg. 2GB
43,203 files, 2,066 directories.
We're dealing with 100+ users



[1]
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/misuses_nfs_perf.htm

Greg Rundlett