I have been monitoring very closely the path of Dell of moving from an IT hardware manufacturer into a company that provides IT services , and given the progress made in the enterprise storage, this event will allow me to learn even more about Dell’s products and strategy .

As usual Dell has this event very well covered within the Social Media , and for that i give my hat to Gina Minks and Alison Krause. To get all the information related to this event from a social media point of view check out this great blog post by Gina, and follow the hashtag #DellSF12 .

The event will be packed of leading IT folks , like :

Dell

• Gina Minks -  gminks.edublogs.org
• Jason Boche - boche.net
• Lance Boley - iscsiking.info
• Will Urban
• Liem Nguyen - compellent.com/blog
• Alison Krause

Bloggers

• Fabio Rapposelli - juku.it
• Hans Deleenheer - hansdeleenheer.blogspot.com
• Greg Knieriemen - NekkidTech.com
• Ed Saipetch - breathingdata.com
• Martin MacLeod - bladewatch.com
• Chris Evans - thestoragearchitect.com
• Chris Mellor - theregister.com
• Nigel Poulton - nigelpoulton.com
• Barry Coombs - virtualisedreality.com
• Martin Glassborow - storagebod.com

I will provide more information during/after the event, so feel free to re-visit this blog any time soon

Disclaimer:

This is an event sponsored by Dell ,and therefore all  my travel and living expenses to attend the Storage Forum are covered by Dell. However, my independency as an IT profissional/blogger is guaranteed. I will attend the Storage Forum due to the fact that within my professional area Dell plays a vital role and knowledge is power! 

I pretty sure that anyone that works in IT has quite some interesting issues to deal with , so read this post keeping in mind the “Your milleage may vary” slogan.

As such ,first things first, and let me start by giving some stats about my current IT environment to better understand what kind of issues i talk about and what decisions have been made to achieve a better quality in the IT services delivered .

HQ IT environment – As of January 2011

• A user base of around 500 , where 50% are WinXP based (laptops and pc’s), 35% thin client based (Windows and Linux Terminal servers based) and 15% based on either HP-UX workstations or high-end PC workstations (XP based as well)
• Data shared across Windows, Linux and HP-UX environments, using CIFS, NFS and a big amount  of “creative solutions”
• Network topology based on 3Com/HP switches , with 1GbE as the dominant link speed. No network redundancy/resiliency deployed, either by using link aggregation, STP, or any other technique. Network strategy based on ad-hoc tasks/growth .
• Windows 2003 as the main file server, with around 7.5 TB of data
• Oracle environment with 12+ instances, using 3TB worth of data with it’s own storage solution (Open-E , NFS based)
• Opensolaris as the NAS/SAN environment, for NFS, and iSCSI
• Usage of end-of-support x86 servers from Sun
• Usage of Citrix XenServer as our ,only,Virtualization Hypervisor
• Server farm of around 80 physical servers, x86 based servers, and 80 virtual machines
• Usage of a miriade of different applications by the business, that had lower and lower availability due to systems crashing, lack of monitoring and “weird issues”
• Team of 6 persons, with 5 of them allocated at 100% to sysadmin tasks, and 1 (myself) with time spited as Sysadmin, Technical IT Leader, Infrastructure Consultant and many many other activities
• Workload of IT team members around 11.5 hours work/day , 6 days/week
• IT environment managed from 6am until 1am CEST time
• IT budget and Business needs with a big mismatch , leading to wrong expectations and bad IT services with a direct consequence of end-user lack of trust in IT

HQ IT environment problems - As of January 2011

• IT could not meet the increasing demand from Business to support more and more applications with a higher SLA (higer means something better than best effort ) and lower RTO/RPO
• Ever increasing pain to backup data, mainly data hosted using virtualization
• Users demand to access the same data any time, any where, from multiple devices, while IT couldn’t deal with this demand
• Network errors and bottleneck encountered without a proper way to troubleshooting it
• Storage devices suffering from multiple crashes, leading to lengthy repair times and increasing user’s frustration
• Lack of support from a vendor that could go beyond the sales phase,or in other words lack of partnership with vendors in order to sell a solution rather than selling equipment
• IT department in charge of designing a new datacenter/network/IT environment for the new HQ to be ready at 2012 while keeping/improving the current environment

By now you , probably, have  a good picture of the environment and i hope you can map this specific environment to your own and hopefully you can take some good tips/points from this post.

Gladly for me (i work for having what i have by the way), i have the type of managers that actually listen to what i say and take me seriously and based on that , by the end of 2009 major decisions have been made by me and supported by my direct manager(s) and from other key users within the organization. Those decisions where made in terms of strategy of the servers/storage to use, network design, virtualization, backup and what can be done or not with the time/budget/resources available aKa expectations management .

The most critical decisions made in late 2009 where:

• Selection of a new IT (servers, laptops, PCs, storage) vendor. The choice was Dell , due to their good products but more important due to the fact that Dell was committed to work out with us to find a solution and not only to sell servers/desktops. Dell has proved to be a partner to us , that help us to find a solution rather than going directly to the sales phase.
• Selection of a new storage solution, based on iscsi technology – Dell Equallogic
• Selection of a new network vendor, to be used only for the new datacenter/HQ – Juniper
• Decision of adopting a layered approach for the virtualization hypervisor in use. This basically means that for diferent SLA, RTO/RPO of an application to be virtualized either Citrix XenServer or VMWare can be used. For lower SLA’s it has been decided to use Citrix XenServer and VMWare as the foundation block of our new environment to support critical business applications, like Databases, CRM, PLM , etc
• Decision of moving ( not consolidation per se) any x86 workload from a physical environment into a virtualized one, unless specific cases (for instance if a vendor doesn’t support virtualization for their product, like Product License Servers)
• Decision of stopping to use the current backup tool (EMC Legato) to backup of virtualized environments. PHD Virtual Backup for Citrix XenServer selected and Veeam for VMware environments . EMC Legato is still our corporate tool to backup any physical server, and i don’t see moving away from such tool anytime soon
• Provide IT Team members training in core technologies, like storage, networking and virtualization

Fast forward a year later , and here we are with outstanding results.

That are many advantages seen in our IT landscape but above all the business recognizes that the time to deliver a service from IT has decreased and the IT can now actually suggest and support and SLA towards the several business units .

Key factors for success

• Selection of the right IT vendor/partner rather than just focus on a pure technical point of view
• Don’t get impressed by buzzwords used by vendors..For instance if vendor A says “Our storage uses a 64 bit OS” , ask to yourself and to the vendor what advantage such feature will bring to your environment . In other words, make smart questions, and do some research prior any talk with any vendor
• Plan, execute, invest in creating tasks in an automatic mode with proper error handling
• Invest proper time/efforts in the art of End-User Expectation Management
• Don’t get stuck to procedures just because they exist for long time.The fact they exist for long time, doesn’t mean they are still suitable for the the present/future needs
• Attend industry recognized seminars , like given by Greg Schulz  – @storageio(http://www.storageio.com/)
• Attend industry recognized events, like VMWare World ( http://www.vmworld.com )
• Invest time in reading blogs, whitepapers, attending webcasts and social media (mainly Twitter)

To wrap up i need to say thank you to some key persons i meet over Twitter that have helping me in many ways for the past 18 months.

Here’s the list (without any sort of preference) , based on categories:

Veeam

@RickVanover  (good technical discussion at VMWorld 2011 Copenhagen ! Looking forward for VMWorld 2012 in Barcelona )

@gostev (If you’re into Veeam…this guy r0cks ! )

@VMDoug

Dell

@DellTechCenter

@DennisMSmith

@KongYang

@JeffSullivan

@supertsai

@iSCSIKing

@PeterVerwulgen

@VirtWillU

@jasonboche

@RafaelKnuth

@gminks ( You’re probably the acquisition made by Dell, regarding how can  social media should be done within a business ! )

Virtualization – VMware

@DuncanYB

@FrankDenneman

@esloof

@lamw

IT people

@hansdeleenheer

@cbarneveld

7IDC9P

The purpose of this gathering is discuss topics around IT, specially storage and virtualization , and to, off course, do some social networking.. After all IT people are above all : real life persons with life beyond IT.

So  feel free to drop by starting at 18.30 (i will wait at least until 20H) for some chat, beers and snacks…To make things easier i will have my VMworld badge on top of the table, so don’t be shy and just introduce yourself !
To make things even easier, here’s a map from Google and one from Hard Rock Café itself .

Hope to see you soon

Bruno | @bsousapt

I hope that during the event i will be able to chat with IT vendors and IT peers regarding virtualization and everything around IT, because virtualization is just part of the “big picture”.

For instance i look forward to discuss real life experiences in databases (Oracle/MS SQL) virtualization, with all their advantages or lack off..

Since i had never been into a VMworld before i really don’t know exactly what to expect, but i do know that i will try to get as much technical information as possible and to meet quite some really smart folks that “orbit” around the VMware/VMworld ecosystem.

As i start to attend the event itself i will try to update informations regarding the sessions and the event itself, so stay tuned aKa bookmark this page and came back later

For now, it’s time to get another dose of some medicines, because the flu decided to come along with me to this event…

@bsousapt

If you ever wondered how to install Dell Openmanage agents in a VMware host only by executing a simple script, fear no more..It’s possible, it’s quite simple and i will show you how

I recently had to install 7 physical hosts [Dell R515] with VMware ESXi 4.1 Update 1 and for that i come up with a very small and easy script to install Dell Openmanage agents.

For this task i just had to download the Dell OpenManage Offline Bundle and VIB for ESXi , create a local directory [c:\vmware] and run my script.

Please be aware that you need to have installed the VMware Vsphere CLI [ i used version 4.1] and in my case i installed in a 64Bits machine.

The script does the following :

1. set’s the physical host in maintenance mode
2. installs the Dell OpenManage Offline Bundle VIB
3. brings the physical host out from the maintenance mode

To make life easier i have zipped the Dell OpenManage Offline Bundle VIB together with the script and you download it here Dell OpenManage Offline Bundle and VIB for ESXi . Please make sure you rename the file from .doc to .zip [wordpress doesn't allow to upload zip files] and extract it to c:\vmware and not c:\vmware\vmware .

At the end you should be able to see the hardware status within the vSphere client .

Script contents  # START #

#Set the physical host in Maintenance Mode

cd “C:\Program Files (x86)\VMware\VMware vSphere CLI\bin\”
vicfg-hostops.pl –operation enter –server=your_vmware_host_ip –username=root –password=yourpassword

## Install OpenManage Agent – Dell OpenManage Offline Bundle and VIB for ESXi

cd c:\vmware

vihostupdate.pl –install –bundle C:\VMware\OM-SrvAdmin-Dell-Web-6.4.0-1266.VIB-ESX41i_A00.7.zip –server=your_vmware_host_ip  –username=root –password=yourpassword

#exit maintenance mode
cd “C:\Program Files (x86)\VMware\VMware vSphere CLI\bin\”

vicfg-hostops.pl –operation exit –server=your_vmware_host_ip –username=root –password=yourpassword

Script contents  # END #

As you know VMware has announced some time ago the vsphere 5  and finally it can be downloaded by anyone

Based on this i decided to test the performance of the 3 types of disks supported by VMware :

Basically the VM had 2 virtual disks assigned for the benchmark :

• Thick Provision Lazy Zeroed
• Thick Provision Eager Zeroed
• Thin Provision

Test setup :

One VM has been configured with 2 vcpu’s, 2 GB ram and 4 disks, all using the LSI SAS SCSI Controller :

• Disk 01 is the Windows “disk”  (c:\ drive)
• Disk 02 it’s a Thick Provision Lazy Zeroed disk
• Disk 03 it’s a Thick Provision Eager Zeroed disk
• Disk 04 it’s a Thin Provision disk

The VM had 1 Gigabit link without any sort of redundancy, link aggregation, no MPIO , no jumbo frames to an ISCSI storage. The purpose of this exercise is to see the performance differences between the 3 types of disks and not to see the performance of the ISCSI storage .

Anyway….let’s go for the results

Results : ( results parsed at http://vmktree.org/iometer/ )

•     I/O of a Thick Provision Lazy Zeroed disk

• I/O of a Thick Provision Eager Zeroed disk

• I/O of a Thin Provision disk

Conclusion :

It seems like VMware has quite similar performance across different types of disks (at least with the used benchmark profile for this test) and for me the Thin Provision disk would probably be the chosen one due to the fact of being…Thin .

During the next days/weeks i will try to get some more tests, against different storage devices and using ISCSI and NFS .

Check it out at www.vmware.com and have fun !

Basically users that had hard-drives with reported soft-errors stopped having access to those drives , and a spare process kicked in leading to performance drop and with one less available hard-drive.

To avoid some confusion the newest firmware is the 5.1.1-H1 (R187334) and the oldest one was the 5.1.1(R185010) .

If you have an Equallogic can you take a look to your current firmware by selecting within the Members tab , the Equallogic Member and select the “Maintenance” tab . You should see something like this :

.

As a side note…If you have this new firmware , within the SANHQ you may see some new attributes…

Have fun, and make sure you plan a maintenance window for any change on your environment, and that also includes any Equallogic firmware upgrade .

Based on the Dell Operating System Support Matrix this raid controller is only supported with Redhat / CentOS 5.5 , and this was a a major issue .
After digging around (Google is really your friend !) i was able to fix the problem by creating a device driver to be loaded during the installation of RedHat / CentOS 5.2 , either x86 or x86-64 bits platforms.

In short i had to download the already compiled driver available at the Supermicro support site , remove the signature from the driver and boot the system using the dd as an argument for the installer.

The signature had to be removed otherwise it would only work with RedHat and not with CentOS . For the procedure of removing the signature i followed the excellent post available at the CentOS forum

I have tested the driver with RedHat / CentOS 5.2 x86 and x86-64 bit installers and it works like a charm.

If the driver is loaded you will see that the disk is detected by RedHat / CentOS

and dmesg will output something like

The working device drivers can be downloaded here for the x86 and for the x86-64 (please remove the .pdf extension of the files…this is due to file upload restrictions) .

Have fun  !

To make use of the driver follow this instructions :
Installing the driver on a Red Hat Enterprise Linux / CentOS system

Perform the following steps to install RHEL/CentOS (version 5)
using the Driver Update Disk (DDU):

1 Boot normally from the RHEL/CentOS installation media.
2 At the command prompt, type:
> linux dd
3 When the install prompts for a driver diskette, insert the driver update
disk (DUD) prepared earlier and press <Enter>.
4 Complete the installation as directed by the installation program.

- iDRAC -

When you use iDrac Enterprise it’s also possible to mount a driver image to a floppy disk, just like
it’s done for mounting an ISO file .

- Create a Driver Update Disk (DDU) –

Using a USB key:

Copy the driver image file(*.img or *.dd) into a USB key.

Using a USB floppy device :

1. Put a floppy disk into a USB floppy device, and plug the device into a USB slot of the system under test.
use “dmesg” to find out which device this USB floppy is enumerated to (for example, sdb, sdc, etc).

2. Then tranfer the driver image to the floppy:

Use > dd if=<image_file_name> of=/dev/sdx

Here the /dev/sdx is the device name found from dmesg in the above step.

3. Umount the USB floppy device before unplug it (preferred).

Using system floppy disk slot:

Insert a floppy disk into the system floppy disk slot, then at the prompt:

# dd if=<image_file_name> of=/dev/fd0

On most systems, the floppy drive is designated by the /dev/fd0 special file.
It is prefferred to execute the dd command before the floppy is mounted.

Module signature removal procedure :

I just fixed it. The signature should be removed from the kernel module:

– BEGIN
# download mpt2sas-02.00.00.00-1-rhel5.3.x86_64.dd from ftp.supermicro.com

# mount driver disk as loop device
mkdir /mnt/loop
mount -o loop mpt2sas-02.00.00.00-1-rhel5.3.x86_64.dd /mnt/loop

# extract modules.cgz
mkdir -p /tmp/dd/cpio
cd /tmp/dd
gzip -dc /mnt/loop/modules.cgz > modules.cpio
cd cpio
cpio -idumv < ../modules.cpio

# check if the module is signed
cd 2.6.18-128.el5/x86_64
readelf -S scsi_transport_sas.ko | grep module_sig

# remove signature from module
objcopy -R .module_sig scsi_transport_sas.ko

# check if signature was removed
readelf -S scsi_transport_sas.ko | grep module_sig

# rebuild modules.cgz
cd /tmp/dd/cpio
find . | cpio -ovBc > ../modules
cd ..
gzip -9 -S .cgz modules
cp -f modules.cgz /mnt/loop

# unmount driver disk
umount /mnt/loop
–END

I got so many questions from some of my co-workers asking how this , (easy and trivial, for me at least) task can be done, that i decided to record a small video of the entire process.

If you’re interested take a look to the above video and see how a firmware upgrade can be done.. For more information check the Dell Techcenter iDrac page and the iDrac6 documentation .